Event Information
Mathematics learning continues to be an important focus for national improvement. It is particularly concerning after COVID-19; math outcomes continued to decline in 2023 (NCES, 2023), likely due in part to the transition to distance learning during COVID-19 (Jack et al., 2023). As a solution, more districts in the United States are investing a considerable amount of their funding in education technologies as primary or supplemental programs with the objective of improving math performance. It is timely to address students’ learning and engagement with math technologies, particularly early in elementary school.
Developmental and cognitive theories highlight the crucial role of early math skills and knowledge in predicting later academic success and preparedness for 21st-century STEM careers (Chu, et al., 2016; Claessens & Engle, 2013; Jordan et al., 2009; Watts, et al., 2014). The results from the National Assessment of Education Progress (NAEP), however, show that only 40 percent of fourth graders are proficient in math (NCES, 2023). The situation is worse for children from low socioeconomic backgrounds who begin school with significantly less math knowledge (Nores & Barnett, 2014) and demonstrate gaps in the development of early math skills relative to their middle-class peers (Nores & Barnett, 2014; Starkey et al., 2004).
Highly engaging educational technology innovations for young learners, such as My Math Academy, have the potential to support teachers and prepare students for success in math. My Math Academy is a supplemental that provides a personalized mastery-based learning ecosystem (Bingham et al., 2018; Plass & Pawar, 2020), aligning with the Common Core State Standards in Mathematics in early elementary grades. This program includes game-based activities with adaptive learning trajectories. As students navigate through the activities, the games include integrated, continuous assessments, which enable ongoing formative feedback cycles and customized content based on learner difficulty levels. It integrates evidence-centered design, an assessment framework that provides a principled alignment of concepts, skills, and abilities that a game is designed to teach with evidence of learning and task design (Clarke-Midura et al., 2012; DiCerbo et al., 2015; Mislevy et al., 2003; Shute, 2011).
My Math Academy leverages games as a vehicle for playful engagement (Piaget, 1962; Fisher et al., 2012), learning in context, and formative assessments (Shute & Kim, 2014). The story context engages children in play (Fisch, 2000; Figure 1) and makes concepts and operations more meaningful to students, providing a framework for understanding what they are expected to do and why (Sullivan, et al., 2003).
The educator dashboard provides real-time data about student progress, proficiency, and learning trajectories, allowing teachers to monitor individual and group progress and to better tailor classroom instruction and accelerate learning. The dashboard also suggests additional learning activities for students that need additional reinforcement of specific concepts and provides offline activities that extend students’ in-game experiences.
My Math Academy has been shown significantly improve young students (ages 4-6)’ math ability in two pilot randomized controlled trial (RCT) studies (Thai and Li, 2018; Bang et al., 2023).
Purpose/Research Questions
The purpose of the study is to examine the impact of My Math Academy, a digital game-based, personalized early math program. The study was funded by a U.S. Department of Education’s Institute of Education Sciences grant to support early elementary students build a strong understanding of fundamental number sense and operations. In particular, we will address My Math Academy’s impact on math learning in kindergarten classrooms and the implications for supporting early math development using game- and technology-based, personalized programs. The study aims to address the following research questions:
• What’s the impact of My Math Academy on kindergarten students’ math learning outcome after being used for a year? Do the effects vary by subgroups of students?
• What’s the relationship between treatment students’ learning outcome and their activities and performance within My Math Academy?
• How are My Math Academy implemented in the classrooms to engage and empower students?
Settings / Population
The present two-cohort, multi-year study used a blocked cluster randomized experimental design where schools were the unit of random assignment. In cohort 1, during the 2023-24 school year, a total of 14 schools (36 kindergarten teachers) were recruited from five public school districts. Seven schools with 21 classrooms were randomly assigned to the treatment group that implemented My Math Academy; seven schools with 15 classrooms were randomly assigned to the control group that implemented their usual math activities. In year 2, schools continued research activities in the same group assignments but in first grade classrooms (analyses forthcoming). In the second cohort, during the 2024-25 school year, 14 schools were randomly assigned to the treatment group and 16 schools to the control group, for a total of 30 schools with 54 kindergarten classrooms from eight school districts. The study was reviewed and approved by the Institutional Review Board at WestEd.
Intervention
During the study, kindergarten teachers in the treatment group were expected to implement My Math Academy with their students for at a minimum of 45 minutes per week over at least 14 weeks of the school year as a supplement to their existing math program. Students accessed My Math Academy on individual devices with headphones where teachers could flexibly implement My Math Academy as a whole-class or in small groups as part of center or independent choice times. Treatment group teachers were encouraged to monitor their students’ progress through the teacher dashboard and use the recommended off-line resources to support their students’ learning. They were also offered access to self-paced professional learning modules created by the program developer designed to help teachers understand the philosophy, design of My Math Academy, how to use data and resources, and functionality of different features to optimize the implementation of My Math Academy in classrooms.
Data Collection and Analysis
Test of Early Mathematics Abilities–3 (TEMA-3; Ginsburg & Baroody, 2003), a test of children’s informal and formal mathematics knowledge, was used to evaluate students in participating classrooms in the fall and spring of the kindergarten year. A subset of those students assessed in kindergarten (N = 100, 50 treatment and 50 control) were followed into first grade and assessed again on the TEMA-3 in the spring of the second year (analyses forthcoming).
The My Math Academy Analytics logged students’ actual exposure to My Math Academy. My Math Academy collects real-time game play usage data as students use it. It collects extensive digital records of all student actions, including student in-system preassessment performance, usage (time spent, activities completed), progress through the system (boss levels passed), and performance (score) on each activity.
Teacher interviews were conducted with 10 kindergarten teachers in treatment schools in year 1 and 11 teachers in year 2. The interviews asked about teacher perspectives on the “5 E’s” theorized to underlie My Math Academy’s effect on learning: helping students stay focused on math (Engagement); complementing or supplementing the core mathematics curriculum (Effectiveness); addressing the needs of diverse learners (Equity); supporting teachers to understand their students through data and tools (Empowerment); and the supportive context in which teachers choose to use My Math Academy (Environment).
Impact Findings. The study saw no school-level attrition and the treatment and control groups did not differ significantly at baseline (p = 0.62). The primary impact analysis included 993 students, 529 treatment and 464 control. Twenty-three additional students (8 treatment and 15 control) participated in the fall pretest but dropped from the study before the posttest. The impact analysis was a hierarchical linear model with students nested in schools controlling for pretest, gender, and race/ethnicity. This analysis did not reveal evidence of a significant effect of the intervention (p = .224, effect size = 0.06), although students in the treatment group scored higher on the end-of-year math assessment than those in the control group (34.86 vs. 34.28, a difference of 0.58 points). Interestingly, students who started the school year performing low on the math assessment (bottom 50% of the pretest) showed a trend towards performing better than the control group at the end of the year (p = .077, effect size = 0.14). These findings suggest that the intervention may be particularly beneficial for lower performing students.
My Math Academy Analytics. Analysis of the My Math Academy usage data for the second cohort is ongoing, but initial evidence from treatment group students in the first cohort indicated that there was a relation between students’ My Math Academy usage and their gain in math score from baseline to the end of the year. Specifically, students who showed high levels of growth in their math scores over the school year tended to pass more activities and math topic areas in My Math Academy and also spent more time in the system. Forthcoming analyses will continue to examine whether growth in math scores is related to My Math Academy usage for both cohorts of treatment group students. If this finding is borne out in the full sample, it could suggest that using My Math Academy more frequently, including covering more math topics and activities within the system, could enhance students’ math skills.
Teacher Interviews. Findings from the teacher interviews addressed teachers’ perspectives on whether My Math Academy supported the 5 E’s. Most teachers reported that they felt that My Math Academy supported their students’ math learning. One teacher commented, “I feel like several of them have gotten more confident [and] picked up the counting a lot quicker.” Another noted, “My Math Academy made it where I could inspire them to learn or encourage them to continue to learn more or to master that skill.” Many of the teachers felt that My Math Academy was well-aligned with their math curriculum (Effectiveness). For example, one teacher said, “I think it's got a good range of activities that are meeting the kindergarten standards as well as reviewing some preschool and introducing some first grade math standards as well.”
Many of the teachers reported that all or most of their students were highly engaged with My Math Academy (Engagement). One teacher shared, “They're absolutely excited. I mean, they're even elated when it's time to go to My Math Academy… I noticed that they concentrate and for the most part, and they are constantly active.” Many teachers reported that My Math Academy helped with differentiated learning (Equity). These teachers reported that My Math Academy provided remediation to struggling students and challenged students who needed enrichment. A teacher commented, “I like that it progresses so the kids who are moving at a faster pace can move on and those kids who need that extra support, it's just going over the basic skills that they need.” Most teachers implemented My Math Academy with their whole class at once while the teacher circulated, monitored students’ usage time, and provided support as needed, while some implemented it as part of center time (Environment). One teacher noted My Math Academy’s implementation flexibility, saying, “The big ‘Aha!’ for me was that there's a lot of flexibility in using the program. It can be beneficial as a center for a small group of students. It can be supportive to let students who are in tutoring use it. And then we really enjoyed using it as a whole group too.”
This study highlights the effects of a technology-based math program that has the flexibility to be implemented in kindergarten classrooms. My Math Academy illustrates how effective design strategies can empower technology-based programs by incorporating playful games, contextualized instruction, and personalized learning pathways to support young children’s engagement in math learning. The findings suggest that My Math Academy can be flexibly and regularly integrated into school instructional practices and shows potential for engaging students and improving their learning of grade-level content. Students who used My Math Academy in kindergarten for one school year tended to score higher than those who experienced business-as-usual math instruction, though the difference did not reach statistical significance. However, students who started the year with lower baseline math performance showed a stronger trend towards outperforming their business-as-usual peers, suggesting that My Math Academy could be particularly beneficial for students who are struggling in math. Further, in interviews, teachers reported that My Math Academy could be flexibly implemented either with the whole class or in small groups, and students were overall engaged while using the program. Most teachers felt that My Math Academy supported their students’ math learning and was beneficial for differentiated learning, providing support to struggling learners and opportunities for enrichment for students who were more advanced. Overall, this study shines new light on how to implement an in-depth, digital game-based program to empower teachers, provide novel learning opportunities and environments for diverse students, and support student engagement and learning.
Bang, H.J., Li, L. & Flynn, K. (2023). Efficacy of an Adaptive Game-Based Math Learning App to Support Personalized Learning and Improve Early Elementary School Students’ Learning. Early Childhood Education. J 51, 717–732.
Bingham, A. J., Pane, J. F., Steiner, E. D., & Hamilton, L. S. (2018). Ahead of the curve: Implementation challenges in personalized learning school models. Educational Policy, 32(3), 454-489.
Chu, F. W., vanMarle, K., & Geary, D. C. (2016). Predicting children’s reading and mathematics achievement from early quantitative knowledge and domain-general cognitive abilities. Frontiers in Psychology, 7, 1–14. https://doi.org/10.3389/fpsyg.2016.00775
Claessens, A., & Engel, M. (2013). How important is where you start? Early mathematics knowledge and later school success. Teachers College Record, 115(6), 1–29. https://doi.org/10.1177/016146811311500603
Clarke-Midura, J., Mayrath, M., Dede, C., & Zap, N. (2012). Thinking Outside the Bubble: Virtual Performance Assessments for Measuring Inquiry Learning. In M. Mayrath, J. Clarke-Midura, D. Robinson (Eds.) Technology Based Assessment for 21st Century Skills: Theoretical and Practical Implications from Modern Research. New York, NY: Springer-Verlag.
DiCerbo, K. E., Bertling, M., Stephenson, S., Jia, Y., Mislevy, R. J., Bauer, M., & Jackson, G. T. (2015). An application of exploratory data analysis in the development of game-based assessments. Serious Games Analytics: Methodologies for Performance Measurement, Assessment, and Improvement, 319.
Fisch, S. M. (2000). A capacity model of children's comprehension of educational content on television. Media psychology, 2(1), 63-91.
Fisher, K., Hirsh-Pasek, K., & Golinkoff, R. M. (2012). Fostering mathematical thinking through playful learning. Contemporary debates on child development and education, 81-92.
Ginsburg, H. & Baroody, A. J. (2003) TEMA-3: Test of early mathematics ability. Pro-Ed.
Jack, R., & Oster, E. (2023). COVID-19, school closures, and outcomes. Journal of Economic Perspectives, 37(4), 51-70.
Jordan, N. C., Kaplan, D., Ramineni, C., & Locuniak, M. N. (2009). Early math matters: kindergarten number competence and later mathematics outcomes. Developmental psychology, 45(3), 85. https://doi.org/10.1037/a0014939
Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A Brief Introduction to Evidence-centered Design (p. 37). Princeton, NJ: Educational Testing Service. Retrieved from https://www.ets.org/Media/Research/pdf/RR-03-16.pdf
Murphy, M., Redding, S., & Twyman, J. (Eds.). (2016). Handbook on personalized learning for states, districts, and schools. IAP.
National Center for Education Statistics (NCES) (2023). NAEP Data Explorer. The Nations Report Card. https://www.nationsreportcard.gov/ndecore/landing
Nores, M. & Barnett, S. (2014). Access to high-quality early care and education: Readiness and opportunity gaps in America. New Brunswick, NJ: Center for Enhancing Early Learning Outcomes and National Institute for Early Education Research.
Piaget, J. (1962). The stages of the intellectual development of the child. Bulletin of the Menninger clinic, 26(3), 120.
Plass, J. L., & Pawar, S. (2020). Toward a taxonomy of adaptivity for learning. Journal of Research on Technology in Education, 52(3), 275-300.
Shute, V. J. (2011). Stealth assessment in computer-based games to support learning. In S. Tobias & J. D. Fletcher (Eds.), Computer games and instruction (pp. 503–524). Charlotte, NC: Information Age Publishers.
Shute, V. J., & Kim, Y. J. (2014). Formative and stealth assessment. In Handbook of research on educational communications and technology (pp. 311-321). Springer, New York, NY.
Starkey, P., Klein, A., & Wakeley, A. (2004). Enhancing young children’s mathematical knowledge through a pre-kindergarten mathematics intervention. Early Childhood Research Quarterly, 19(1), 99–120.
Sullivan, P., Zevenbergen, R., & Mousley, J. (2003). The contexts of mathematics tasks and the context of the classroom: Are we including all students?. Mathematics Education Research Journal, 15(2), 107-121. https://doi.org/10.1007/BF03217373
Thai, K., Bang, H.J., & Li, L. (2021). Accelerating early math learning with research-based personalized learning games: A cluster randomized controlled trial. Journal of Research on Educational Effectiveness. https://doi.org/10.1080/19345747.2021.1969710.
Watts, T. W., Duncan, G. J., Siegler, R. S., & Davis-Kean, P. E. (2014). What’s past is prologue: Relations between early mathematics knowledge and high school achievement. Educational Researcher, 43(7), 352–360. https://doi.org/10.3102/0013189X14553660
Other presentations in this group:
Back
Trips and Tours