Our session will address the steps Boddle, an edtech solution provider, has taken in recent years to examine the efficacy of its intervention and build an evidence base showing impact on student outcomes. In partnership with Instructure, Boddle conducted ESSA-aligned studies in 2022 and 2025. The research we will discuss in this presentation adheres to the Every Student Succeeds Act (ESSA) requirements, meeting Level IV (Demonstrates Rationale) criteria through the development of a logic model and underlying research base, and Level III (Promising Evidence) criteria through a correlational study with statistical controls. This approach ensures the evaluation is grounded in rigorous methodology and focused on validating edtech effectiveness.
Boddle's design was guided by previous research examining game-based learning. Studies of K-16 learners have shown that playing digital games may be positively related to a variety of competencies, attributes, and outcomes, such as college grades (Ventura, Shute, & Kim, 2012), persistence (Ventura, Shute, & Zhao, 2012), and creativity (Jackson et al., 2012). Furthermore, meta-analyses suggest that digital games significantly improve student learning relative to conventional instruction methods (e.g., lectures, reading, drill and practice, or hypertext learning environments), but that effects vary by game mechanics, visual and narrative, and research quality characteristics (Clark, Tanner-Smith, & Killingsworth, 2015).
Superficial characteristics, such as 3D graphics, are less indicative of a well-designed game compared to games with an underlying architecture drawing on a player’s zone of proximal development (Gee, 2003; Vygotsky, 1978) that incorporate active, engaging, goal-oriented, contextualized, adaptive, and interesting tasks (Bransford, Brown, & Cocking, 2000; Shute, Ke, Almond, Rahimi, Smith, & Lu, 2019). Adaptive challenges and ongoing performance feedback within games may also foster growth mindset (Dweck, 2006).
Boddle also draws on the Octalysis Framework, a human-focused gamification design framework that analyzes and lays out the driving forces behind human motivation (Chou, Y. K., 2019).
***
Studies that are being presented:
Scanlan, A. & Henschel, M. (2025). Boddle Learning Logic Model: Study Type: ESSA Evidence Level III. LearnPlatform
Shah, M., Long, C., & Styers M. (2022). Boddle Learning Logic Model: Study Type: ESSA Evidence Level IV. LearnPlatform
Our presentation focuses on two stages of evidence building: ESSA Level IV (Demonstrating Rationale) and ESSA Level III (Promising Evidence).
ESSA Level IV Study
The Level IV study, completed in 2022, focused on defining the foundational research base for Boddle and documenting the program's logic model. This established a roadmap detailing program inputs, participants reached (K-6 students, educators, and parents/guardians), program activities (e.g., creating assignments, students attempting practice problems, earning coins), outputs (quantifiable indicators like number of student logins and proportion of correct responses), and short-term, intermediate, and long-term outcomes. The Level IV study also drafted the design for the subsequent ESSA Level III evaluation.
ESSA Level III Study
This was a correlational study (only looking at users) conducted during the 2024–25 school year. This design meets the requirements for ESSA Level III (Promising Evidence). The study aimed to examine the relationship between Boddle platform usage and student outcomes.
The sample included 83 3rd grade students from one elementary school in a diverse, urban/rural public school district in Alabama. All participating students used Boddle as a supplemental math program.
Data Sources:
Usage Metrics (Independent Variables): Data collected directly from the Boddle platform included the percentage of correct responses and the number of sessions completed during the 2024–25 school year.
Academic Outcomes: Student academic performance was measured using Beginning-of-Year (BOY) and End-of-Year (EOY) i-Ready Math assessments.
Affective Outcomes (Math Anxiety): Math anxiety was measured using an 11-item student survey adapted from Primi et al.’s (2020) Early Elementary School Abbreviated Math Anxiety Scale. The survey used a five-point Likert scale (1 = really, really worried; 5 = not worried at all). The survey was administered once to a subset of 61 students using the Qualtrics platform between April 29 and May 6, 2025.
Demographic Data: The school provided student enrollment and demographic data (classroom, gender, and race/ethnicity).
Methods of Analysis:
Implementation Analysis: Descriptive statistics were calculated for usage metrics, and ANOVA was used to test for group differences by gender and race/ethnicity.
Outcome Analysis: Partial correlations were employed to analyze the associations between Boddle usage and outcomes. Statistical controls were applied to account for selection bias. BOY i-Ready Math scores were used as a control variable in all correlational analyses. Gender was also included as a statistically significant control variable for the analysis examining EOY i-Ready Math scores.
The ESSA Level III study was completed in September, 2025, and yielded several positive, statistically significant findings.
Achievement Outcomes (EOY i-Ready Math Scores):
Positive Finding: Third grade students who achieved a higher percentage of correct responses on Boddle had statistically significantly higher end-of-year (EOY) i-Ready Math scores (r = 0.34; p = .002), after controlling for BOY i-Ready Math scores and gender.
Affective Outcomes (Math Anxiety Survey):
Positive Findings: Third grade students who had a higher percentage of correct responses on Boddle reported feeling less anxious when learning a brand-new kind of math problem (r = 0.28, p = .033). Also, Third grade students who completed more sessions on Boddle reported feeling less anxious when using Boddle for math (r = 0.26, p = .047).
Negative Finding: Students who had a higher percentage of correct responses on Boddle reported feeling more anxious when they had to ask for help with math (r = -0.26, p = .048). Researchers noted that this finding suggests that higher-performing students might feel added pressure to perform in math, a dynamic which needs further exploration.
This study is significant because it documents the early stages of Boddle’s evidence-building journey and illustrates how an edtech provider can take honest, incremental steps to validate its impact. While many tools make unsubstantiated claims, Boddle has committed to a transparent process aligned with ESSA, beginning with a Level IV logic model grounded in socio-cultural, cognitive, and motivational theories, and progressing to a Level III correlational study with statistical controls. These steps are foundational, not final, but represent an intentional and rigorous approach to building a long-term evidence base - and can yield lessons for other edtech companies of a similar size and with similar programatic goals.
The findings contribute both academically and practically. Positive associations between Boddle usage and end-of-year math scores highlight the promise of well-designed game-based learning. Equally important, the inclusion of a math anxiety survey adds a qualitative dimension to a quantitative design, revealing insights into student experiences that test scores alone could not capture. In this specific case, positive associations between Boddle usage and students’ math anxiety suggest improved students’ well-being as well as test scores. However, the negative association between students who had a higher percentage of correct response and feeling comfortable asking for help with math, suggests an interesting dynamic around potential math pressure amongst high performers that warrants more research.
The Level III study offers a compelling example of evidence-building but also has clear limitations. The sample was small and drawn from a single school, and all participants were Boddle users—meaning causal claims cannot yet be made. Replication in other contexts and with larger, more diverse samples is needed. The natural next step is an ESSA Level II quasi-experimental study incorporating non-user comparison groups to test whether Boddle directly causes improvements in outcomes.
Including an educator in this presentation also affords a unique opportunity to discuss some of the themes from these findings with a real-world practitioner with relevant expertise in classrooms. This perspective grounds each study, connecting the research directly to where the vital practice of improving student outcomes is taking place.
For conference audiences, this research provides a replicable model of early-stage evidence-building that bridges theory, data, and classroom realities. It demonstrates how combining usage metrics, achievement outcomes, and student perceptions produces a fuller picture of edtech’s impact, even in a small-scale ‘pilot’ study, while underscoring that evidence-building is a process—one requiring humility, transparency, and continuous improvement in the pursuit of student success and well-being.
References:
Bransford, J., Brown, A., & Cocking, R. (2000). How people learn: Brain, mind, and experience & school. Washington, DC: National Academies Press.
Chou, Y.-K. (2019). Actionable Gamification: Beyond Points, Badges, and Leaderboards. Octalysis Media.
Clark, D. B., Tanner-Smith, E. E., & Killingsworth, S. S. (2016). Digital Games, Design, and Learning: A Systematic Review and Meta-Analysis. Review of Educational Research, 86(1), 79–122.
Dweck, C. (2006). Mindset: The new psychology of success. New York: Random House.
Every Student Succeeds Act, Pub. L. No. 114-95 (2015).
https://www.govinfo.gov/app/details/PLAW-114publ95.
Gee, J. P. (2003). What digital games have to teach us about learning and literacy. New
York: Palgrave Macmillan.
Jackson, L. A., Witt, E. A., Games, A. I., Fitzgerald, H. E., von Eye, A., & Zhao, Y. (2012). Information technology use and creativity: Findings from the children and technology project. Computers in Human Behavior, 28, 370–376.
Primi, C., Donati, M. A., Izzo, V. A., Guardabassi, V., O’Connor, P. A., Tomasetto, C., & Morsanyi, K.
(2020). The Early Elementary School Abbreviated Math Anxiety Scale (the EES-AMAS): A new adapted version of the AMAS to measure math anxiety in young children. Frontiers in Psychology, 11, Article 1014. https://doi.org/10.3389/fpsyg.2020.01014
Scanlan, A. & Henschel, M. (2025). Boddle Learning Logic Model: Study Type: ESSA Evidence Level IV. LearnPlatform
Shah, M., Long, C., & Styers M. (2022). Boddle Learning Logic Model: Study Type: ESSA Evidence Level IV. LearnPlatform
Shute, V. J., Ke, F., Almond, R. G., Rahimi, S., Smith, G., & Lu, X. (2019a). How to increase learning while not decreasing the fun in educational games. In R. Feldman (Ed.), Learning Science: Theory, Research, and Practice (pp. 327–357). New York, NY: McGraw Hill.
Ventura, M., Shute, V. J., & Kim, Y. J. (2012). Video gameplay, personality and academic performance. Computers & Education, 58, 1260–1266.
Ventura, M., Shute, V., & Zhao, W. (2012). The relationship between video game use and a performance-based measure of persistence. Computers & Education, 60, 52–58.
Vygotsky, L. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.
What Works Clearinghouse (2022). What Works Clearinghouse procedures and standards
handbook, version 5.0. U.S. Department of Education, Institute of Education Sciences,
National Center for Education Evaluation and Regional Assistance (NCEE). This report is
available on the What Works Clearinghouse website at
https://ies.ed.gov/ncee/wwc/Handbooks
Back
Trips and Tours
