Support Math Learning with Personalized Learning System That Responds to Student Affect

Framework

Emotion/affect plays a critical role in human cognition (Picard, 1997; Schutz & Pekrun, 2007). Engagement, motivation, and interest are precursors to learning, effortful problem solving, and deep thinking (Rose & Meyer, 2002; Seligman, 1991; Sweller et al., 1998). Students’ affect influences performance and strongly predicts achievement (Craig et al., 2004; Csikszentmihalyi, 1990; D’Mello & Graesser, 2007; Goleman, 1996; Pardos et al., 2014). Students who feel anxious or depressed fail to properly assimilate information (Goleman, 1996) and have limitations in “active” or “working memory” (Baddeley, 1986). Further, certain affective experiences (e.g., frustration, boredom) increase unproductive behaviors in interactive learning environments and hinder learning (Baker et al., 2004; Baker et al., 2008; Shute et al., 2015).

Emotions can be influenced in a variety of ways. For instance, the presence of someone who cares, or at least appears to care, can make students’ experiences more personal and help them persist at a task (Burleson & Picard, 2007). Empathic responses from a teacher or graphic character might work when students do not themselves feel positive about a learning experience (Graham & Weiner, 1996; McQuiggan et al., 2008; Zimmerman, 2000). Thus, a computer persona that appears to enjoy math experiences could transmit these positive feelings to students.

Methods

Setting and Population
The study was conducted in Massachusetts, which was well-suited to the state’s emphasis on personalized learning and differentiated instruction to better engage students and meet the needs of an increasingly diverse student body. The study sample included 80 5th- or 6th-grade mathematics teachers (41 treatment; 39 control) in three cohorts from 59 public schools in 42 districts with diverse backgrounds, including low-performing schools and schools with lower-socioeconomic status student bodies.

Research Design
The study employed a multisite, clustered randomized experimental design in which teachers were blocked by districts and randomly assigned to either use the MathSpring platform to support mathematics problem-solving for their 5th- or 6th-grade students or to a business-as-usual control condition. In the control condition, teachers continued to use their existing instructional practices and supplemental technologies (other than MathSpring).

Treatment Condition and Control Condition
Teachers in the treatment group received professional development training after randomization during the summer and started using MathSpring in their classrooms in the fall. Throughout the school year, they participated monthly in Professional Learning Community virtual meetings and had 1-on-1 check-in meetings with the implementation coordinator. To incentivize participation for control teachers, a delayed treatment design was implemented: teachers in the control condition were offered training and access to MathSpring after their cohort’s participation in the experiment was complete.

Data Sources
The primary measure of student mathematics achievement was the end-of-school-year Massachusetts Comprehensive Assessment System (MCAS) assessment. In addition, we administered the online grade-level Mathematics Diagnostic Testing Project (MDTP) Math Readiness Test as a supplemental measure (Anthony, 2005) as the beginning and end of the school year.

We used several previous validated scales to measure students’ growth mindset, learning strategies, and their dispositions toward mathematics. All measures were administered online to all students as both pre- and post-surveys.

To understand the nature of implementation and influence of the program on teachers’ practices, and to establish a contrast between conditions, we collected data from multiple sources, including pre- and post-surveys and instructional logs from all participating teachers, interviews with sampled teachers and administrators, and in-person observations of selected classrooms. MathSpring backend system log data that tracked student problem-solving actions and teacher usage of reports were collected continuously during the study and will serve as the primary data source for treatment implementation fidelity.

Results

We will document attrition and reasons for the observed attrition at the teacher level, as well as at student levels whenever possible. We will establish baseline equivalence between the two conditions based on students’ prior MCAS scores.

To analyze the efficacy of MathSpring, we will use a two-level hierarchical linear regression model to compare mean differences in the outcome measure between students in the two conditions, controlling for prior achievement and other covariates. Moderator analyses will examine the impact of the intervention on the learning of students with low-baseline mathematics achievement and different demographics. Mediator analysis will examine the link between student learning outcomes, students’ growth mindsets and attitudes toward mathematics, and teachers’ use of the reports toinform class instruction.

To provide a comprehensive understanding of the impact of MathSpring, we will also conduct qualitative analyses of data collected through interviews, surveys, or observations. These analyses will help clarify the context and provide an understanding of the contrast in instructional practices between the treatment and control classrooms. We have noticed that participating districts adopted multiple supplemental technology programs in remote instruction during the pandemic, which competed for classroom instruction time. We plan to analyze the MathSpring system records to calculate intervention dosage and use instructional logs to compare use of similar educational technology products between conditions. Additionally, we will conduct an analysis to examine cost and cost-effectiveness of MathSpring.

In June 2023, the final of the three cohorts of teachers and students completed the study. All data collection activities have concluded, with the exception of the spring 2023 MCAS test scores, which we will receive later this year. The analysis is proceeding as planned, to be completed by spring 2024. We will be able to share the findings at ISTE in June 2024. Preliminary analysis of teacher interviews suggested that teachers found great value in the unique social emotional learning components of the program and felt the MathSpring online PD modules were very useful.

Importance

Abundant visions exist for using technology to improve mathematics outcomes and help students recover from the learning loss caused by the pandemic. Educational technology programs are becoming widely available, and as they go to scale, it is important to measure their impact on teaching and learning. It is essential to attend to student affective states and promote student engagement and positive dispositions towards learning. This study investigates the potential effectiveness of an educational technology intervention that attends to both mathematics learning outcomes and students’ attitudes and mindsets. The findings from the study will add to the evidence base and inform the adoption and classroom practices of implementing similar educational technology programs.

References

Anthony, M. (2005). Consequential-related validity evidence for MDTP tests. Data submitted in response to California Community Colleges Assessment Standards for renewal of placement test instruments. Retrieved from Mathematics Diagnostic Testing Project website: http://mdtp.ucsd.edu
Baddeley, A. D. (1986). Working Memory. Oxford: Clarendon Press.
Baker, R. S., Corbett, A. T., & Koedinger, K. R. (2004). Detecting Student Misuse of Intelligent Tutoring Systems. In Proceedings of the 7th International Conference on Intelligent Tutoring Systems (pp. 531–540).
Baker, R. S. J. d., Walonoski, J. A., Heffernan, N. T., Roll, I., Corbett, A. T., & Koedinger, K. R. (2008). Why students engage in “gaming the system” behavior in interactive learning environments. Journal of Interactive Learning Research, 19(2), 185–224.
Burleson, W., & Picard, R.W. (2007). Evidence for Gender Specific Approaches to the Development of Emotionally Intelligent Learning Companions. IEEE Intelligent Systems, Special issue on Intelligent Educational Systems, Vol 22, No 4, July 2007, pp. 62-69.
Craig, S., Graesser, A., Sullins, J., & Gholson, B. (2004). Affect and learning: an exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29(3), 241–250.
Csikszentmihalyi, M. (1990). Flow: the psychology of optimal experience (1st ed.). New York: Harper & Row.
D’Mello, S. & Graesser, A. (2007). Mind and body: Dialogue and posture for affect detection in learning environments. In Proceedings of the 2007 Conference on Artificial Intelligence in Education: Building Technology Rich Learning Contexts That Work. 2007, IOS Press. pp. 161–168.
Goleman, D. (1996). Emotional intelligence: Why it can matter more than IQ. London: Bloomsbury.
Graham, S., & Weiner, B. (1996). Theories and principles of motivation. In D. Berliner, & R. Calfee (Eds.), Handbook of Educational Psychology (pp. 63–84). New York: Macmillan.
McQuiggan, S., Rowe, J., & Lester, J. (2008) The effects of empathetic virtual characters on presence in narrative-centered learning environments. In Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI'08), 1511–1520.
Pardos, Z. A., Baker, R. S., San Pedro, M. O. C. Z., Gowda, S. M., & Gowda, S.M. (2014). Affective states and state tests: Investigating how affect and engagement during the school year predict end of year learning outcomes. Journal of Learning Analytics, 1(1), 107–128.
Picard, R. W. (1997). Affective computing. Cambridge, MA: MIT press.
Roschelle, J., Feng, M., Murphy, R., & Mason, C. (2016). Online mathematics homework increases student achievement. AERA Open Journal. 2(4). https://doi.org/10.1177/2332858416673968
Rose, D. H., & Meyer, A. (2002). Teaching Every Student in the Digital Age: Universal Design for Learning. Alexandria, VA: Association for Supervision & Curriculum Development.
Schutz, P. A., & Pekrun, R. (2007). Emotion in education. Academic Press: San Diego, CA.
Seligman, M. (1991). Learned optimism. New York, NY: Knopf.
Shute, V. J., D’Mello, S., Baker, R., Cho, K., Bosch, N., Ocumpaugh, J., Ventura, M., & Almeda, V. (2015). Modeling how incoming knowledge, persistence, affective states, and in-game progress influence student learning from an educational game. Computers & Education, 86, 224–235.
Sweller, J., Van Merriënboer, J., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10, 251–296.
Zimmerman, B. J. (2000). Attaining Self-Regulation: A social cognitive perspective. In M. Boekaerts, P. Pintrich, & M. Zeodmer (Eds.), Handbook of Self-Regulation (pp. 13–39). Academic Press.