Learning Loss to Unfinished Learning: Mixed-Methods Research Identifying Accelerated Learning Strategies

Change display time — Currently: Central Daylight Time (CDT) (Event time)
Location: Virtual
Experience live: All-Access Package Year-Round PD Package Virtual Lite
Watch recording: All-Access Package Year-Round PD Package Virtual Lite

Listen and learn : Research paper
Lecture presentation

Research papers are a pairing of two 18 minute presentations followed by 18 minutes of Discussion led by a Discussant, with remaining time for Q & A.
This is presentation 2 of 2, scroll down to see more details.

Other presentations in this group:

Dr. Caitlin McLemore  
Dr. Beth Holland  

We'll share an explanatory mixed-methods study designed to understand learner growth during the COVID-19 pandemic. Using district benchmark data, findings indicated varied growth across learner subpopulations. Analysis of existing district supports and learning loss research revealed enabling strategies that may accelerate learning for all learners.

Audience: Chief technology officers/superintendents/school board members, Curriculum/district specialists, Principals/head teachers
Attendee devices: Devices not needed
Topic: Assessment/evaluations/use of data
Grade level: PK-12
Subject area: Math, Language arts
ISTE Standards: For Education Leaders:
Connected Learner
  • Develop the skills needed to lead and navigate change, advance systems and promote a mindset of continuous improvement for how technology can improve learning.
Equity and Citizenship Advocate
  • Ensure all students have access to the technology and connectivity necessary to participate in authentic and engaging learning opportunities.
Visionary Planner
  • Evaluate progress on the strategic plan, make course corrections, measure impact and scale effective approaches for using technology to transform learning.
Additional detail: ISTE author presentation

Proposal summary


Based on analysis of assessment data and the projection of various statistical models, learning loss - defined as the presence of a gap in learner performance - did occur across the United States during the COVID-19 pandemic (Bailey et al., 2021; Di Pietro et al., 2020; Engzell et al., 2021; Maldonado & De Witte, 2020; Schult et al., 2021). At the same time, individual learners exhibited progress during the pandemic, though some learning remained unfinished. According to The Education Trust (2021), unfinished learning considers learning to be incomplete but not necessarily lost. This shift changes the conversation from a focus on remediation and assessment to a focus on change management and equity.

The impact of the pandemic on individual learners’ academic growth thus depended on multiple, interdependent factors including a learner’s existing resources and supports, prior learning, and school/district readiness and response (CRPE, 2021). Since the publication of the 1966 Equality of Educational Opportunity report (Coleman et al., 1966), the influence of external factors and social context on learning has been well documented. Therefore, looking solely at academic factors to identify strategies to support learners, without considering surrounding social services, would not provide a complete picture.

Rather than focus primarily on in-school factors such as curriculum or instruction, this study adopted an ecological approach to identify the essential supports that may have contributed to learner growth and progress. This ecosystems perspective considers the interconnected social systems that directly and indirectly influence a learner’s development (Bronfenbrenner, 1979; Neal & Neal, 2013). With this theoretical perspective in mind, this study examined the presence of overlapping systems of support to shed light on strategies to accelerate unfinished learning while simultaneously supporting growth for all learners.


Given the exploratory nature of the study, the research team used an explanatory sequential mixed methods design (Creswell & Plano Clark, 2017). For the district included in this study, growth models were constructed from benchmark data, and descriptive statistics were used to detail the attributes of learner subpopulations based on district demographic data. While the quantitative data allowed for analysis of trends, qualitative focus groups and interviews offered further insights and presented rich descriptions of reality in context. After analyzing the quantitative and qualitative data separately, the results were mixed to complete the analysis and synthesis of findings (Creswell & Plano Clark, 2017).

To compare pre- and post-pandemic growth, as well as to deeply explore areas of concern presented by the district, the research team used the following criteria to identify the sample of learners to include in the initial study. With the exception of kindergarten, only learners who enrolled in the district during both the 2019-20 and 2020-21 school years were included. Based on 2020-21 data, analysis examined learners within the following grade ranges: 4-5, 6-8, and 10-12. These age groups were chosen because they had equivalent assessment data for both school years.

Because no prior data existed for kindergarten, a separate analysis focused on identified learner subpopulations as well as those who had previously enrolled in transitional kindergarten (TK). The district asked that ninth grade also be analyzed separately for two reasons: learners took different assessments in each school year, and they transitioned from separate K-8 schools into high school while still remote.

Quantitative Analysis
Using benchmark data in reading and math from the 2019-20 and 2020-21 school years, the research team modeled learner growth by age group and subpopulation - learners classified as English Learner, Special Education, Migrant, Homeless, or receiving Free or Reduced-Price Meals (a proxy for determining family income). The research team also examined differences in growth based on those who returned to in-person instruction, those who received additional services associated with a Healthy Start program for households with children under the age of five, and learners with overlapping needs (e.g. qualifying for Migrant, English Learner, and Special Education supports).

During the 2019-2020 school year, the district measured learner progress in reading via the Scholastic Reading Inventory (SRI). A criterion-referenced test, the SRI measures reading using the LEXILE Framework® for Reading (Scholastic Inc., 2020). Because expected annual growth in SRI reading scores is higher in elementary than middle or upper grade levels (National Center for Education Statistics [NCES], 2015), the analysis grouped learners into three grade ranges: elementary (grades 4-5), middle (grades 6-8), and secondary (grades 10-12).

Though learners normally complete SRI assessments four times each school year, due to COVID closure, an end-of-year score did not exist for the 2019-20 school year. To predict where learners might have ended, the researchers used historical data from the 2016-17, 2017-18, and the 2018-19 school years to calculate an average historical growth rate for each age range. Those growth rates were then applied to determine a predicted end of year score.

For the 2020-21 school year, the district shifted from the SRI to either Curriculum Associates’ iReady Assessment (grades 4-8) or the NWEA MAP assessment (grades 9-12) to measure learners' progress in reading. Although different assessment measures, they both used the LEXILE Framework®. However, learners take iReady and MAP assessments three times per year, rather than four, eliminating one of the data points in the 2020-21 school year.

To measure growth in math, the district used iReady with K-8 learners and NWEA MAP with 9-12 learners. Both of these assessments provide a composite score at three points in time. Because of COVID closure, the district only collected two data points during the 2019-20 school year. The research team assumed linear growth to predict a third point for the 2019-20 school year because historical data was not available.

Qualitative Analysis
To better understand the trends and patterns observed in the data, a focus group was conducted with school counselors from both K-8 schools and at the secondary level. Questions solicited additional information about the amount of outreach that occurred in each school, how learners responded to remote learning, factors that may have influenced learners’ participation in assessments, and which supports appeared most beneficial to learner growth.

To analyze the qualitative data, the research team coded the focus group transcript and research team notes to identify emergent themes (Saldaña, 2015). After coding, emergent themes were organized by research questions and used to supplement the quantitative data.


The research team identified several findings of note:
Learners in the studied district exhibited more growth in reading than their national peers (when compared to schools with similar demographics). The differences in progress were striking and certainly a testament to the efforts of the district to ensure that learners continued to grow during distance learning.
Although national trends showed greater loss for younger learners, in the district included in this study, elementary and middle grade learners demonstrated more progress in both reading and math. Notably, despite making steady progress prior to COVID-19 closure while still in a K-8 learning environment, learners in 9th grade experienced a drop during the subsequent school year as they entered high school. This trend mirrored that of other secondary learners (grades 10-12) who also experienced a decline.
District programs and strategies for supporting learners benefited multiple subpopulations. The district included in this study ensured that learners received regular contact, consistent live instruction, as well as additional materials and services to meet specific needs. As a result, while it may have been expected that learners classified as English Learner, Migrant, or Homeless would have experienced consistent decline, within the district included in this study, these learners generally made progress during the 2020-21 school year.
Instructional setting had varying effects on learner growth. At the elementary and middle levels, those in the early-return cohort model demonstrated more growth compared to their peers who remained in distance learning or returned to regular in-person instruction. In contrast, joining the early-return cohort did not have an effect on secondary learners’ growth, particularly in reading.
Learner growth varied across learning communities. Overall, the district included in this study has six TK-8 learning communities, one 9-12 high school, and one Alternative Education community (which consisted of three small 9-12 learning communities during the 2020-21 school year). Each learning community varied in size, learner demographics, and faculty composition. Given the myriad challenges facing older learners, and despite the enabling systems and structures implemented by the district, substantially less progress could be detected at the secondary level. The average scores were consistently lower for Alternative Education when compared to the traditional high school. However, it is important to note that these two learning communities had vastly different sample sizes and learner populations.


Most of the existing research on the effects of the COVID-19 pandemic on learner growth has taken a deficit-based approach (Harry & Klinger, 2007) and examined what learning was lost rather than what progress learners achieved despite challenging circumstances. Recommendations from these studies, though extremely valid, have largely focused on academic interventions such as individual and small-group instruction to both scaffold and accelerate learning (Allensworth & Schwartz, 2020; Kuhfeld et al., 2020; Nordengren & Jensen, 2020; Myung et al., 2020; Oregon Department of Education, 2021; Steiner, 2020), high-dosage tutoring where learners work with the same tutor over an extended period of time on academic skills (Sass & Goldring, 2021), expanded learning time (Patrick et al., 2021), as well as continued exposure to grade-level materials (Council of the Great City Schools, 2020; Dorn et al., 2020b). While necessary, these programs do not account for the myriad ecological factors that influence learners’ capacity to engage with school.

This study serves as a model for future research to help to identify the ecological supports that can accelerate learning for all. Though this study focused on one district, the intent is to apply the research design from this study to multiple districts across the country to better understand how learning has varied for different groups of learners. By undertaking a cross-district analysis, the research team aims to uncover a range of strategies that can be transferred to different contexts as education leaders seek to equitably support learners both as the COVID-19 pandemic wanes and in the years to come.


Agostinelli, F., Doepke, M., Sorrenti G., & Zilibotti, F. (2020). When the great equalizer shuts down: Schools, peers, and parents in pandemic times. National Bureau of Economic Research. doi:10.3386/w28264
Ali, T., Chandra, S., Cherukumilli, S., Fazlullah, A., Galicia, E., Hill, H., McAlpine, N., McBride, L., Vaduganathan, N., Weiss, D., Wu, M. (2021). Looking back, looking forward: What it will take to permanently close the K–12 digital divide. Common Sense Media.
Allensworth, E. & Schwartz, N. (2020). School practices to address student learning loss. EdResearch for Recovery Project.
Angrist, N., de Barros, A., Bhula, R., Chakera, S., Cummiskey, C., DeStefano, J., Floretta, J., Kaffenberger, M., Piper, B., & Stern., J. (2021). Building back better to avert a learning catastrophe: Estimating learning loss from COVID-19 school shutdowns in Africa and facilitating short-term and long-term learning recovery. International Journal of Educational Development, 84.
Bacher-Hicks, A. & Goodman, J. (2021). The Covid-19 pandemic is a lousy natural experiment for studying the effects of online learning​​. Education Next, 21(3).
Bailey, D. H., Duncan, G. J., Murnane, R. J., & Au Yeung, N. (2021). Achievement gaps in the wake of COVID-19. Educational Researcher, 50(5), 266-275.
Bang, M., Bricker, L. Darling-Hammond, L., Edgerton, A. K., Grossman, P., Gutierrez, K. D., Ishimaru, A., Klevan, S., Lee, C. D., Miyashiro, S., Nasir, N. S., Noguera, P. A., Payne, C., Penuel, B., Plasencia, S., Vossoughi, S. (2021). Summer learning and beyond: Opportunities for creating equity. Learning Policy Institute.
Benner, A. D. & Mistry, R. S. (2020). Child development during the COVID‐19 pandemic through a life course theory lens. Child Development Perspectives, 14, 236-243. doi:10.1111/cdep.12387
Borman, G. D. (2020). What can be done to address learning losses due to school closures? The Answer Lab at USC Rossier.
Bronfenbrenner, U. (1979). The ecology of human development. Harvard University Press.
CAST, Inc. (2021). About Universal Design for Learning.
Chang-Bacon, C. K. (2021). Generation interrupted: Rethinking “Students with Interrupted Formal Education” (SIFE) in the wake of a pandemic. Educational Researcher, 50(3), 187-196.
Charania, M. (2021, June 10). Assessments don’t always translate outside the classroom, but students’ networks do. Smart Brief.
Chiefs for Change. (2020). The return: How should educational leaders prepare for reentry and beyond?
Coleman, J. (1966). Equality of education opportunity. (Report No. 0E-36001). National Center for Education Statistics.
Council of the Great City Schools. (2020). Addressing unfinished learning after COVID-19 school closures.
Creswell, J. W. & Plano Clark, V. L. (2017). Designing and conducting mixed methods research. Sage.
Center on Reinventing Public Education. (2021). School district responses to COVID-19 closures.
Curriculum Associates. (2021). What we’ve learned about unfinished learning: Insights from midyear diagnostic assessments.
Darling-Hammond, L., Cantor, P., Hernández, L. E., Schachner, A., Plasencia, S., Theokas, C., Tijerina, E. (2021). Design principles for schools: Putting the science of learning and development into action. Learning Policy Institute.
Darling-Hammond, L., Flook, L., Cook-Harvey, C., Barron, B., & Osher, D. (2020). Implications for educational practice of the science of learning and development. Applied Developmental Science, 24(2), 97-140.
Defeyter, M. A., von Hippel, P., Shinwell, J., Mann, E., Henderson, E., Brownlee, I., Pepper, G. Stretesky, P., Long, M., McKenna, J., Daly-Smith, A., Lalli, G., Bundy, D., Drake, L. (2020) Covid-19: Back to school, rebuilding a better future for all children. UK Education Committee.
Di Pietro, G., Biagi, F., Costa, P., Karpiński Z., & Mazza, J. (2020). The likely impact of COVID-19 on education: Reflections based on the existing literature and international datasets, EUR 30275 EN, Publications Office of the European Union. doi:10.2760/126686, JRC121071
Dorn, E., Hancock, B., Sarakatsannis, J., & Viruleg, E. (2021). COVID-19 and education: The lingering effects of unfinished learning. McKinsey & Company.
Dorn, E., Hancock, B., Sarakatsannis, J., & Viruleg, E. (2020a). COVID-19 and student learning in the United States: The hurt could last a lifetime. McKinsey & Company.
Dorn, E., Hancock, B., Sarakatsannis, J., & Viruleg, E. (2020b). COVID-19 and learning loss—disparities grow and students need help. McKinsey & Company.
Education Endowment Foundation. (n.d.). Parental engagement.
Education Week. (2020, May 15). Map: Coronavirus and school closures.
Ellis, S. (2019, August 28). What is mastery learning? Getting Smart.
Engzell, P., Frey, A., & Verhagen, M. D. (2021). Learning loss due to school closures during the COVID-19 pandemic. Proceedings of the National Academy of Sciences, 118(17). doi:10.1073/pnas.2022376118
Eyles, A., Gibbons, S., & Montebruno, P. (2020). Covid-19 school shutdowns: What will they do to our children’s education? Centre for Economic Performance.
Gore, J., Fray, L., Miller, A., Harris, J., Taggart, W. (2021). The impact of COVID‐19 on student learning in New South Wales primary schools: An empirical study. The Australian Educational Researcher, 48.
Hamilton, L., Gross, B., Adams, D., Bradshaw, C. P., Cantor, P., Gurwitch, R., Jagers, R., Murry, V. M., Wong, M. (2021). How has the pandemic affected students’ social-emotional well-being? A review of the evidence to date. The Center on Reinventing Public Education.
Hanushek, E. A. & Woessman, L. (2020). The economic impacts of learning losses: OECD Education Working Papers No. 225. Organisation for Economic Cooperation and Development.
Hanushek, E. A., Peterson, P. E., Talpey, L. M., & Woessman, L. (2019). The achievement gap fails to close. Education Next, 19(3).
Harry, B. & Klingner, J. (2007). Discarding the deficit model. Educational Leadership, 64(5), 16-21.
Javurek, A. & Mendenhall, J. (2020). How a crisis can transform learning, teaching, and assessment. State Education Standard, 20(3).
Kaffenberger, M. (2021). Modelling the long-run learning impact of the Covid-19 learning shock: Actions to (more than) mitigate loss. International Journal of Educational Development, 81,
Katz, V. & Rideout, V. (2021). Learning at home while under-connected: Lower-income families during the COVID-19 pandemic. New America.
Kennedy, K. (2020). Learning loss, trauma, and our window of tolerance. Tech & Learning.
Kuhfeld, M., Soland, J., Tarasawa, B., Johnson, A., Ruzek, E., & Liu, J. (2020). Projecting the potential impact of COVID-19 school closures on academic achievement. Educational Researcher, 49(8), 549-565. doi:10.3102/0013189X20965918
Ladson-Billings, G. (1995). Toward a theory of culturally relevant pedagogy. AmericanEducational Research Journal, 32, 465-491.
Ladson-Billings, G. (2021). I’m here for the hard re-set: Post pandemic pedagogy to preserve our culture. Equity & Excellence in Education, 54(1), 68-78, doi:10.1080/10665684.2020.1863883
Lake, R. & Olson, L. Learning as we go: Principles for effective assessment during the COVID-19 pandemic. Center on Reinventing Public Education.
Lewis, K., Kuhfeld, M., Ruzek, E., & McEachin, A. (2021). Learning during COVID-19: Reading and math achievement in the 2020-2021 school year. Center for School and Student Progress.
Lynch, K. & Hill, H. (2020). Broad-based academic supports for all students: Brief No. 6. EdResearch for Recovery.
Maldonado, J. & De Witte, K. (2020). The effect of school closures on standardised student test outcomes. FEB Research Report Department of Economics.
Muhayimana, T. (2020). Teacher professional learning during the global pandemic: Five critical areas to address. Journal of School Administration Research and Development, 5(S2), 66-79.
Myung, J., Gallagher, A., Cottingham, B., Gong, A., Kimner, H., Witte, J., Gee, J., & Hough, H. (2020). Supporting learning in the COVID-19 context: Research to guide distance and blended instruction. Policy Analysis for California Education, PACE.
Neal, J. W. & Neal, Z. P. (2013). Nested or networked? Future directions for ecological systems theory. Social Development, 22(4), 722–737. doi:10.1111/sode.12018
National Center for Education Statistics [NCES]. (2015). The condition of education: Reading and mathematics score trends.
Nordengren, C. & Jensen, N. (2020). Turning a new page: Guiding instruction and using MAP growth amid COVID-19 school closures. NWEA. ​​
Oregon Department of Education. (2021). Student learning: Unfinished, not lost. An asset-based frame that humanizes learning & supports action.,%20Not%20Lost.pdf
Owen, D. & Vista, A. (2017). Strategies for teaching metacognition in classrooms. Brookings.
Pan, W. & Sass, T. (2020). Potential remediation strategies in the wake of COVID-19 school closures: A review of the literature. Georgia Policy Labs.
Raymond, M. E., Han, C., Goulas, S., Lee, W. F., & Waeiss, C. (2020). Estimates of learning loss in the 2019-2020 school year. The Center for Research on Education Outcomes at Stanford University.
Saldaña, J. (2015). The coding manual for qualitative researchers. Sage.
Sass, T. & Goldring, T. (2021). Student achievement growth during the COVID-19 pandemic: Insights from Metro Atlanta school districts. Georgia Policy Labs.
Schult, J., Mahler, N., Fauth, B., & Lindner, M. (2021). Did students learn less during the COVID-19 pandemic? Reading and mathematics competencies before and after the first pandemic wave. PsyArXiv. doi:10.31234/
Scholastic Inc. (2020). Reading and assessment overview. Scholastic Reading Inventory.
Steiner, D. (2020). Don’t remediate, accelerate! Effective catch-up learning strategies – evidence from the United States. UNESCO.
The Education Trust. (2021). Strategies to solve unfinished learning.
Tomasik, M. J., Helbling, L. A., & Moser, U. (2020). Educational gains of in‐person vs. distance learning in primary and secondary schools: A natural experiment during the COVID‐19 pandemic school closures in Switzerland. International Journal of Psychology, 56(4), 566-576.
Tracy, S. J. (2010). Qualitative quality: Eight “big-tent” criteria for excellent qualitative research. Qualitative Inquiry 16(10), 837-851. doi:10.1177/1077800410383121
U.S. Department of Education Office of Civil Rights. (2021). Education in a pandemic: The disparate impacts of COVID-19 on America’s students.
Zhao, Y. (2021). Build back better: Avoid the learning loss trap. Prospects. ​​

More [+]


Dr. Caitlin McLemore, Blank Crayon LLC
ISTE Certified Educator

Caitlin is an educational technology consultant who specializes in curriculum and instructional design, educational research, and professional learning. She holds an Ed.D. from Johns Hopkins School and a B.A./M.Ed. from the University of Florida (Go Gators). Previously, Caitlin worked as an academic technology specialist and librarian at several independent schools. Caitlin is a Google Certified Innovator, Trainer, and ISTE Certified Educator. She was named an ISTE Emerging Leader in 2017 and the 2018 ISTE Outstanding Young Educator. Caitlin is the co-author of the ISTE-published book Stretch Yourself: A Personalized Journey to Deepen Your Teaching Practice.

Dr. Beth Holland, The Learning Accelerator

Beth Holland is a Partner at The Learning Accelerator. where she leads their Research & Measurement work. Over the past 20+ years, she has taught in K-12 classrooms, served as a Director of Academic Technology, designed professional learning programs and leaders, as well as conducted research evaluations. Beth holds an Education Doctorate (EdD) from Johns Hopkins University, a Master's degree (EdM) from the Harvard Graduate School of Education, and a Bachelor of Science (B.S.) degree from Northwestern University.

People also viewed

Antiracist Education in an Asynchronous Online Class: A Learning Module
Chromebooks Can Do That?!
TeachMeet ISTE