Learning Loss to Unfinished Learning: Mixed-Methods Research Identifying Accelerated Learning Strategies

Framework

Based on analysis of assessment data and the projection of various statistical models, learning loss - defined as the presence of a gap in learner performance - did occur across the United States during the COVID-19 pandemic (Bailey et al., 2021; Di Pietro et al., 2020; Engzell et al., 2021; Maldonado & De Witte, 2020; Schult et al., 2021). At the same time, individual learners exhibited progress during the pandemic, though some learning remained unfinished. According to The Education Trust (2021), unfinished learning considers learning to be incomplete but not necessarily lost. This shift changes the conversation from a focus on remediation and assessment to a focus on change management and equity.

The impact of the pandemic on individual learners’ academic growth thus depended on multiple, interdependent factors including a learner’s existing resources and supports, prior learning, and school/district readiness and response (CRPE, 2021). Since the publication of the 1966 Equality of Educational Opportunity report (Coleman et al., 1966), the influence of external factors and social context on learning has been well documented. Therefore, looking solely at academic factors to identify strategies to support learners, without considering surrounding social services, would not provide a complete picture.

Rather than focus primarily on in-school factors such as curriculum or instruction, this study adopted an ecological approach to identify the essential supports that may have contributed to learner growth and progress. This ecosystems perspective considers the interconnected social systems that directly and indirectly influence a learner’s development (Bronfenbrenner, 1979; Neal & Neal, 2013). With this theoretical perspective in mind, this study examined the presence of overlapping systems of support to shed light on strategies to accelerate unfinished learning while simultaneously supporting growth for all learners.

Methods

Given the exploratory nature of the study, the research team used an explanatory sequential mixed methods design (Creswell & Plano Clark, 2017). For the district included in this study, growth models were constructed from benchmark data, and descriptive statistics were used to detail the attributes of learner subpopulations based on district demographic data. While the quantitative data allowed for analysis of trends, qualitative focus groups and interviews offered further insights and presented rich descriptions of reality in context. After analyzing the quantitative and qualitative data separately, the results were mixed to complete the analysis and synthesis of findings (Creswell & Plano Clark, 2017).

Sample
To compare pre- and post-pandemic growth, as well as to deeply explore areas of concern presented by the district, the research team used the following criteria to identify the sample of learners to include in the initial study. With the exception of kindergarten, only learners who enrolled in the district during both the 2019-20 and 2020-21 school years were included. Based on 2020-21 data, analysis examined learners within the following grade ranges: 4-5, 6-8, and 10-12. These age groups were chosen because they had equivalent assessment data for both school years.

Because no prior data existed for kindergarten, a separate analysis focused on identified learner subpopulations as well as those who had previously enrolled in transitional kindergarten (TK). The district asked that ninth grade also be analyzed separately for two reasons: learners took different assessments in each school year, and they transitioned from separate K-8 schools into high school while still remote.

Quantitative Analysis
Using benchmark data in reading and math from the 2019-20 and 2020-21 school years, the research team modeled learner growth by age group and subpopulation - learners classified as English Learner, Special Education, Migrant, Homeless, or receiving Free or Reduced-Price Meals (a proxy for determining family income). The research team also examined differences in growth based on those who returned to in-person instruction, those who received additional services associated with a Healthy Start program for households with children under the age of five, and learners with overlapping needs (e.g. qualifying for Migrant, English Learner, and Special Education supports).

During the 2019-2020 school year, the district measured learner progress in reading via the Scholastic Reading Inventory (SRI). A criterion-referenced test, the SRI measures reading using the LEXILE Framework® for Reading (Scholastic Inc., 2020). Because expected annual growth in SRI reading scores is higher in elementary than middle or upper grade levels (National Center for Education Statistics [NCES], 2015), the analysis grouped learners into three grade ranges: elementary (grades 4-5), middle (grades 6-8), and secondary (grades 10-12).

Though learners normally complete SRI assessments four times each school year, due to COVID closure, an end-of-year score did not exist for the 2019-20 school year. To predict where learners might have ended, the researchers used historical data from the 2016-17, 2017-18, and the 2018-19 school years to calculate an average historical growth rate for each age range. Those growth rates were then applied to determine a predicted end of year score.

For the 2020-21 school year, the district shifted from the SRI to either Curriculum Associates’ iReady Assessment (grades 4-8) or the NWEA MAP assessment (grades 9-12) to measure learners' progress in reading. Although different assessment measures, they both used the LEXILE Framework®. However, learners take iReady and MAP assessments three times per year, rather than four, eliminating one of the data points in the 2020-21 school year.

To measure growth in math, the district used iReady with K-8 learners and NWEA MAP with 9-12 learners. Both of these assessments provide a composite score at three points in time. Because of COVID closure, the district only collected two data points during the 2019-20 school year. The research team assumed linear growth to predict a third point for the 2019-20 school year because historical data was not available.

Qualitative Analysis
To better understand the trends and patterns observed in the data, a focus group was conducted with school counselors from both K-8 schools and at the secondary level. Questions solicited additional information about the amount of outreach that occurred in each school, how learners responded to remote learning, factors that may have influenced learners’ participation in assessments, and which supports appeared most beneficial to learner growth.

To analyze the qualitative data, the research team coded the focus group transcript and research team notes to identify emergent themes (Saldaña, 2015). After coding, emergent themes were organized by research questions and used to supplement the quantitative data.

Results

The research team identified several findings of note:
Learners in the studied district exhibited more growth in reading than their national peers (when compared to schools with similar demographics). The differences in progress were striking and certainly a testament to the efforts of the district to ensure that learners continued to grow during distance learning.
Although national trends showed greater loss for younger learners, in the district included in this study, elementary and middle grade learners demonstrated more progress in both reading and math. Notably, despite making steady progress prior to COVID-19 closure while still in a K-8 learning environment, learners in 9th grade experienced a drop during the subsequent school year as they entered high school. This trend mirrored that of other secondary learners (grades 10-12) who also experienced a decline.
District programs and strategies for supporting learners benefited multiple subpopulations. The district included in this study ensured that learners received regular contact, consistent live instruction, as well as additional materials and services to meet specific needs. As a result, while it may have been expected that learners classified as English Learner, Migrant, or Homeless would have experienced consistent decline, within the district included in this study, these learners generally made progress during the 2020-21 school year.
Instructional setting had varying effects on learner growth. At the elementary and middle levels, those in the early-return cohort model demonstrated more growth compared to their peers who remained in distance learning or returned to regular in-person instruction. In contrast, joining the early-return cohort did not have an effect on secondary learners’ growth, particularly in reading.
Learner growth varied across learning communities. Overall, the district included in this study has six TK-8 learning communities, one 9-12 high school, and one Alternative Education community (which consisted of three small 9-12 learning communities during the 2020-21 school year). Each learning community varied in size, learner demographics, and faculty composition. Given the myriad challenges facing older learners, and despite the enabling systems and structures implemented by the district, substantially less progress could be detected at the secondary level. The average scores were consistently lower for Alternative Education when compared to the traditional high school. However, it is important to note that these two learning communities had vastly different sample sizes and learner populations.

Importance

Most of the existing research on the effects of the COVID-19 pandemic on learner growth has taken a deficit-based approach (Harry & Klinger, 2007) and examined what learning was lost rather than what progress learners achieved despite challenging circumstances. Recommendations from these studies, though extremely valid, have largely focused on academic interventions such as individual and small-group instruction to both scaffold and accelerate learning (Allensworth & Schwartz, 2020; Kuhfeld et al., 2020; Nordengren & Jensen, 2020; Myung et al., 2020; Oregon Department of Education, 2021; Steiner, 2020), high-dosage tutoring where learners work with the same tutor over an extended period of time on academic skills (Sass & Goldring, 2021), expanded learning time (Patrick et al., 2021), as well as continued exposure to grade-level materials (Council of the Great City Schools, 2020; Dorn et al., 2020b). While necessary, these programs do not account for the myriad ecological factors that influence learners’ capacity to engage with school.

This study serves as a model for future research to help to identify the ecological supports that can accelerate learning for all. Though this study focused on one district, the intent is to apply the research design from this study to multiple districts across the country to better understand how learning has varied for different groups of learners. By undertaking a cross-district analysis, the research team aims to uncover a range of strategies that can be transferred to different contexts as education leaders seek to equitably support learners both as the COVID-19 pandemic wanes and in the years to come.

References

https://bit.ly/LUSD-ULReferences

Agostinelli, F., Doepke, M., Sorrenti G., & Zilibotti, F. (2020). When the great equalizer shuts down: Schools, peers, and parents in pandemic times. National Bureau of Economic Research. doi:10.3386/w28264
Ali, T., Chandra, S., Cherukumilli, S., Fazlullah, A., Galicia, E., Hill, H., McAlpine, N., McBride, L., Vaduganathan, N., Weiss, D., Wu, M. (2021). Looking back, looking forward: What it will take to permanently close the K–12 digital divide. Common Sense Media. https://www.commonsensemedia.org/sites/default/files/uploads/pdfs/final_-_what_it_will_take_to_permanently_close_the_k-12_digital_divide_vfeb3.pdf
Allensworth, E. & Schwartz, N. (2020). School practices to address student learning loss. EdResearch for Recovery Project. https://annenberg.brown.edu/sites/default/files/EdResearch_for_Recovery_Brief_1.pdf
Angrist, N., de Barros, A., Bhula, R., Chakera, S., Cummiskey, C., DeStefano, J., Floretta, J., Kaffenberger, M., Piper, B., & Stern., J. (2021). Building back better to avert a learning catastrophe: Estimating learning loss from COVID-19 school shutdowns in Africa and facilitating short-term and long-term learning recovery. International Journal of Educational Development, 84. https://doi.org/10.1016/j.ijedudev.2021.102397
Bacher-Hicks, A. & Goodman, J. (2021). The Covid-19 pandemic is a lousy natural experiment for studying the effects of online learning​​. Education Next, 21(3). https://www.educationnext.org/covid-19-pandemic-lousy-natural-experiment-for-studying-the-effects-online-learning/
Bailey, D. H., Duncan, G. J., Murnane, R. J., & Au Yeung, N. (2021). Achievement gaps in the wake of COVID-19. Educational Researcher, 50(5), 266-275. https://doi.org/10.3102%2F0013189X211011237
Bang, M., Bricker, L. Darling-Hammond, L., Edgerton, A. K., Grossman, P., Gutierrez, K. D., Ishimaru, A., Klevan, S., Lee, C. D., Miyashiro, S., Nasir, N. S., Noguera, P. A., Payne, C., Penuel, B., Plasencia, S., Vossoughi, S. (2021). Summer learning and beyond: Opportunities for creating equity. Learning Policy Institute. https://learningpolicyinstitute.org/product/summer-learning-creating-equity-report
Benner, A. D. & Mistry, R. S. (2020). Child development during the COVID‐19 pandemic through a life course theory lens. Child Development Perspectives, 14, 236-243. doi:10.1111/cdep.12387
Borman, G. D. (2020). What can be done to address learning losses due to school closures? The Answer Lab at USC Rossier. https://theanswerlab.rossier.usc.edu/
Bronfenbrenner, U. (1979). The ecology of human development. Harvard University Press.
CAST, Inc. (2021). About Universal Design for Learning. https://www.cast.org/impact/universal-design-for-learning-udl
Chang-Bacon, C. K. (2021). Generation interrupted: Rethinking “Students with Interrupted Formal Education” (SIFE) in the wake of a pandemic. Educational Researcher, 50(3), 187-196. https://doi.org/10.3102%2F0013189X21992368
Charania, M. (2021, June 10). Assessments don’t always translate outside the classroom, but students’ networks do. Smart Brief. https://www.smartbrief.com/original/2021/06/assessments-don%E2%80%99t-always-translate-outside-classroom-students%E2%80%99-networks
Chiefs for Change. (2020). The return: How should educational leaders prepare for reentry and beyond? https://chiefsforchange.org/wp-content/uploads/2020/06/CFC-TheReturn_5-13-20.pdf
Coleman, J. (1966). Equality of education opportunity. (Report No. 0E-36001). National Center for Education Statistics. https://files.eric.ed.gov/fulltext/ED012275.pdf
Council of the Great City Schools. (2020). Addressing unfinished learning after COVID-19 school closures. https://www.cgcs.org/cms/lib/DC00001581/Centricity/Domain/313/CGCS_Unfinished%20Learning.pdf
Creswell, J. W. & Plano Clark, V. L. (2017). Designing and conducting mixed methods research. Sage.
Center on Reinventing Public Education. (2021). School district responses to COVID-19 closures. https://www.crpe.org/current-research/covid-19-school-closures
Curriculum Associates. (2021). What we’ve learned about unfinished learning: Insights from midyear diagnostic assessments. https://www.curriculumassociates.com/-/media/mainsite/files/i-ready/iready-understanding-student-needs-paper-winter-results-2021.pdf
Darling-Hammond, L., Cantor, P., Hernández, L. E., Schachner, A., Plasencia, S., Theokas, C., Tijerina, E. (2021). Design principles for schools: Putting the science of learning and development into action. Learning Policy Institute. https://learningpolicyinstitute.org/product/sold-design-principles-report
Darling-Hammond, L., Flook, L., Cook-Harvey, C., Barron, B., & Osher, D. (2020). Implications for educational practice of the science of learning and development. Applied Developmental Science, 24(2), 97-140. https://doi.org/10.1080/10888691.2018.1537791
Defeyter, M. A., von Hippel, P., Shinwell, J., Mann, E., Henderson, E., Brownlee, I., Pepper, G. Stretesky, P., Long, M., McKenna, J., Daly-Smith, A., Lalli, G., Bundy, D., Drake, L. (2020) Covid-19: Back to school, rebuilding a better future for all children. UK Education Committee. http://hdl.handle.net/2436/623310
Di Pietro, G., Biagi, F., Costa, P., Karpiński Z., & Mazza, J. (2020). The likely impact of COVID-19 on education: Reflections based on the existing literature and international datasets, EUR 30275 EN, Publications Office of the European Union. doi:10.2760/126686, JRC121071
Dorn, E., Hancock, B., Sarakatsannis, J., & Viruleg, E. (2021). COVID-19 and education: The lingering effects of unfinished learning. McKinsey & Company. https://www.mckinsey.com/industries/public-and-social-sector/our-insights/covid-19-and-education-the-lingering-effects-of-unfinished-learning
Dorn, E., Hancock, B., Sarakatsannis, J., & Viruleg, E. (2020a). COVID-19 and student learning in the United States: The hurt could last a lifetime. McKinsey & Company. https://www.mckinsey.com/industries/public-and-social-sector/our-insights/covid-19-and-student-learning-in-the-united-states-the-hurt-could-last-a-lifetime
Dorn, E., Hancock, B., Sarakatsannis, J., & Viruleg, E. (2020b). COVID-19 and learning loss—disparities grow and students need help. McKinsey & Company. https://www.mckinsey.com/industries/public-and-social-sector/our-insights/covid-19-and-learning-loss-disparities-grow-and-students-need-help
Education Endowment Foundation. (n.d.). Parental engagement. https://educationendowmentfoundation.org.uk/school-themes/parental-engagement/
Education Week. (2020, May 15). Map: Coronavirus and school closures. https://www.edweek.org/ew/section/multimedia/map-coronavirus-and-school-closures.html
Ellis, S. (2019, August 28). What is mastery learning? Getting Smart. https://www.gettingsmart.com/2019/08/what-is-mastery-learning/
Engzell, P., Frey, A., & Verhagen, M. D. (2021). Learning loss due to school closures during the COVID-19 pandemic. Proceedings of the National Academy of Sciences, 118(17). doi:10.1073/pnas.2022376118
Eyles, A., Gibbons, S., & Montebruno, P. (2020). Covid-19 school shutdowns: What will they do to our children’s education? Centre for Economic Performance. http://eprints.lse.ac.uk/104675/3/Eyles_covid_19_school_shutdowns_published.pdf
Gore, J., Fray, L., Miller, A., Harris, J., Taggart, W. (2021). The impact of COVID‐19 on student learning in New South Wales primary schools: An empirical study. The Australian Educational Researcher, 48. https://doi.org/10.1007/s13384-021-00436-w
Hamilton, L., Gross, B., Adams, D., Bradshaw, C. P., Cantor, P., Gurwitch, R., Jagers, R., Murry, V. M., Wong, M. (2021). How has the pandemic affected students’ social-emotional well-being? A review of the evidence to date. The Center on Reinventing Public Education. https://www.crpe.org/sites/default/files/sel_report_2021_final.pdf
Hanushek, E. A. & Woessman, L. (2020). The economic impacts of learning losses: OECD Education Working Papers No. 225. Organisation for Economic Cooperation and Development. https://dx.doi.org/10.1787/21908d74-en
Hanushek, E. A., Peterson, P. E., Talpey, L. M., & Woessman, L. (2019). The achievement gap fails to close. Education Next, 19(3). https://www.educationnext.org/achievement-gap-fails-close-half-century-testing-shows-persistent-divide/
Harry, B. & Klingner, J. (2007). Discarding the deficit model. Educational Leadership, 64(5), 16-21. http://www.ask-force.org/web/Discourse/Harry-Discarding-Deficit-Model-2007.pdf
Javurek, A. & Mendenhall, J. (2020). How a crisis can transform learning, teaching, and assessment. State Education Standard, 20(3). https://www.nasbe.org/how-a-crisis-can-transform-learning-teaching-and-assessment/
Kaffenberger, M. (2021). Modelling the long-run learning impact of the Covid-19 learning shock: Actions to (more than) mitigate loss. International Journal of Educational Development, 81, https://doi.org/10.1016/j.ijedudev.2020.102326
Katz, V. & Rideout, V. (2021). Learning at home while under-connected: Lower-income families during the COVID-19 pandemic. New America. https://www.newamerica.org/education-policy/reports/learning-at-home-while-underconnected/
Kennedy, K. (2020). Learning loss, trauma, and our window of tolerance. Tech & Learning. https://www.techlearning.com/how-to/learning-loss-trauma-and-our-window-of-tolerance
Kuhfeld, M., Soland, J., Tarasawa, B., Johnson, A., Ruzek, E., & Liu, J. (2020). Projecting the potential impact of COVID-19 school closures on academic achievement. Educational Researcher, 49(8), 549-565. doi:10.3102/0013189X20965918
Ladson-Billings, G. (1995). Toward a theory of culturally relevant pedagogy. AmericanEducational Research Journal, 32, 465-491. https://doi.org/10.3102%2F00028312032003465
Ladson-Billings, G. (2021). I’m here for the hard re-set: Post pandemic pedagogy to preserve our culture. Equity & Excellence in Education, 54(1), 68-78, doi:10.1080/10665684.2020.1863883
Lake, R. & Olson, L. Learning as we go: Principles for effective assessment during the COVID-19 pandemic. Center on Reinventing Public Education. https://files.eric.ed.gov/fulltext/ED606373.pdf
Lewis, K., Kuhfeld, M., Ruzek, E., & McEachin, A. (2021). Learning during COVID-19: Reading and math achievement in the 2020-2021 school year. Center for School and Student Progress. https://www.nwea.org/content/uploads/2021/07/Learning-during-COVID-19-Reading-and-math-achievement-in-the-2020-2021-school-year.research-brief-1.pdf
Lynch, K. & Hill, H. (2020). Broad-based academic supports for all students: Brief No. 6. EdResearch for Recovery. https://annenberg.brown.edu/sites/default/files/EdResearch_for_Recovery_Brief_6.pdf
Maldonado, J. & De Witte, K. (2020). The effect of school closures on standardised student test outcomes. FEB Research Report Department of Economics. https://lirias.kuleuven.be/3189074?limo=0
Muhayimana, T. (2020). Teacher professional learning during the global pandemic: Five critical areas to address. Journal of School Administration Research and Development, 5(S2), 66-79. https://doi.org/10.32674/jsard.v5iS2.2840
Myung, J., Gallagher, A., Cottingham, B., Gong, A., Kimner, H., Witte, J., Gee, J., & Hough, H. (2020). Supporting learning in the COVID-19 context: Research to guide distance and blended instruction. Policy Analysis for California Education, PACE. https://files.eric.ed.gov/fulltext/ED609208.pdf
Neal, J. W. & Neal, Z. P. (2013). Nested or networked? Future directions for ecological systems theory. Social Development, 22(4), 722–737. doi:10.1111/sode.12018
National Center for Education Statistics [NCES]. (2015). The condition of education: Reading and mathematics score trends. https://nces.ed.gov/programs/coe/indicator_cnj.asp
Nordengren, C. & Jensen, N. (2020). Turning a new page: Guiding instruction and using MAP growth amid COVID-19 school closures. NWEA. ​​https://www.vassonline.org/images/vass/coronavirus-resources/NWEA-guiding-instruction-and-using-map-growth-amid-covid-19-school-closures.pdf
Oregon Department of Education. (2021). Student learning: Unfinished, not lost. An asset-based frame that humanizes learning & supports action. https://www.oregon.gov/ode/students-and-family/healthsafety/Documents/Student%20Learning%20Unfinished,%20Not%20Lost.pdf
Owen, D. & Vista, A. (2017). Strategies for teaching metacognition in classrooms. Brookings. https://www.brookings.edu/blog/education-plus-development/2017/11/15/strategies-for-teaching-metacognition-in-classrooms/
Pan, W. & Sass, T. (2020). Potential remediation strategies in the wake of COVID-19 school closures: A review of the literature. Georgia Policy Labs. https://scholarworks.gsu.edu/gpl_reports/4/
Raymond, M. E., Han, C., Goulas, S., Lee, W. F., & Waeiss, C. (2020). Estimates of learning loss in the 2019-2020 school year. The Center for Research on Education Outcomes at Stanford University. https://credo.stanford.edu/sites/g/files/sbiybj6481/f/short_brief_on_learning_loss_final_v.3.pdf
Saldaña, J. (2015). The coding manual for qualitative researchers. Sage.
Sass, T. & Goldring, T. (2021). Student achievement growth during the COVID-19 pandemic: Insights from Metro Atlanta school districts. Georgia Policy Labs. https://scholarworks.gsu.edu/cgi/viewcontent.cgi?article=1008&context=gpl_reports
Schult, J., Mahler, N., Fauth, B., & Lindner, M. (2021). Did students learn less during the COVID-19 pandemic? Reading and mathematics competencies before and after the first pandemic wave. PsyArXiv. doi:10.31234/osf.io/pqtgf
Scholastic Inc. (2020). Reading and assessment overview. Scholastic Reading Inventory. http://teacher.scholastic.com/products/sri_reading_assessment/programoverview.htm
Steiner, D. (2020). Don’t remediate, accelerate! Effective catch-up learning strategies – evidence from the United States. UNESCO. https://unesdoc.unesco.org/ark:/48223/pf0000374029
The Education Trust. (2021). Strategies to solve unfinished learning. https://edtrust.org/strategies-to-solve-unfinished-learning/
Tomasik, M. J., Helbling, L. A., & Moser, U. (2020). Educational gains of in‐person vs. distance learning in primary and secondary schools: A natural experiment during the COVID‐19 pandemic school closures in Switzerland. International Journal of Psychology, 56(4), 566-576. https://doi.org/10.1002/ijop.12728
Tracy, S. J. (2010). Qualitative quality: Eight “big-tent” criteria for excellent qualitative research. Qualitative Inquiry 16(10), 837-851. doi:10.1177/1077800410383121
U.S. Department of Education Office of Civil Rights. (2021). Education in a pandemic: The disparate impacts of COVID-19 on America’s students. https://www2.ed.gov/about/offices/list/ocr/docs/20210608-impacts-of-covid19.pdf
Zhao, Y. (2021). Build back better: Avoid the learning loss trap. Prospects. ​​https://doi.org/10.1007/s11125-021-09544-y