Assessing Classroom Technology Use for 21st Century Skills: A Research-Based Rubric
Listen and learn : Research paper
Wednesday, December 2, 2:15–3:00 pm PST (Pacific Standard Time)
Presentation 1 of 2
Virtual Reality and Learners' Self-perception of Success
Jason Ravitz Dr. Mahsa Bakhshaei John Seylar
To help teachers in a nationwide coaching program reflect on their progress in technology use, researchers developed an Impactful Technology Use (ITU) Rubric and associated survey questions. We'll share the development process, the final version of the rubric, its reliability and suggestions for its use.
|Audience:||Coaches, Teachers, Principals/head teachers|
|Attendee devices:||Devices not needed|
|Topic:||Assessment/evaluations/use of data|
|ISTE Standards:||For Educators:
Professional Development and Program Evaluation
It is well documented that K-12 instruction that prepares students to navigate in a global society should foster 21st century skills, including the “4Cs”, communication, collaboration, creativity and critical thinking (NRC, 2013). It is also well documented that technology can enhance teaching and learning (US Department of Education, 2017). Previous studies provided resources that help advance students’ 21st century competencies in classroom practices (NEA, 2012; Hixson, Ravitz, & Whisman, 2012). However, there is still a need for high-quality tools that help educators purposefully use technology to develop 21st century skills through learning experiences. The existing tools for supporting meaningful technology use are neither fully aligned with 21st century skills (e.g., SAMR and Technology Integration Matrix) nor do they provide educators with a framework that guides reflection on growth in technology integration over time (Budhai & Taddei, 2015).
We addressed this gap by conceptualizing and designing our ITU Rubric that focuses on technology uses for developing students' 21st century skills. It provides educators with a tool for professional growth and helps them reflect on their progress in using technology to develop students' 21st century skills.
In this study, we investigate four research questions to establish the usefulness of the Rubric in helping educators improve impactful technology use in their practices:
1. What are teacher self-ratings of their ability to engage students in impactful technology use?
2. How frequently do students use technology in impactful ways in their classrooms? Through which specific classroom practices?
3. To what extent do teachers believe these students’ technology uses have a positive impact on their engagement and learning?
4. How do teacher self-ratings of their own capabilities in teaching each skill and their reports of student technology use correspond to their perception of impact on student engagement and learning?
Following West Virginia’s 21st Century Survey (Hixson, Ravitz, & Whisman, 2012), our ITU Framework was born in August 2017. It included five core areas of 21st century skills from the West Virginia survey that showed very strong reliability: Critical thinking, collaboration, communication, creativity, and agency. Definitions were drawn from a general body of research, but expanded to include an explicit link to technology use (NEA, 2012; Framework of the 21st century skills, 2012). We also conceptualized a sixth indicator focused on the selection of relevant technology tools. These six indicators together formed the ITU Framework (Figure 1).
Figure 1. ITU Framework
This framework was employed in our surveys, asking teachers the extent to which they agreed they had the ability to engage their students in developing each of the skills (Figure 2).
Figure 2. Sample survey question on teacher confidence in selecting and using technology in impactful ways
While this framework helped educators conceptualize ITU, it did not provide them with a coherent set of criteria to formatively reflect on their progress in ITU. Therefore, a team of educational practitioners and researchers worked together to develop the ITU Rubric (Figure 3). This Rubric defined five descriptors of development for each of the six indicators (‘ready to use’, ‘emerging’, ‘developing’, ‘mastering’ and ‘innovating’) with one or two illustrative — but non-exhaustive — examples at each level.
Figure 3. The first version of ITU Rubric
User testing among teachers and coaches suggested this version could be improved to provide clearer dimensions. While the descriptors accompanied by illustrative practices helped them understand what ITU could look like at different levels of performance, these example practices could be misinterpreted as being required rather than just illustrative, and made the Rubric feel text-heavy and difficult to digest across different grades or subjects. In addition, we realized this version presented more advanced levels of development as the addition of more practices, without capturing the frequency or differences in these activities. For example, having students involved in explaining concepts to peers, providing feedback to peers, and completing group assignments are three ways of using technology for collaboration that may or may not represent a “mastering” level of student proficiency or positive impact on teaching and learning.
The above insights resulted in an updated version of the Rubric that includes dimensions of both frequency and proficiency (Figure 4). The frequency rating concerns how often students engage in ITU activities. The proficiency rating concerns teacher perception of student ITU proficiency. In this version, instead of providing example practices for each proficiency level, we provide illustrative examples for the indicator as a whole as well as a description of student use at the lowest and highest levels of proficiency.
Among coached teachers, we measure these two dimensions in teacher surveys at the beginning and end of each eight-week coaching cycle, then share data reports to help teachers and coaches reflect on their progress in using technology to develop student 21st century skills. These coaching experiences and ratings of student proficiency are kept between teachers and coaches and are not used in our research.
Figure 4. The last version of ITU Rubric
In our research, we use a pre- and post- survey that is administered annually to all teachers, regardless of whether they received coaching or not, to measure:
1. Teachers’ self-rating ability to engage students in ITU (question presented in Figure 2)
2. Frequency of student ITU (Figure 5)
3. Impact of ITU on student engagement and learning (Figure 5).
We present teacher responses (N for coached teachers = 1,546; N for non-coached teachers = 1,162) from our year 2 post-survey to show the extent to which these three sets of questions are strong measures to assess teacher and student ITU. Including teachers not participating in our coaching program gives us a chance to see whether these rubric-based items are “sensitive to treatment” or able to pick up distinctions in teachers’ experiences, practices and perceptions.
Figure 5. Sample survey question on impact of ITU on student engagement and learning
Teacher self-rating ability to engage students in ITU
As shown in the following chart, a majority of teachers agreed they were able to use technology in impactful ways with students. This was especially true for teachers who received coaching.
Chart 1. Teacher self-rating of ability to engage students in ITU
Frequency of student ITU
Teacher confidence in using technology in impactful ways was reflected in frequent impactful technology use by their students (Chart 2). At least 35% of teachers reported impactful technology use by their students in each skill area at least monthly. This rises to nearly half of the teachers who received coaching. The most frequent monthly impactful technology use involved students using technology to collaborate in pairs or small groups. This was followed by students deciding what tools or resources to use. Students less frequently used technology for exploring ideas in a way other than writing, creating something unique for the class or deciding what activities will help them learn.
Chart 2. Frequency of monthly ITU, examples from each skill area
Perceived impact of ITU on student engagement and learning
As shown in the chart below, a majority of teachers reported a “positive impact on student engagement and learning” from student ITU, with an average of 62% of coached teachers reporting a positive impact for each use compared to 52% of non-coached teachers. The uses of technology that teachers reported had the greatest impact on student engagement and learning involved student development of collaboration skills, creativity skills, ability to select relevant technology, and communication skills.
Chart 3. Percent of teachers reporting positive impact on student engagement and learning
Predicting perceived impact of ITU on student engagement and learning
As Table 1 shows, the perceived impact of ITU on student engagement and learning is correlated to teacher self-rating of ability to engage students in ITU and to frequency of student ITU to practice these skills. However, the stronger correlation was to frequency of student ITU index scores. For example, for impact of ITU creativity, critical thinking and agency the correlation to frequency of ITU practices was .55 or higher, while the correlation to self ratings was .40 or lower.
Table 1. Correlation between impact of ITU on students’ engagement and learning and teacher self-rating of ability in ITU and frequency of student ITU
This suggests that teacher reports of the frequency of actual activities occurring in the classroom can be a better predictor of the impact of ITU on student engagement and learning, compared to teacher self-report of their abilities. Furthermore, in our analyses comparing teachers who received coaching outside our technology-focused program to those in the program, we saw relatively small differences in self-rating of ability to engage students in ITU, and larger differences in the frequency of student ITU and the perceived impact of ITU. Therefore, while teacher self-ratings of capabilities may have a place in the evaluation of the implementation of meaningful use of technology, it seems more helpful to consider the actual frequency of technology use activities.
Reliability of rubric measures
Analyses of the teacher survey showed the measures of teacher self-rating ability to engage students in ITU, frequency of student ITU practices, and teacher perceived impact of ITU on student engagement and learning all three demonstrated strong reliability (Standardized Alpha > .90 for combined indices). The frequency of student ITU practices items produced reliable index scores consisting of 2-4 items for each skill (Standardized Alpha > .80 or item correlations > .74).
The purpose of this research session is to share the process of developing, user testing and validating a rubric that educators can use to self-assess their growth in classroom technology use. We employed the rubric items into a survey study to better understand the perceptions of teachers regarding their ability to engage students in ITU. We found that teachers’ reports of frequency of actual activities occurring in the classroom as captured by our newer version of the rubric were better predictors of impact than teacher self-ratings, and they were also more closely tied to participation in the program. The resulting rubric and surveys from this study can help educators and researchers reflect on which are the most impactful technology uses in specific grades and subjects, where teachers think students need help, and how to increase impact through coaching and use of rubrics.
Budhai, S. S., & Taddei, L. M. (2015). Teaching the 4Cs with Technology: How do I use 21st century tools to teach 21st century skills? ASCD.
Hixson, N. K., Ravitz, J., & Whisman, A. (2012). Extended Professional Development in Project-Based Learning: Impacts on 21st Century Skills Teaching and Student Achievement. West Virginia Department of Education.
Kivunja, C. (2015). Exploring the pedagogical meaning and implications of the 4Cs “super skills” for the 21st century through Bruner’s 5E lenses of knowledge construction to improve pedagogies of the new learning paradigm. Creative Education, 6(02), 224.
NEA. (2012). Preparing 21st century students for a global society. National Education Association.
NRC. (2013). Education for life and work: Developing transferable knowledge and skills in the 21st century. National Academies Press. National Research Council.
Partnership for 21st Century Skills (2007). The intellectual and policy foundations of the 21st century skills framework.
Saavedra, A. R., & Opfer, V. D. (2012). Learning 21st-century skills requires 21st-century teaching. Phi Delta Kappan, 94(2), 8-13.
The William and Flora Hewlett Foundation (2010). Education program strategic plan. Menlo Park, CA: Author.
Trilling, B., & Fadel, C. (2009). 21st Century Skills.: Learning for Life in Our Times. John Wiley & Sons.
US Department of Education. (2017). Reimagining the role of technology in education: 2017 national education technology plan update.