Improve Failures, Completion and Grades with Gamification

Framework

This study draws on principles of behavioral economics, motivation theory, and learning cognition theory to design information systems that help students WANT to improve and CONNECT with their teachers.

Methods

We use a quasi-experimental design with a treatment and control group. Data was drawn from four sections of a freshman/sophomore level Technology Management course at a large open enrollment university (n=193). T-tests were used to identify significant differences in academic outcomes between part- and full-time teachers both with and without a gamified course overlay.

The study evaluated the following six hypothesis:

Evaluate if there is a difference in academic outcomes between part- and full-time instructors.
H1: A course without an EEIS taught by a part-time instructor will have significantly more students who fail than a course without an EEIS taught by an experienced full-time instructor.
H2: A course without an EEIS taught by a part-time instructor will have significantly more students who drop out than a course without an EEIS taught by an experienced full-time instructor.
H3: A course without an EEIS taught by a part-time instructor will have significantly fewer students who earn a B- or above grade than a course without an EEIS taught by an experienced full-time instructor.

Evaluate if using an EEIS reduces the difference in academic outcomes between part- and full-time instructors.
H4: A course with an EEIS taught by a part-time instructor will not have significantly more students who fail than a course with an EEIS taught by an experienced full-time instructor.
H5: A course with an EEIS taught by a part-time instructor will not have significantly more students who drop out than a course with an EEIS taught by an experienced full-time instructor.
H6: A course with an EEIS taught by a part-time instructor will not have significantly fewer students who earn a B- or above grade than a course with an EEIS taught by an experienced full-time instructor.

Results

The purpose of this study was to examine possible differences in academic outcomes between part- and full-time instructors, with and without the use of an EEIS. Without an EEIS, students in the part-time instructor's course sections demonstrated significantly higher failure rates (a 143.8% increase) and dropout rates (a 110.4% increase) with significantly fewer students scoring a B- or higher (39.8% decrease) when compared to students in the course sections taught by a full-time instructor. These findings support H1, H2, and H3. The effect size for each of these observations is in the moderate range.
These findings do not support research that grades in courses taught by part-time instructors are higher than those taught by full-time instructors (Ran & Xu, 2017; Ran & Xu, 2018; Xu, 2019). This is possibly because assignments and exams in this study were held consistent across course sections, and rigor and requirements were not manipulated (Benjamin, 2002, 2003).
Students in sections taught by the part-time instructor without an EEIS had fewer B- or above grades, suggesting less understanding of the course content. Assuming that mastery of content (an intrinsic motivator) is linked to grades (an extrinsic motivator), students may have lost interest when their effort did not result in expected rewards (e.g., see McEvoy, 2011). It is concerning that students in the part-time instructor’s course without an EEIS had significantly lower academic outcomes. This could impact retention and continuation in a major as indicated by previous research (Ran & Xu, 2017, 2018; Bettinger & Long, 201; Jaeger & Hinz, 2008).
With an EEIS, when comparing part- and full-time instructors, there was no significant difference in failure and dropout rates or in the number of students scoring a B- or higher in the course. In fact, with an EEIS, the failure and dropout rates were statistically identical for part- and full-time instructor courses. These findings support H4, H5, and H6. When using an EEIS (compared with not using an EEIS), the part-time instructor showed a 62.1% decrease in failures, a 61.4% decrease in dropouts, and a 41.7% increase in the number of students scoring a B- or higher in the course. We are unaware of other interventions that yield such large improvements in academic performance. This suggests that using an EEIS such as Delphinium may compensate for part-time instructors’ limitations of expertise, time, or rewards that can have a negative impact on students’ academic outcomes.
The EEIS had only a minimal impact on failure rates (7.7% decrease) and dropout rates (18.8% decrease) for the full-time instructor. This suggests there is a ceiling effect for the improvements that an EEIS can make in student performance. This may be because experienced instructors are already doing the kinds of things that an EEIS does, such as motivating students, tracking grades, and providing feedback about progress. The role of task feedback in increasing intrinsic motivation is supported by previous research (e.g., see Debnath et al., 2007, Garaus et al., 2016, McEvoy, 2011). Additionally, full-time instructors have more time to dedicate to students outside of class than part-time instructors and more rewards for doing so (Jaeger & Hinz, 2008, Jaeger & Eagan, 2009, 2011; Ran & Xu, 2017).
When the full-time instructor used an EEIS, the number of students that scored a B- or above actually decreased by 2.8% and there were fewer A grades when compared to not using an EEIS. After interviewing the instructor and reviewing responses to qualitative, open-ended questions from students, we speculate that this is because Delphinium makes progress very apparent, and students know when to stop working to achieve the grade they desire. An explanation for this may be related to the extrinsic motivation of grades; students may have been extrinsically motivated by the features of Delphinium, such as the grade tracker, and with the platform itself rather than being intrinsically motivated by the content (e.g., see Deci et al., 1999; Mallin & Pullins, 2009; McEvoy, 2011; Morgan-Thomas & Dudau, 2019). Non-EEIS versions of the course did not have this clear indication of course progress and high-performing students continued to work, we speculate, because they were unsure of their final grade, or perhaps because they were more intrinsically motivated, and wanted to ensure that they cleared the threshold for the grade they desired. This suggests that satisficing students will stop working on assignments when the EEIS interface tells them they have reached their goal, or possibly that the grade tracker feature encouraged performance-based on extrinsic motivation that did not sustain students' interest once they had achieved an acceptable grade (Bain, 2004; Deci et al., 1999; Mallin & Pullins, 2009; McEvoy, 2011). The cafeteria style course design, which offered students a choice of assignments, should have strengthened interest and motivation for both sections (Debnath et al., 2007; Dobrow et al., 2011), however choice alone resulted in significantly lower outcomes in the part-time instructor course with no EEIS. We recommend that courses using an EEIS consider ways to encourage satisficing students to persist to course completion. This might be done, for example, by having a relatively large number of points in the final assignment that students need to earn to improve their grade.

Importance

Using adjunct and other types of part-time instructors will likely remain a prevalent practice in higher education management courses. Given that using part-time instructors can have a negative impact on student graduation and persistence in a field of study, it is important to identify ways we can augment part-time instructors’ performance. In this study we introduced the concept of an educational engagement information system (EEIS) designed to help students WANT to perform and to give teachers tools for monitoring and communicating with their students. We demonstrated that when part-time instructors use an EEIS, it can result in significantly lower students’ failure and dropout rates and an increase in the rate of students earning a B- or above; and bring their students’ performance to parity with the performance of students taught by a full-time instructor.

References

References
Abrams, S. S., & Walsh, S. (2014). Gamified vocabulary. Journal of Adolescent & Adult Literacy, 58(1), 49–58.
American Association of University Professors. (n.d.). Background facts on contingent faculty positions. https://www.aaup.org/issues/contingency/background-facts
Andrade, M. S. (2013). Global learning by distance: Principles and practicalities for learner support. International Journal of Online Pedagogy and Course Design, 3(1), 66–81. http://dx.doi.org/10.4018/ijopcd.2013010105
Andrade, M. S. (20141). Course embedded support for online English language learners. Open Praxis, 6(1), 65–73. http://openpraxis.org/index.php/OpenPraxis/issue/view/7/showToc. http://dx.doi.org/10.5944/openpraxis.6.1.90
Andrade, M. S. (2014b). Dialogue and structure: Enabling learner self-regulation in technology enhanced learning environments. European Journal of Educational Research 13(5), 563–574. www.wwwords.eu/eerj/content/pdfs/13/issue13_5.asp
Andrade, M. S., & Bunker, E. L. (2009). Language learning from a distance: A new model for success. Distance Education, 30(1), 47–61. http://dx.doi.org/10.1080/01587910902845956
Benjamin, E. (2002). How over reliance upon contingent appointments diminishes
faculty involvement in student learning. Peer Review, 5(1), 4–10.
Benjamin, E. (2003). Reappraisal and implications for policy and research. New
Directions for Higher Education, 123, 79–113.
Bettinger, E. P., & Long, B. T. (2010). Does cheaper mean better? The impact of using adjunct instructors on student outcomes. The Review of Economics and Statistics, 92(3), 598– 613.
Burke, L. A., & Moore, J. E. (2003). A perennial dilemma in OB education: Engaging the traditional student. Academy of Management Learning & Education, 2(1), 37–52.
Chapman, J. R., & Rich, P. J. (2018). Does educational gamification improve students’ motivation? If so, which game elements work best? Journal of Education for Business, 93(7), 315–322.
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1992). Extrinsic and intrinsic motivation to
use computers in the workplace. Journal of Applied Social Psychology, 22, 1111–1132.
Debnath, S. C., Tandon, S., & Pointer, L. V. (2007). Designing business school courses to promote student motivation: An application of the job characteristics model. Journal of Management Education, 31(6), 812–831.
Deci, E. L., & Ryan, R. M. (1985). Self-determination. New Jersey: John Wiley and Sons Inc.
Deci, E. L., & Ryan, R. M. (2000). The “what” and “why” of goal pursuits: human needs and the self-determination of behavior. Psychological Inquiry, 11(4), 227–268. http://dx.doi.org/10.1207/S15327965PLI1104_01
de-Marcos, L., Domınguez, A., Saenz-de-Navarrete, J., & Pages, C. (2014). An empirical study comparing gamification and social networking on e-learning. Computers & Education, 75, 82–91. https://portal.uah.es/portal/page/portal/epd2_profesores/prof23288/publicaciones/deMarcos2013_Gamif2_v6_CAE_preprint.pdf
Delphinium (2021). delphi m.e. | Home. http://delphi-me.com/
Dembo, M. H., Junge, L.G., & Lynch, R. (2006). Becoming a self-regulated learner: Implications for web-based education. In H. F. O’Neil, & R. S. Perez (Eds.), Web-based learning: Theory, research, and practice (pp. 185–202). Mahwah, N. J: Lawrence Erlbaum Associates.
Denny, P., McDonald, F., Empson, R., Kelly, P., & Petersen, A. (2018). Empirical support for a causal relationship between gamification and learning outcomes. Paper presented at the Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal QC, Canada.
Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011, September). From game design elements to gamefulness: Defining “gamification”. Paper presented at the 15th Annual International MindTrek, Tampere, Finland. https://doi.org/10.1145/2181037.2181040
https://www.researchgate.net/publication/230854710_From_Game_Design_Elements_to_Gamefulness_Defining_Gamification
Dobrow, Shoshana R., Wendy K. Smith, and Michael A. Posner. "Managing the grading paradox: Leveraging the power of choice in the classroom." Academy of Management Learning & Education 10.2 (2011): 261–276.
Drace, K. (2013). Gamification of the laboratory experience to encourage student engagement. Journal of Microbiology & Biology Education, 14(2), 273–274. https://doi.org/doi:10.1128/jmbe.v14i2.632
Eagen, M. K., & Jaeger, A. J. (2008). Effects of exposure to part-time faculty on community college transfer. Research in Higher Education, 50(2), 168–188.
Holman, C., Aguilar, S., & Fishman, B. (2013, April). GradeCraft: What can we learn from a game-inspired learning management system?. In Proceedings of the third international conference on learning analytics and knowledge (pp. 260-264).
Hurst, D., Cleveland-Innes, M., Hawranik, P., & Gauvreau, S. (2013). Online graduate student identity and professional skills development. Canadian Journal of Higher Education,
43(3), 36–55.
Farzan, R., DiMicco, J. M., Millen, D. R., Dugan, C., Geyer, W., & Brownholtz, E. A. (2008). Results from deploying a participation incentive mechanism within the enterprise. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 563–572.
Figlio, D. N., Schapiro, M. O., & Soter, K. B. (2015). Are tenure track professors better teachers? Review of Economics and Statistics, 97(4), 715–724.
Garaus, C., Furtmüller, G., & Güttel, W. H. (2016). The hidden power of small rewards: The effects of insufficient external rewards on autonomous motivation to learn. Academy of Management Learning & Education, 15(1), 45–59.
Goehle, G. (2013). Gamification and web-based homework. PRIMUS, 23(3), 234–246. https://doi.org/doi:10.1080/10511970.2012.736451
Goodwin, J. A., & Gilbert, B. D. (2001). Cafeteria-style grading in general chemistry. Journal of Chemical Education, 78(4), 490.
Gravetter, F. J., & Forzano, L.-A. B. (2018). Research methods for the behavioral sciences. Cengage Learning.
Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does gamification work?–A literature review of empirical studies on gamification. Paper presented at the Hawaii International Conference on System Sciences.
Jaeger, A. J., & Hinz, D. (2008). The effects of part-time faculty on first semester
freshmen retention: A predictive model using logistic regression. Journal of College
Student Retention: Research, Theory & Practice, 10, 265–286.
Jaeger, A. J., & Eagan, M. K. (2009). Effects of exposure to part-time faculty on associate’s
degree completion. Community College Review, 36, 167–194.
Jaeger, A. J., & Eagen, M. K. (2011). Examining retention and contingent faculty use in a state system of public higher education. Educational Policy, 25(3), 507–537. https://doi.org/10.1177/0895904810361723
Kenton, W., & Walters, T., (2020, September 28). Behavioral economics. https://www.investopedia.com/terms/b/behavioraleconomics.asp#:~:text=Behavioral%20Economics%20is%20the%20study,2.
Hanus, M. D., & Fox, J. (2015). Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance. Computers & Education, 80, 152–161.
Instructure. (2020). Educational software development. https://www.instructure.com/
Koivisto, J., & Hamari, J. (2019). The rise of motivational information systems: A review of gamification research. International Journal of Information Management, 45, 191–210.
Lepper, M. R., Greene, D., & Nisbett, R. E. (1973). Undermining children's intrinsic interest with extrinsic reward: a test of the “overjustification” hypothesis. Journal of Personality & Social Psychology, 28(1), 129–137. http://dx.doi.org/10.1037/h0035519
McCarney, R., Warner, J., Iliffe, S., Van Haselen, R., Griffin, M., & Fisher, P. (2007). The Hawthorne Effect: A randomised, controlled trial. BMC Medical Research Methodology, 7(1), 30.
McEvoy, G. M. (2011). Increasing intrinsic motivation to learn in organizational behavior classes. Journal of Management Education, 35(4), 468–503.
Morgan-Thomas, A., & Dudau, A. (2019). Of possums, hogs, and horses: Capturing the duality of student engagement in eLearning. Academy of Management Learning & Education, 18(4), 564–580.
National Center for Education Statistics. (2020, May). The condition of education. https://nces.ed.gov/programs/coe/indicator_csc.asp
Nevin, C. R., Westfall, A. O., Rodriguez, J. M., Dempsey, D. M., Cherrington, A., Roy, B., … Willig, J. H. (2014). Gamification as a tool for enhancing graduate medical education. Postgraduate Medical Journal, 90(1070), 685–693. https://doi.org/doi:10.1136/postgradmedj-2013-132486
Ran, F. X., & Xu, D. (2017). How and why do adjunct instructors affect students’ academic outcomes? Evidence from four-year colleges. A CAPSEE working paper. https://ccrc.tc.columbia.edu/media/k2/attachments/how-and-why-do-adjunct-instructors-affect-students-academic-outcomes.pdf
Ran, X., & Xu, D. (2018). Does contractual form matter? The impact of different types of non-tenure-track faculty on college students’ academic outcomes. Journal of Human Resources, 54(4), 1081–1120.
Reichardt, C. S. (2009). Quasi-experimental design. The SAGE Handbook of Quantitative Methods in Psychology, 46(71), 490–500.
Reeve, J. (2002). Self-determination theory applied to educational settings. In E. L. Deci & R. M. Ryan (Eds.), Handbook of self-determination research (pp. 183–203). University of Rochester Press.
Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic
motivation, social development, and well-being. The American Psychologist,
55(1), 68–78.
Sanchez, E., Young, S., & Jouneau-Sion, C. (2017). Classcraft: from gamification to ludicization of classroom management. Education and Information Technologies, 22(2), 497-513.
Thaler, R. H., & Sunstein, C. R. (2008). Nudge: Improving decisions about health, wealth, and happiness. Yale University Press.
Tsay, H.-H. C., Kofinas, A., & Luo, J. (2018). Enhancing student learning experience with technology-mediated gamification: An empirical study. Computers and Education, 121, 1–7.
UBS (2020). Richard H. Thaler. Nobel 2017. Perspectives on nudge theory & behavioral economics. https://www.ubs.com/microsites/nobel-perspectives/en/laureates/richard-thaler.html?gclid=CjwKCAjwq_D7BRADEiwAVMDdHr_wvVr9S35RUk-_55U-yNW-e125oNvpW-tepEnAL5eZ4H69o_PXvhoCxPQQAvD_BwE&campID=SEM-BRAND&s_kwcid=AL!602!3!460211322638!e!!g!!behavioral%20economics&ef_id=CjwKCAjwq_D7BRADEiwAVMDdHr_wvVr9S35RUk-_55U-yNW-e125oNvpW-tepEnAL5eZ4H69o_PXvhoCxPQQAvD_BwE:G:s&s_kwcid=AL!602!3!460211322638!e!!g!!behavioral%20economics
van der Heijden, H. (2004). User acceptance of hedonic information systems. MIS
Quarterly, 28(4), 695–704.
Xu, D. (2019). Academic performance in community colleges. The influences of part-time and full-time instructors. American Educational Research Journal, 56(2), 368–406. https://doi.org/10.3102/0002831218796131
Zimmerman, B. J. (1994). Dimensions of academic self-regulation: A conceptual framework for education. In D. H. Schunk, & B. J. Zimmerman (Eds.), Self-regulation of learning and performance (pp. 3–21). Lawrence Erlbaum
Zimmerman, B. J. (1998). Academic studying and the development of personal skill: A self-regulatory perspective. Educational Psychologist, 33, 73–86.
Zimmerman, B. J., & Kitsantis, A. (1997). Developmental phases in self-regulation: Shifting from process to outcome goals. Journal of Educational Psychology, 89, 29–36.