This research integrates multiple theoretical frameworks to understand the impact of writing support tools across diverse higher education contexts:
Scaffolded Learning and Vygotsky's Zone of Proximal Development: The study positions automated writing tools as digital scaffolds operating within students' zones of proximal development (Vygotsky, 1978). These tools provide scaffolded learning support that allows students to accomplish writing tasks beyond their independent abilities. Immediate Feedback Theory suggests that timely correction enhances learning retention, while these tools provide scaffolded learning consistent with Vygotsky's Zone of Proximal Development (Opitz & Mecklinger, 2011; Doo et al., 2020). This framework helps explain how automated feedback bridges the gap between students' current writing capabilities and their potential development.
Online Learning and Institutional Contexts: The research recognizes that institutional contexts significantly influence technology adoption and effectiveness. Drawing on Fischer et al. (2021), we examine how automated writing support tools function differently across institutional types, with particular attention to how community colleges and first-year students benefit from these tools. As Fischer et al. note, "first-year college students demonstrated particularly strong outcomes from technology integration," with online support tools showing significant impact on retention for this vulnerable transition population.
Explicit Strategy Instruction for Writing: Drawing from Gillespie & Graham's (2014) meta-analysis demonstrating the strong impact of explicit strategy instruction on writing outcomes (effect size = 1.09), our research examines how automated feedback tools implement strategic approaches to writing improvement. This theoretical perspective aligns with findings that participation in writing-intensive or support programs has been associated with improvements in course completion rates of 20-40% (Graham & Perin, 2007), providing context for our observed improvements in student retention and course completion, particularly among first-year students.
Intelligent Tutoring Systems and Human-AI Collaboration: Our framework incorporates VanLehn's (2011) meta-analysis demonstrating that intelligent tutoring systems can achieve comparable effectiveness to human tutoring (effect sizes of 0.76 vs. 0.79) when they provide step-based interaction and complement rather than replace human instruction. As faculty readiness research by Scherer et al. (2021) shows, successful integration depends on faculty perceptions of technology value and their own self-efficacy with these tools, which directly impacts student outcomes in critical transition years.
Stigma Reduction and Help-Seeking Behavior: Our qualitative findings contribute the concept of "technology-mediated destigmatization" of writing support. Faculty consistently observed reduced psychological barriers when students accessed writing help through software rather than face-to-face interactions. This theoretical perspective explains why vulnerable student populations showed stronger engagement with automated feedback tools, particularly first-year students who might otherwise avoid traditional writing support due to stigma. Our findings demonstrate dramatically lower dropout rates among first-year Grammarly users (3.4%) compared to non-users (14.3%)—a remarkable 10.9 percentage point difference—illustrating the critical impact of reducing help-seeking barriers during this key transition period.
Technology Integration and Academic Outcomes Framework: This study addresses significant research gaps where published articles about these tools typically provide feature reviews or user feedback surveys rather than rigorous empirical studies (Damayanti & Santosa, 2024). By examining both quantitative outcomes and implementation factors across diverse institutional contexts, our theoretical framework extends beyond isolated pilot studies to understand the comprehensive impacts of writing support tools within authentic educational ecosystems. This integrated approach helps explain the tension between demonstrated student gains and faculty adoption concerns highlighted in our findings, while providing empirical evidence for supporting students in critical transition periods.
This multifaceted theoretical framework provides a comprehensive lens for understanding how writing support tools function at the intersection of learning science, educational psychology, and technology integration in higher education settings, with particular attention to their impact on first-year students and other diverse student populations navigating institutional challenges.
Our study employed a rigorous quasi-experimental design following ESSA Level II evidence standards to examine the effectiveness of Grammarly in supporting student writing and academic performance. The research was conducted across multiple higher education institutions, with primary findings drawn from Phoenix College and Florida Atlantic University during 2023-2024. This mixed-methods approach combined quantitative analysis of student performance with qualitative stakeholder feedback to provide a comprehensive understanding of the platform's impact and implementation factors.
Primary Research Questions
Impact Assessment: What is the impact of Grammarly on student academic performance and course completion in writing-intensive courses?
Usage Patterns: How does the intensity of Grammarly usage relate to improvements in student performance over time?
Implementation Analysis: How is Grammarly implemented across different courses and disciplines, and what factors affect engagement and outcomes?
Instructor Perspectives: How do instructors perceive the usability and effectiveness of Grammarly in supporting student writing development?
Study Design and Participant Selection
The research design employed a quasi-experimental cross-cohort comparison approach examining writing-intensive courses, comparing Grammarly users to non-users. For replication purposes, our participant selection process followed these specific steps:
Institutional Selection: We identified institutions where Grammarly had been implemented (Phoenix College and Florida Atlantic University), allowing for comparison of users and non-users in similar academic contexts.
Treatment Group Identification:
At Phoenix College, the study included 569 Grammarly student-users taking writing-intensive courses (405 in Fall 2023 and 164 in Fall 2024)
At Florida Atlantic University, the study included 898 unique Grammarly users across Fall 2023 (n=660) and Fall 2024 (n=345)
Comparison Group Formation: The comparison groups consisted of students enrolled in similar writing-intensive courses who did not use Grammarly:
At Phoenix College: 3,067 non-user students (1,302 in Fall 2023 and 1,767 in Fall 2024)
At Florida Atlantic University: 11,685 non-user students
Sample Demographics:
Phoenix College: 50% Pell Grant eligible, 23% transfer students, 63% Latino, 8% Black, 18% White; average high school GPA of Grammarly users: 3.06, non-users: 3.00
FAU: 30% Pell Grant eligible, 9% first-generation college students, 25% Latino, 13% Black, 6% international students
Faculty Selection: At Phoenix College, six faculty members were purposively sampled to ensure representation across diverse disciplines, including Biological Science, Pre-Med Sciences, Computer Science, Humanities, Business departments, and academic administration (Assistant Dean).
Data Sources and Collection Methods
Our study utilized the following specific data sources and collection methods:
1. Institutional Academic Records:
Course grades (collected at end of semester)
End-of-term GPA (outcome measure)
Term-to-term retention status
Course completion status (with DFW rates calculated)
Demographic data (collected from institutional student information systems)
High school GPA (baseline measure)
2. Grammarly Usage Analytics:
Session frequency (total logins and average weekly sessions)
Writing Performance Scores (Grammarly's 1-100 scale measurement of writing quality)
Features used (percentage usage of grammar correction, tone adjustments, clarity enhancements)
Engagement patterns (average session duration and time spent on revisions)
Error correction patterns (types and frequencies of corrections accepted)
Document revision data (draft-to-final improvement metrics)
3. Faculty Interviews: Semi-structured interviews with faculty (N=6) explored the following specific questions:
How have you incorporated Grammarly into your course structure and assignments?
What changes, if any, have you observed in student writing quality since Grammarly implementation?
How has Grammarly affected your grading and feedback process?
What specific student populations seem to benefit most from Grammarly?
What challenges have you encountered in implementing Grammarly?
How has Grammarly affected your teaching practices and time allocation?
Interviews were recorded, transcribed, and coded using thematic analysis techniques with two independent coders establishing inter-rater reliability (Cohen's kappa > 0.80).
Methods of Analysis
The specific analytical methods employed included:
1. Quantitative Analysis:
Chi-Square Tests: Used to compare categorical outcomes (completion rates, retention) between Grammarly users and non-users. For example: χ²(1, N = 1,837) = 6.05, p = .014, φ = .057 for online course completion rates.
Multiple Regression Models: Constructed to predict GPA while controlling for:
Prior academic performance
Demographic factors
Course-specific variables
Grammarly usage intensity (measured as days per week)
Model specification: GPA = β₀ + β₁(Grammarly_Use) + β₂(Prior_GPA) + β₃(Demographics) + β₄(Course_Variables) + ε
Longitudinal Analysis: Paired t-tests comparing Grammarly Performance Scores from first half to second half of semester (Fall 2023: t(563) = 4.26, p < .001; Fall 2024: t(306) = 1.99, p = .047).
Dose-Response Analysis: ANOVA comparing academic outcomes across three usage intensity levels (low: <2 days/week, medium: 2-4 days/week, high: 5+ days/week).
3. Qualitative Analysis:
Interview transcripts were analyzed using thematic coding to identify key patterns in faculty perceptions
Dovetail software was used for transcript management and coding
Themes were organized to address research questions about implementation practices and perceived student impact
Validity Measures
Several specific strategies were employed to strengthen the validity of findings:
Baseline Equivalence: Established through t-tests comparing pre-intervention metrics, with standardized mean differences maintained below 0.25 standard deviations.
Multiple Outcome Measures: Academic performance was measured through course grades, writing quality metrics, and retention indicators.
Attrition Analysis: Overall attrition maintained below 11% with differential attrition below 7%.
WWC Alignment: All analytical procedures aligned with What Works Clearinghouse standards for quasi-experimental studies.
This detailed methodological approach follows ESSA Level II evidence standards and provides sufficient information for replication across diverse institutional contexts.
Our mixed-methods investigation yielded consistent findings across both institutional contexts, demonstrating positive impacts on student writing development, academic performance, and course completion rates. This section presents key results organized by qualitative faculty perspectives followed by quantitative academic outcomes.
Faculty Perspectives on Implementation
Interviews with instructors across diverse disciplines at Phoenix College and Florida Atlantic University revealed several consistent themes regarding the impact of automated writing assistance on student writing and instructional practices:
Enhanced Writing Support for Diverse Student Populations
Faculty consistently reported that the writing support tool provided differentiated assistance that was particularly beneficial for specific student populations:
English Language Learners: Instructors emphasized the value for non-native English speakers, with one Biological Science instructor noting: "Half of our students are Hispanic, and whatever percentage of them are English second language learners... When we get to the end of the class, how much better the class as a whole has done... I'm always telling students that overall their writing has improved."
Scientific Writing Support: STEM instructors highlighted the tool's ability to recognize specialized terminology while still providing grammar assistance. A Pre-Med Sciences instructor explained: "A lot of the Latin terms... they've actually gotten much better, because basically, we teach Latin... With spell check, if you were just to use Word... it doesn't recognize that term. But [this tool]... knows that that's the correct usage."
Reduced Writing Anxiety: Faculty observed that students demonstrated greater confidence in their writing abilities, with one instructor commenting: "Today's student is much more apt and much more comfortable engaging with software where they're not looking someone in the eyes and feeling dumb."
Improved Instructional Efficiency
Instructors reported that the automated writing assistance enhanced their teaching effectiveness and efficiency in several ways:
Focus on Content Over Mechanics: Faculty could concentrate on evaluating content mastery rather than mechanical errors. As one Pre-Med Sciences instructor stated: "[It] takes care of commas and capitalizing the 'I'... I can pay attention to if they're understanding the content."
Enhanced Feedback Quality: Instructors noted they could provide more substantive feedback when basic writing issues were addressed. A Biological Sciences instructor explained: "Most people don't have the time that it takes to give adequate feedback on a written assignment like that every week."
Visible Writing Progress: Faculty could more easily identify students' writing development over time, with a Computer Science instructor noting: "[It] is great again, because I've noticed, and I noticed a massive improvement since I started doing that in the class... I can tell who's using it and who's not using it."
Professional Communication Benefits
Beyond student assignments, faculty reported personal benefits from using the writing assistance tool for their professional communications:
Improved Instructional Materials: Faculty used the tool to enhance the clarity of their course materials and instructions
Enhanced Student Communications: As one Computer Science instructor mentioned: "While I'm constructing messages to the students, believe it or not, [the tool] is sitting there, making suggestions on my messages to the students."
Quantitative Academic Outcomes
Analysis of academic performance metrics across both institutions revealed significant positive relationships between usage of the automated writing assistance tool and student success measures:
Course Completion Rates
Students using the writing assistance tool demonstrated significantly higher course completion rates compared to non-users across both institutions:
FAU: Tool users achieved a 92.7% completion rate compared to 89.4% for non-users (p < .001)
Phoenix College: Completion rate advantages were consistent across course modalities:
Online courses: 83.4% for users vs. 77.0% for non-users (χ²(1, N = 1,837) = 6.05, p = .014, φ = .057)
Hybrid courses: Users showed a 5.5% completion rate advantage
In-person courses: Users showed a 5.2% completion rate advantage
Student Retention
Tool usage was associated with improved retention rates across academic disciplines at Phoenix College:
Business and Professional Studies: 93.5% vs. 73.1% (χ² = 13.68, p < .001, φ = 0.175)
Specialized Professional Programs: 91.0% vs. 74.2% (χ² = 22.84, p < .001, φ = 0.138)
STEM Fields: 85.6% vs. 71.5% (χ² = 10.85, p < .01, φ = 0.125)
Social Sciences and Humanities: 84.8% vs. 74.1% (χ² = 7.62, p < .01, φ = 0.090)
At FAU, students who used the writing assistance tool had higher retention/graduation rates compared to non-users (79.5% vs. 74.0%), representing a 5.5% advantage.
Academic Performance
Analysis of course grades and GPA demonstrated meaningful academic advantages for tool users:
FAU: Students using the writing tool 5-7 days per week achieved an average GPA of 3.69 compared to 3.29 for non-users, representing a 0.40 grade point advantage
Usage Intensity Effect: Both institutions showed a clear dose-response relationship between tool usage frequency and academic outcomes, with more frequent users demonstrating stronger results
Writing Performance Improvement
Longitudinal analysis of Writing Performance Scores revealed consistent improvement in writing quality over time:
Fall 2023 Users: Demonstrated significant improvement from first to second half of the semester, with an average increase of 2.14 points (t(563) = 4.26, p < .001)
Fall 2024 Users: Showed continued improvement with an average increase of 1.28 points (t(306) = 1.99, p = .047)
Consistent Users Across Both Years: Students who used the writing tool consistently across both academic years showed progressive improvement from 76.7 in Fall 2023 to 81.3 by Fall 2024, demonstrating cumulative benefits of sustained usage
These findings demonstrate consistent positive impacts across diverse institutional contexts, with faculty observations of improved student writing aligning with quantitative measures of academic success and writing development.
This research addresses critical gaps in understanding how automated writing feedback tools affect student outcomes in authentic higher education settings, particularly for traditionally higher risk populations. While AI-powered writing tools have proliferated rapidly, few studies have examined their impact through rigorous quasi-experimental designs with large sample sizes across multiple institutions.
Evidence-Based Implementation Insights
Our findings provide compelling evidence that automated writing assistance can significantly improve academic outcomes beyond writing mechanics alone. The documented improvements in course completion rates (3.3-6.4 percentage points), retention (11-21% across majors), and GPA (0.40 points for high-usage students) demonstrate that writing support technologies can address fundamental barriers to academic persistence. This research clarifies that these tools deliver benefits across subject areas and student subgroups, showing that usage frequency and substantially influences outcomes.
Addressing Educational Access and Equity
The study makes a unique contribution to understanding how writing technologies can expand educational access across different modalities and student populations. Our findings at Phoenix College revealed that automated writing support had the strongest impact in online courses (+6.4% completion rate) compared to hybrid (+5.5%) and in-person courses (+5.2%), suggesting these tools can help bridge the support gap in distance learning environments where students typically have less direct access to instructional support. Additionally, the FAU study showed first-year students with access to writing support were significantly less likely to drop out, indicating the tool's potential to address critical transition challenges. For ISTE attendees working with diverse institutional contexts and delivery modes, these results provide evidence-based strategies for supporting students who have traditionally had limited access to writing assistance, particularly in online and remote learning environments.
Faculty Adoption and Pedagogical Integration
Unlike many technology studies that focus solely on student metrics, our mixed-methods approach illuminates the faculty experience, revealing how automated writing assistance can enhance rather than replace instructor expertise. The documented shift from mechanical correction to content-focused feedback represents a significant pedagogical advancement. ISTE attendees will gain practical insights into how faculty across disciplines—from STEM to humanities—can effectively integrate these tools to enhance writing instruction without compromising academic rigor.
Broader Implications for Educational Technology
This research contributes to the evolving understanding of AI's role in educational contexts by:
Demonstrating how writing technologies can support institutional retention and completion initiatives
Providing evidence that technological interventions can deliver sustained benefits rather than temporary novelty effects
Identifying implementation approaches that maximize student engagement and faculty acceptance
Establishing methodological frameworks for evaluating AI writing tools in authentic educational settings
For ISTE attendees navigating rapid technological change in education, this study offers a model for thoughtful, evidence-based technology adoption that centers educational outcomes while acknowledging both the potential and limitations of AI writing tools in supporting student success.
Doo, M. Y., Bonk, C. J., & Heo, H. (2020). A meta-analysis of scaffolding effects in online learning in higher education. International Review of Research in Open and Distributed Learning, 21(3).
Fischer, C., Baker, R., Li, Q., Orona, G. A., & Warschauer, M. (2021). Increasing success in higher education: The relationships of online course taking with college completion and time-to-degree. Educational Evaluation and Policy Analysis, 43(3), 355-377.
Gillespie, A., & Graham, S. (2014). A meta-analysis of writing interventions for students with learning disabilities. Exceptional Children, 80(4), 454–473.
Graham, S., & Perin, D. (2007). Writing next: Effective strategies to improve writing of adolescents in middle and high schools – A report to Carnegie Corporation of New York. Washington, DC: Alliance for Excellent Education.
López, M. J., Santelices, M. V., & Taveras, C. M. (2023). Academic performance and adjustment of first-generation students to higher education: A systematic review. Cogent Education, 10(1).
Opitz, B., Ferdinand, N. K., & Mecklinger, A. (2011). Timing matters: The impact of immediate and delayed feedback on artificial language learning. Frontiers in human neuroscience, 5, 8.
Scherer, R., Howard, S. K., Tondeur, J., & Siddiq, F. (2021). Profiling teachers' readiness for online teaching and learning in higher education: Who's ready? Computers in Human Behavior, 118, 106675.
VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197-221.
Other presentations in this group:
Back
Trips and Tours
