Abstract

Against the backdrop of rapid developments of algorithm-based feedback tools — from older tools mainly providing feedback on grammar and spelling to advanced tools based on generative artificial intelligence offering more comprehensive writing support — our meta-analysis examines to what extent algorithm-based feedback improves not only surface- (e.g., grammar and spelling) but also deep-level (e.g., structure, content, coherence) writing outcomes for different learners at secondary school and university. We reviewed experimental and quasi-experimental studies published between 2011 and the end of 2024, covering five European languages. Results from the 33 included studies indicated that algorithm-based feedback was beneficial for improving writing in general ( g = 0.36). Specifically, positive effects were observed for surface-level outcomes at posttest ( g = 0.31), though no lasting effects were found at maintenance ( g = −0.02). In contrast, deep-level writing outcomes showed sustained improvement, with positive effects both at posttest ( g = 0.31) and maintenance ( g = 0.54). No significant differences between secondary and university students were observed. However, L2 learners, in general, seemed to profit most from algorithm-based feedback, showing gains in surface- ( g = 0.77, bordering on significance), and deep-level outcomes ( g = 0.46). While no significant differences were found between the effects of specific types of algorithm-based feedback tools, feedback from Grammarly and Pigai statistically enhanced students’ writing, but effects of ChatGPT feedback were non-significant. We discuss implications for future research and educational practice, also in light of the small transfer of learning to new writing tasks. • Small effects for surface- and deep-level outcomes at posttest. • L2 learners particularly benefit. • No effect for surface-level outcomes at maintenance. • No significant difference between tools. • Small transfer effect to new writing tasks.

Journal
Assessing Writing
Published
2026-04-01
DOI
10.1016/j.asw.2026.101034
CompPile
Open Access
OA PDF Hybrid
Topics
Export

Citation context

Cited by in this index (1)

  1. Assessing Writing

References (106) · 5 in this index

  1. The effects of automatic writing evaluation and teacher- focused feedback on CALF measure…
    Asian-Pacific Journal of Second and Foreign Language Education
  2. Investigating EFL students’ writing skills through artificial intelligence: Wordtune appl…
    Journal of Language Teaching and Research  ↗
  3. Does e-feedback impact minimizing ESL writing errors? An experimental study
    International Journal of Emerging Technologies in Learning  ↗
  4. Using automated written corrective feedback in the writing classrooms: Effects on L2 writ…
    Computer Assisted Language Learning  ↗
  5. Using spatial contiguity and signaling to optimize visual feedback on students’ written e…
    Journal of Educational Psychology  ↗
Show all 106 →
  1. Enhancing university level English proficiency with generative AI: Empirical insights int…
    Contemporary Educational Technology
  2. Exploring EFL students’ writing performance and their acceptance of AI-based automated wr…
    Proceedings of the 2021 2nd International Conference on Education Development and Studies  ↗
  3. The effects of AWE and peer feedback on cohesion and coherence in continuation writing
    Journal of Second Language Writing
  4. The impact of online automated feedback on students’ reflective journal writing in an EFL…
    Internet Higher Education  ↗
  5. The effects of a theory-based summary writing tool on students’ summary writing
    Journal of Computer Assisted Learning  ↗
  6. AI-generated feedback on writing: insights into efficacy and ENL student preference
    International Journal of Educational Technology in Higher Education
  7. The effect of two forms of computer-automated metalinguistic corrective feedback
    Language Learning & Technology  ↗
  8. Instructor feedback on free writing and automated corrective feedback in drills: Intensit…
    Language Teaching Research  ↗
  9. Effectiveness of automated writing evaluation program to enhance EFL undergraduates’ writ…
    Asia Pacific Education Review
  10. Implementing an automated feedback program for a foreign language writing course: A learn…
    Journal of Computer Assisted Learning  ↗
  11. Mind the gap! Automated concept map feedback supports students in writing cohesive explanations
    Journal of Experimental Psychology: Applied
  12. Impact of ChatGPT on ESL students’ academic writing skills: a mixed methods intervention study
    Smart Learning Environments
  13. Using LLMs to bring evidence-based feedback into the classroom: AI-generated feedback inc…
    Computers and Education: Artificial Intelligence
  14. EssayCritic: Writing to learn with a knowledge-based design critiquing system
    Journal of Educational Technology & Society
  15. Teacher implementation of self-regulated strategy development with an automated writing e…
    Contemporary Educational Psychology  ↗
  16. The impact of online automated writing evaluation: A case study from Dalian
    Chinese Journal of Applied Linguistics
  17. The impact of automated written corrective feedback on EFL learners’ academic writing accuracy
    Journal of Teaching English for Specific and Academic Purposes  ↗
  18. The potentiality of computer-based feedback in fostering EFL learners’ writing performanc…
    Journal of Computers in Education  ↗
  19. Exploring the effect of computer-mediated teacher feedback on the writing achievement of …
    Education and Information Technologies  ↗
  20. The effects of an AWE-aided assessment approach on business English writing performance a…
    Studies in Educational Evaluation
  21. Automated writing evaluation in an EFL setting: Lessons from China
    The JALT CALL Journal  ↗
  22. The effect of integrating automated writing evaluation on EFL writing apprehension and gr…
    Innovation in Language Learning and Teaching
  23. An empirical study on the impact of an automated writing assessment on Chinese college st…
    International Journal of Language and Linguistics  ↗
  24. Writing and Pedagogy
  25. The impact of automated writing evaluation on second language writing skills of Chinese E…
    Frontiers in Psychology
  26. Automated writing evaluation and feedback: Multiple metrics of efficacy
    Journal of Educational Computing Research  ↗
  27. *Xie, Y., Huang, L., & Wang, Y. (2020). The impact of AWE and peer feedback on Chinese EFL learners’ English …
     ↗
  28. *Zhao, X. & Kidston, M. (2022). AWE Feedback on the Effectiveness of the Automatic Scoring System for English…
     ↗
  29. The psychology of written composition
  30. The effectiveness of feedback for L1-English and L2-writing development: A meta-analysis
    ETS Research Report Series  ↗
  31. Comprehensive Meta-Analysis (Version 2.2.027) [Computer software]
    Biostat
  32. Feedback to diverse language learners: Where are we and where do we need to go in literac…
    Learning and Instruction
  33. Longitudinal perspectives on students’ experiences of feedback: A need for teacher–studen…
    Higher Education Research & Development  ↗
  34. The development of student feedback literacy: Enabling uptake of feedback
    Assessment & Evaluation in Higher Education  ↗
  35. Written Communication
  36. How methodological features affect effect sizes in education
    Educational Researcher  ↗
  37. Synthesizing research: A guide for literature reviews
  38. Cooper, H. (2017). Research synthesis and meta-analysis: A step-by-step approach (5th ed.). Sage.
     ↗
  39. Effect size for ANOVA designs
  40. Tutor feedback on draft essays: Developing students’ academic writing and subject knowledge
    Journal of Further and Higher Education  ↗
  41. Automated writing evaluation systems: A systematic review of Grammarly, Pigai, and Criter…
    Education and Information Technologies  ↗
  42. A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis
    Journal of the American Statistical Association
  43. The effects of automated writing evaluation (AWE) feedback on students’ English writing q…
    Language Teaching Research Quarterly  ↗
  44. What does AI bring to second language writing? A systematic review (2014-2024)
    Language Learning & Technology  ↗
  45. Automated feedback and writing: A multi-level meta-analysis of effects on students’ performance
    Frontiers in Artificial Intelligence  ↗
  46. Cognitive models of writing
    German as a Foreign Language
  47. Stochastically dependent effect sizes
    The handbook of research synthesis and meta-analysis
  48. A Revised Writer(s)-Within-Community model of writing
    Educational Psychologist  ↗
  49. Evidence-based writing practices: a meta-analysis of existing meta-analyses
    Design Principles for Teaching Effective Writing: Theoretical and Empirical Grounded Principles
  50. Formative assessment and writing: A meta-analysis
    The Elementary School Journal  ↗
  51. A meta-analysis of writing instruction for adolescent students
    Journal of Educational Psychology  ↗
  52. The power of feedback
    Review of Educational Research  ↗
  53. A new framework for understanding cognition and affect in writing
    The science of writing: Theories, methods, individual differences and applications
  54. Identifying the organization of writing processes
    Cognitive processes in writing: An interdisciplinary approach
  55. Estimation of effect size from a series of independent experiments
    Psychological Bulletin  ↗
  56. Identifying feedback that has impact
    The impact of feedback in higher education: Improving assessment outcomes for learners
  57. Cochrane handbook for systematic reviews of interventions
  58. Research on written composition: new directions for teaching
  59. What research on second language writing tells us and what it doesn’t
    Handbook of research in second language teaching and learning
  60. International Organization for Migration. (2020). World migration report 2020. 〈https://publications.iom.int/…
  61. An argument-based approach to validation
    ACT
  62. Validity and fairness
    Language Testing  ↗
  63. The efficacy of written corrective feedback in improving L2 written accuracy: A meta-analysis
    The Modern Language Journal  ↗
  64. Attentional overload and writing performance: Effects of rough draft and outline strategies
    Journal of Experimental Psychology: Learning, Memory, and Cognition
  65. Journal of Writing Research
  66. Writing instruction improves students’ writing skills differentially depending on focal i…
    Educational Research Review  ↗
  67. Feedback in written instruction: The place of response certitude
    Educational Psychology Review  ↗
  68. Validity evidence based on testing consequences
    Psicothema  ↗
  69. Assessing Writing
  70. Generative AI and Second Language Writing
    Digital Studies in Language and Literature  ↗
  71. A review of feedback models and theories: Descriptions, definitions, and conclusions
    Frontiers in Education  ↗
  72. Practical meta-analysis
  73. The effects of online feedback on ESL/EFL writing: A meta-analysis
    The Asia-Pacific Education Researcher  ↗
  74. Exploring ChatGPT literacy in language education: A global perspective and comprehensive …
    Computers and Education: Artificial Intelligence
  75. Providing feedback in computer-based instruction: what the research tells us
  76. Validity of test interpretation and use
    Encyclopedia of educational research
  77. Computer-mediated corrective feedback to improve L2 writing skills: A meta-analysis
    Journal of Educational Computing Research  ↗
  78. 10.1007/s11618-023-01158-8
    Zeitschrift für Erziehungswissenschaft  ↗
  79. Feedback strategies for interactive learning tasks
    Handbook of research on educational communications and technology
  80. The effectiveness of automated writing evaluation in EFL/ESL writing: A three-level meta-…
    Interactive Learning Environments  ↗
  81. Effectiveness of automated writing evaluation systems in school settings: A systematic re…
    Journal of Computer Assisted Learning
  82. A framework for deliberate practice: Self-regulated strategy development and an automated writing evaluation program. North Carolina State University
  83. A review of feedback models and typologies: Towards an integrative model of feedback elements
    Educational Research Review  ↗
  84. Assessing Writing
  85. Effects of feedback on deep-level features of argumentative writing over multiple drafts:…
    Reading and Writing
  86. Meta-analysis with complex research designs: Dealing with dependence from multiple measur…
    Review of Educational Research  ↗
  87. How effective is feedback to writing for different language learners: what we know from c…
    Handbook of writing research
  88. How effective is feedback for L1, L2, and FL learners’ writing? A meta-analysis
    Learning and Instruction  ↗
  89. Focus on formative feedback
    Review of Educational Research  ↗
  90. Toward an understanding of the distinct nature of L2 writing: The ESL research and its im…
    TESOL Quarterly  ↗
  91. Comparing the quality of human and ChatGPT feedback of students’ writing
    Learning and Instruction  ↗
  92. Publication bias in meta-analysis: Its causes and consequences
    Journal of Clinical Epidemiology  ↗
  93. Exploratory data analysis
    Addison Wesleyan
  94. Teacher feedback to young children in formative assessment: A typology
    British Educational Research Journal  ↗
  95. Enhancing L2 Writing with generative AI: A Systematic Review of pedagogical Integration and outcomes [Preprint]
  96. Toward critically and humanely grounded futures for L2 writing in the GenAI era: Sparking…
    Journal of Second Language Writing  ↗
  97. Recommendations for integrating automated writing evaluation with evidence-based instruct…
    International Journal of Changes in Education  ↗
  98. Automated essay evaluation software in English Language Arts classrooms: Effects on teach…
    Computers & Education  ↗
  99. Towards automated writing evaluation: A comprehensive review with bibliometric, scientome…
    Education and Information Technologies  ↗
  100. The effectiveness of automated writing evaluation on writing quality: A meta-analysis
    Journal of Educational Computing Research  ↗
  101. The role of generative AI and hybrid feedback in improving L2 writing skills: a comparati…
    Innovation in Language Learning and Teaching