Against the backdrop of rapid developments of algorithm-based feedback tools — from older tools mainly providing feedback on grammar and spelling to advanced tools based on generative artificial intelligence offering more comprehensive writing support — our meta-analysis examines to what extent algorithm-based feedback improves not only surface- (e.g., grammar and spelling) but also deep-level (e.g., structure, content, coherence) writing outcomes for different learners at secondary school and university. We reviewed experimental and quasi-experimental studies published between 2011 and the end of 2024, covering five European languages. Results from the 33 included studies indicated that algorithm-based feedback was beneficial for improving writing in general ( g = 0.36). Specifically, positive effects were observed for surface-level outcomes at posttest ( g = 0.31), though no lasting effects were found at maintenance ( g = −0.02). In contrast, deep-level writing outcomes showed sustained improvement, with positive effects both at posttest ( g = 0.31) and maintenance ( g = 0.54). No significant differences between secondary and university students were observed. However, L2 learners, in general, seemed to profit most from algorithm-based feedback, showing gains in surface- ( g = 0.77, bordering on significance), and deep-level outcomes ( g = 0.46). While no significant differences were found between the effects of specific types of algorithm-based feedback tools, feedback from Grammarly and Pigai statistically enhanced students’ writing, but effects of ChatGPT feedback were non-significant. We discuss implications for future research and educational practice, also in light of the small transfer of learning to new writing tasks. • Small effects for surface- and deep-level outcomes at posttest. • L2 learners particularly benefit. • No effect for surface-level outcomes at maintenance. • No significant difference between tools. • Small transfer effect to new writing tasks.

Students are usually expected to write full texts in English as a foreign language (EFL) at the end of secondary education. However, research on EFL writing at school is scarce, especially regarding less proficient writers, and seldom focuses on deep-level text features such as structure and coherence. Based on a sample of 166 EFL students in Year 9 attending German middle and lower performance track schools, this study examined 326 narrative and argumentative texts. First, we assessed structure and coherence via analytic ratings using detailed rubrics to gain insights into possible challenges for students. Our analysis showed that relevant text parts (such as the conclusion) were mostly missing and that students struggled to establish a broad common thread with argumentative texts being overall less structured and coherent than narrative texts. Second, we used the software Comproved® to conduct holistic ratings of overall text quality and compared them with our analytic ratings. Large correlations between both ratings suggest that structure and coherence are important aspects of text quality. We discuss how our rubrics can serve as a useful tool for assessment for learning and assist less proficient writers in establishing deep-level features in their texts.