Assessing Writing Jul 2026 Open Access

From keystrokes to scores: Toward a multidimensional predictive model of writing evaluation by humans and large language models across linguistic, cognitive, and social dimensions

Qiao Gan University of Canterbury ; Benjamin Adams University of Canterbury

Abstract

Automated writing evaluation (AWE) has traditionally emphasized textual features such as vocabulary and syntax, while often overlooking writers’ social identities and cognitive behaviors – factors central to understanding writing as a multidimensional construct. With the increasing integration of large language models (LLMs) into AWE, questions remain about how their assessments align with human judgments and the sources of potential divergences. This study investigates how linguistic (e.g., lexical diversity), cognitive (e.g., pausing behavior), and social (e.g., gender) factors covary with essay scores assigned by human raters and LLMs. We analyzed 4245 argumentative essays paired with demographic metadata and keystroke-logging data, using correlation analyses, random forest models, and regression-based approaches to examine relationships among writer characteristics, writing-process features, textual features, and essay scores. Results showed moderate agreement between human and LLM scores, but the two scoring systems exhibited different patterns of association with linguistic, cognitive, and social variables. These findings suggest that human and LLM evaluations rely on partially different cues and demonstrate how socio-cognitive metadata can be used to examine the factors associated with writing assessment decisions. By moving beyond text-only comparisons, this approach provides a complementary lens for understanding why and how human and machine judgments converge or diverge.

Journal
Assessing Writing
Published
2026-07-01
DOI
10.1016/j.asw.2026.101090
CompPile
Open Access
OA PDF Hybrid
Topics
Export
BibTeX RIS

Citation context

Cited by in this index (0)

No articles in this index cite this work.

References (87) · 12 in this index

  1. Andreev (2024)
    The secret life of connectives: A taxonomy to study individual differences in mid-adolesc…
    Reading and Writing  ↗
  2. Barkaoui (2019)
    What can L2 writers’ pausing behavior tell us about their L2 writing processes?
    Studies in Second Language Acquisition  ↗
  3. Barzilay (2008)
    Modeling local coherence: An entity-based approach
    Computational Linguistics  ↗
  4. Bates (2015)
    Fitting linear mixed-effects models using lme4
    Journal of Statistical Software  ↗
  5. Beigman Klebanov (2020)
    Automated evaluation of writing – 50 years and counting
    Proceedings of the 58th annual meeting of the association for computational linguistics
Show all 87 →
  1. Brehm (2022)
    Contrast coding choices in a decade of mixed models
    Journal of Memory and Language  ↗
  2. Breiman (2001)
    Random forests
    Machine Learning  ↗
  3. Bridgeman (2012)
    Comparison of human and machine scoring of essays: Differences by gender, ethnicity, and …
    Applied Measurement in Education  ↗
  4. Bui (2025)
    ChatGPT as an automated essay scoring tool in the writing classrooms: How it compares wit…
    Education and Information Technologies  ↗
  5. Cheshire (2011)
    Contact, the feature pool and the speech community: The emergence of Multicultural London…
    Journal of Sociolinguistics  ↗
  6. Coles-White (2004)
    Negative concord in child African American English: Implications for specific language im…
    Journal of Speech, Language, and Hearing Research  ↗
  7. Crossley (2025)
    Developing linguistic constructs of text readability using Natural Language Processing
    Scientific Studies of Reading  ↗
  8. Crossley (2024)
    Plagiarism detection using keystroke logs
    Proceedings of the 17th international conference on educational data mining
  9. Deane (2013)
    On the relation between automated essay scoring and modern views of the writing…
    Assessing Writing
  10. Deane (2014)
    Using writing process and product features to assess writing quality and explore how thos…
    ETS Research Report Series
  11. Eckert (2012)
    Three waves of variation study: The emergence of meaning in the study of sociolinguistic …
    Annual Review of Anthropology  ↗
  12. Fogal (2020)
    Complex dynamic systems theory and L2 writing development
  13. Galbraith (2019)
    Aligning keystrokes with cognitive processes in writing
    Observing writing
  14. Gan (2024)
    Ethnic variation in its social context: Evidence from the English of Chinese Australians
    PhD Thesis
  15. Gan (2026)
    A multi-dialectal, longitudinal corpus of human-AI hybrid language production
    Proceedings of the fifteenth language resources and evaluation conference (LREC 2026)  ↗
  16. Gelman (2008)
    Scaling regression inputs by dividing by two standard deviations
    Statistics in Medicine  ↗
  17. Gries (2021)
    Statistics for Linguistics with R: A practical introduction
  18. Guo (2019)
    Writing process differences in subgroups reflected in keystroke logs
    Journal of Educational and Behavioral Statistics  ↗
  19. Halliday (2014)
    Halliday's Introduction to Functional Grammar
  20. Hastie (2009)
    The elements of statistical learning: Data mining, inference, and prediction
  21. He (2025)
    Beyond final products: Multi-dimensional essay scoring using keystroke logs and deep learning
    LAK25: The 15th international learning analytics and knowledge conference
  22. Hewlett Foundation (2012)
    The Hewlett Foundation automated essay scoring
    The William and Flora Hewlett Foundation
  23. Hyland (2007)
    Genre pedagogy: Language, literacy and L2 writing instruction
    Journal of Second Language Writing  ↗
  24. Jarvis (2013)
    Capturing the diversity in lexical diversity
    Language Learning  ↗
  25. Ke, Z., & Ng, V. (2019). Automated essay scoring: A survey of the state of the art. Proceedings of the 28th I…
     ↗
  26. Kendall (1938)
    A new measure of rank correlation
    Biometrika  ↗
  27. Kendro (2026)
    Do large language models produce texts with "human-like" lexical diversity? Evidence from…
    International Journal of Applied Linguistics, Online first
  28. Kerz (2020)
    Understanding the dynamics of second language writing through keystroke logging and compl…
    Proceedings of the twelfth language resources and evaluation conference
  29. Kyle (2017)
    Assessing syntactic sophistication in L2 writing: A usage-based approach
    Language Testing  ↗
  30. Kyle (2021)
    Assessing the validity of lexical diversity using direct judgements
    Language Assessment Quarterly  ↗
  31. Labov (1972)
    Negative attraction and negative concord in English grammar
    Language  ↗
  32. Labov (1990)
    The intersection of sex and social class in the course of linguistic change
    Language Variation and Change  ↗
  33. Labov (2001)
    Principles of linguistic change
    Volume 2: Social factors
  34. Lan et al. (2025)
    Investigating a customized generative AI chatbot for automated essay scoring in…
    Assessing Writing
  35. Larsen-Freeman (2022)
    A complex dynamic systems perspective to researching language classroom dynamics
    Research Questions in Language Education and Applied Linguistics
  36. Larsen-Freeman, D., & Cameron, L. (2008). Complex systems and applied linguistics. Oxford University.
  37. Lee (2023)
    A human-centric automated essay scoring and feedback system for the development of ethica…
    Educational Technology & Society
  38. Leijten et al. (2013)
    Keystroke Logging in Writing Research: Using Inputlog to Analyze and Visualize …
    Written Communication
  39. Lenth, R.V. (2025). emmeans: Estimated marginal means, aka least-squares means. R package version 1.11.1. 〈ht…
  40. Limpo et al. (2017)
    Written Language Bursts Mediate the Relationship Between Transcription Skills a…
    Written Communication
  41. Liu et al. (2025)
    Comparing GPT-based approaches in automated writing evaluation
    Assessing Writing
  42. Loukina (2019)
    The many dimensions of algorithmic fairness in educational applications
    Proceedings of the fourteenth workshop on innovative use of NLP for building educational applications
  43. Lu (2010)
    Automatic analysis of syntactic complexity in second language writing
    International Journal of Corpus Linguistics  ↗
  44. Lu (2011)
    A corpus-based evaluation of syntactic complexity measures as indices of college-level ES…
    TESOL Quarterly  ↗
  45. Malvern (2004)
    Lexical diversity and language development: Quantification and assessment
  46. Mansour (2024)
    Can large language models automatically score proficiency of written essays?
    Proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation
  47. McCarthy (2007)
    vocd: A theoretical and empirical evaluation
    Language Testing  ↗
  48. McCarthy (2010)
    MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversi…
    Behavior Research Methods  ↗
  49. McNamara et al. (2010)
    Linguistic Features of Writing Quality
    Written Communication
  50. McNamara et al. (2015)
    A hierarchical classification approach to automated essay scoring
    Assessing Writing
  51. Mizumoto (2023)
    Exploring the potential of using an AI language model for automated essay scoring
    Research Methods in Applied Linguistics  ↗
  52. Morris et al. (2025)
    Distinguishing effective writing styles in the PERSUADE corpus
    Journal of Writing Research
  53. Mulac (2013)
    The gender-linked language effect: an empirical test of a general process model
    Language Sciences  ↗
  54. Ngo (2024)
    The effectiveness of automated writing evaluation in EFL/ESL writing: A three-level meta-…
    Interactive Learning Environments  ↗
  55. Obermeyer (2019)
    Dissecting racial bias in an algorithm used to manage the health of populations
    Science  ↗
  56. Ouyang (2018)
    Can the probability distribution of dependency distance measure language proficiency of s…
    Journal of Quantitative Linguistics
  57. Ouyang et al. (2022)
    Dependency distance measures in assessing L2 writing proficiency
    Assessing Writing
  58. Page (1966)
    The imminence of… grading essays by computer
    The Phi Delta Kappan
  59. Powers (2015)
    Validating automated essay scoring: A (modest) refinement of the “gold standard”
    Applied Measurement in Education  ↗
  60. Révész (2019)
    Exploring second language writers' pausing and revision behaviors: A mixed-methods study
    Studies in Second Language Acquisition  ↗
  61. Révész (2023)
    Exploring the relationship of working memory to the temporal distribution of pausing and …
    Studies in Second Language Acquisition  ↗
  62. Schwarz (1978)
    Estimating the dimension of a model
    Annals of Statistics  ↗
  63. Sinharay (2019)
    Prediction of essay scores from writing process and product features using data mining methods
    Applied Measurement in Education  ↗
  64. Stevenson (2006)
    Revising in two languages: A multi-dimensional comparison of online writing revisions in …
    Journal of Second Language Writing  ↗
  65. Strobl (2009)
    An introduction to recursive partitioning: Rationale, application, and characteristics of…
    Psychological Methods  ↗
  66. Suhan (2026)
    A comparative study of the human, automated scoring model, and GPT-4 ratings of young EFL…
    Language Testing  ↗
  67. Tian et al. (2025)
    The KLiCKe corpus: Keystroke logging in compositions for knowledge evaluation
    Journal of Writing Research
  68. Tian (2025)
    Exploring the application of keystroke logging techniques to research in second language …
    Research Methods in Applied Linguistics  ↗
  69. Travis (2025)
    The intersection of ethnicity and social class in language variation and change
    Language Variation and Change  ↗
  70. Uto, M., Xie, Y., & Ueno, M. (2020). Neural automated essay scoring incorporating hand crafted features. Proc…
     ↗
  71. Waes et al. (2015)
    Fluency in Writing: A Multidimensional Perspective on Writing Fluency Applied t…
    Computers and Composition
  72. Voeten, C.C. (2025). buildmer: Stepwise Elimination and Term Reordering for Mixed-Effects Regression. R packa…
  73. West (2021)
    Best practice in statistics: The use of log transformation
    Annals of Clinical Biochemistry  ↗
  74. Wilcox (2005)
    Trimming and winsorization
    Encyclopedia of Biostatistics  ↗
  75. Wright (2017)
    ranger: A fast implementation of Random Forests for highdimensional data in C++ and R
    Journal of Statistical Software  ↗
  76. (2009)
    Writing in foreign language contexts: Learning, teaching, and research
    Multilingual Matters
  77. Wu (1993)
    An accurate computation of the hypergeometric distribution function
    ACM Transactions on Mathematical Software  ↗
  78. Xiao (2025)
    Human-AI collaborative essay scoring: A dual-process framework with LLMs
    LAK25: The 15th international learning analytics and knowledge conference (LAK 2025)
  79. Yamashita (2025)
    Exploring potential biases in GPT-4o’s ratings of English language learners’ essays
    Language Testing  ↗
  80. Yancey (2023)
    Rating short L2 essays on the CEFR scale with GPT-4
    Proceedings of the 18th workshop on innovative use of NLP for building educational applications (BEA 2023)
  81. Zenker et al. (2021)
    Investigating minimum text lengths for lexical diversity indices
    Assessing Writing
  82. Zhai (2023)
    The effectiveness of automated writing evaluation on writing quality: A meta-analysis
    Journal of Educational Computing Research  ↗
CrossRef global citation count: 0 View in citation network → Build reading path →

Related articles

  1. Pedagogy Jan 2024
    Contributors ↗
    decolonial rhetorics first-year composition teacher development writing centers digital rhetoric multimodality artificial intelligence grammar and mechanics literacy studies race and writing gender and writing community literacy affect and writing editorial matter
  2. Pedagogy Jan 2022
    Contributors ↗
    modern rhetorical theory rhetorical criticism genre theory discourse analysis african american rhetorics decolonial rhetorics first-year composition writing pedagogy basic writing writing across the curriculum graduate education teacher development argument collaborative writing transfer assessment portfolios writing program administration writing centers peer tutoring technical communication professional writing archival research digital rhetoric social media grammar and mechanics literacy studies race and writing gender and writing disability studies public rhetoric community literacy literary studies editorial matter
  3. Rhetorica Jan 2000
    Critiques of Knowing: Situated Textualities in Science, Computing, and the Arts by Lynette Hunter ↗
    C. Jan Swearingen
    rhetorical criticism discourse analysis argument assessment digital rhetoric artificial intelligence race and writing gender and writing book reviews
  4. Assessing Writing Jul 2026
    GPTZero and the challenges of AI detection in assessing writing ↗
    Louie Giray; Kershnee Sevnarayan; Kgabo Bridget Maphoto
    assessment artificial intelligence multilingual writers literacy studies race and writing editorial matter
  5. Journal of Response to Writing Apr 2026
    Examining Automated Writing Evaluation Error Coverage in Relation to Uptake and Retention ↗
    Mehrzad, Mohaddeseh; Rahimi, Mohammad; Link, Stephanie
    genre theory writing pedagogy teacher development revision assessment artificial intelligence multilingual writers grammar and mechanics