Replicating and expanding a validity argument for eRevise as a formative assessment: Empirical support for theories of conceptual change

Richard Correnti University of Pittsburgh ; Lindsay Clare Matsumura University of Pittsburgh ; Elaine Lin Wang RAND Corporation ; Diane Litman University of Pittsburgh ; Zhexiong Liu University of Pittsburgh ; Haoran Zhang Meta (United States) ; Tianwen Li University of Pittsburgh

Abstract

As automated writing evaluation (AWE) systems proliferate, it is important to assess them for the extent to which they serve an authentic formative assessment purpose. We developed eRevise as an AWE system to engage upper elementary students in text-based writing, receive automated feedback, and revise their essay based on that feedback. In past work, we presented a validity argument for a response-to-text formative assessment. Here, we replicate evidence for the mediational processes we identified with a second response-to-text formative assessment. Beyond replication, we expand the validity argument in several ways: We examine multiple writing outcomes to understand whether the relationships in the data generalize. We also explore patterns of relationships to better understand which students are most likely to benefit from eRevise . Furthermore, we expand the investigation from feature score improvement alone to also consider students’ conceptual development over time. Our findings provide further evidence for sociocultural mechanisms supporting students’ conceptual development of evidence use in writing. We discuss the implications of our findings for future design of eRevise and AWE systems, in general. We also discuss the need for a recursive process where our findings also contribute to refined theory development and design of future AWE systems. (1). Formative Assessment; (2) Argumentative Writing; (3) Adaptive Expertise; (4) Conceptual Change; (5) Validity Argument.

Journal
Assessing Writing
Published
2026-10-01
DOI
10.1016/j.asw.2026.101095
CompPile
Open Access
OA PDF Hybrid
Topics
Export

Citation context

Cited by in this index (1)

  1. Assessing Writing

References (112) · 9 in this index

  1. Boundary crossing and boundary objects
    Review of Educational Research  ↗
  2. Developing dialogic teaching: Genesis, process, trial
    Research Papers in Education  ↗
  3. A dialogic teaching companion
  4. American Educational Research Association, American Psychological Association, & National Council on Measurem…
  5. Assessing the fairness of graduation predictions
    Proceedings of the 12th International Conference on Educational Data Mining
Show all 112 →
  1. Argumentation in higher education
  2. Exploring the feedback and revision features of Criterion
    Journal of Second Language Writing
  3. Authors (2020a).
  4. Dialogic writing in the upper grades of primary school: How to support peer feedback conv…
    Learning and Instruction  ↗
  5. Write, talk and rewrite: The effectiveness of a dialogic writing intervention in upper el…
    Reading and Writing  ↗
  6. Sourcing in text comprehension: A review of interventions targeting sourcing skills
    Educational Psychology Review  ↗
  7. Considering fairness and validity in evaluating automated scoring
    The Annual Meeting of the National Council on Measurement in Education (NCME)
  8. Third and fourth grade teacher’s classroom practices in writing: A national survey
    Reading and Writing  ↗
  9. Automated Essay scoring for nonnative English speakers In Computer mediated Language Assessment and evaluation in Natural Language Processing
  10. How experts deal with novel situations: A review of adaptive expertise
    Educational Research Review  ↗
  11. Learning through writing: Teaching critical thinking skills in writing assignments
    PS: Political Science & Politics
  12. Validity arguments for diagnostic assessment using automated writing evaluation
    Language Testing  ↗
  13. An analysis of the argumentative writing skills of academically underprepared college students
  14. Complex declarative learning
    The Cambridge Handbook of Thinking and reasoning
  15. Epistemic design: Design to promote transferable epistemic growth in the PRACCIS project
    Promoting spontaneous use of learning and reasoning strategies
  16. Reconsidering student feedback literacy from an ecological perspective
    Assessment & Evaluation in Higher Education  ↗
  17. A coefficient of agreement for nominal scales
    Educational and Psychological Measurement  ↗
  18. Common Core State Standards Initiative. (2010). National governors association center for best practices and …
  19. Combining multiple measures of students' opportunities to develop analytic, text-based wr…
    Educational Assessment  ↗
  20. Assessing students' skills at writing analytically in response to texts
    The Elementary School Journal  ↗
  21. Automated scoring of students’ use of text evidence in writing
    Reading Research Quarterly  ↗
  22. Building a validity argument for an automated writing evaluation system (eRevise) as a fo…
    Computers and Education Open  ↗
  23. Supporting Students’ Text-Based Evidence Use via Formative Automated Writing and Revision…
    The Routledge international handbook of automated essay evaluation
  24. Journal of Writing Research
  25. Effects of the Triple Q Intervention on Argument Writing: Findings from a Small-Scale Clu…
    Journal of Research on Educational Effectiveness  ↗
  26. A review of automated feedback systems for learners: Classification framework, challenges…
    Computers & Education  ↗
  27. Knowledge in pieces
    The Jean Piaget symposium series. Constructivism in the computer age
  28. A friendly introduction to “knowledge in pieces”: Modeling types of knowledge and their r…
    Invited lectures from the 13th international congress on mathematical education
  29. Benchmarking educational llms with analytics: A case study on gender bias in feedback
    arXiv Preprint arXiv
  30. Evidence-based reasoning: Results from an intervention
    Applied Cognitive Psychology  ↗
  31. Contributions and silence in academic talk: Exploring learner experiences of dialogic int…
    Learning, Culture and Social Interaction  ↗
  32. What do we really know about students’ written arguments? Evaluating written argumentatio…
    European Journal of Science and Mathematics Education  ↗
  33. The dialogic mind: A dialogic approach to the higher mental functions
    New ideas in Psychology  ↗
  34. Knowledge of persuasion and writing goals predict the quality of children’s persuasive writing
    Reading and Writing  ↗
  35. Research on revision in writing
    Review of Educational Research  ↗
  36. Executive control in the revising of students with learning and writing difficulties
    Journal of Educational Psychology  ↗
  37. Teaching writing to middle school students: A national survey
    Reading and Writing  ↗
  38. The effects of writing on learning in science, social studies, and mathematics: A meta-analysis
    Review of Educational Research  ↗
  39. An argument-based framework for validating formative assessment in the classroom
    Frontiers in Education  ↗
  40. A Systematic Review of Automated Writing Evaluation Tools in Argumentative Writing for En…
    Research Synthesis in Applied Linguistics
  41. Understanding classroom feedback practices: A study of New Zealand student experiences, p…
    Evaluation and Accountability  ↗
  42. The power of feedback
    Review of Educational Research  ↗
  43. A dialogic path to evidence-based argumentive writing
    Journal of the Learning Sciences  ↗
  44. Teaching writing as reflective practice
  45. A general framework for the validation of embedded formative assessment
    Journal of Educational Measurement  ↗
  46. Negotiating the political and pedagogical tensions of writing rubrics: Using conceptualiz…
    English Education  ↗
  47. Towards Fair Educational Data Mining: A Case Study on Detecting At-Risk Students
  48. Computers and Composition
  49. Articulating a validity argument
    The Routledge handbook of language testing
  50. The importance of dialogic processes to conceptual development in mathematics
    Educational Studies in Mathematics  ↗
  51. The inquiry laboratory as a source for development of metacognitive skills
    International Journal of Science and Mathematics Education  ↗
  52. Teaching writing to high school students: A national survey
    Journal of Educational Psychology  ↗
  53. Journal of Writing Research
  54. Validation of rating processes within an argument-based framework
    Language Testing  ↗
  55. Can language models guess your identity? analyzing demographic biases in ai essay scoring
    Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024)
  56. Calculating and reporting effect sizes to facilitate cumulative science: a practical prim…
    Frontiers in Psychology  ↗
  57. Moral Machines or Tyranny of the Majority? A Systematic Review on Predictive Bias in Educ…
    13th International Learning Analytics and Knowledge Conference
  58. A review of feedback models and theories: Descriptions, definitions, and conclusions
    Frontiers in Education  ↗
  59. Argumentation tasks in secondary English language arts, history, and science: Variations …
    Reading Research Quarterly  ↗
  60. A fairness evaluation of automated methods for scoring text evidence usage in writing
    International Conference on Artificial Intelligence in Education
  61. Liu, Z., Litman, D., Wang, E., Li, T., Gobat, M., Matsumura, L. C., & Correnti, R. (2025). eRevise+ RF: A Wri…
     ↗
  62. Liu, Z., Litman, D., Wang, E., Matsumura, L., & Correnti, R. (2023, July). Predicting the quality of revision…
     ↗
  63. Knowledge of revision and revising behavior among students with learning disabilities
    Learning Disability Quarterly  ↗
  64. Classroom writing tasks and students' analytic text‐based writing
    Reading Research Quarterly  ↗
  65. Assessing Writing
  66. Deepfakes, intellectual cynics, and the cultivation of digital sensibility
    Royal Institute of Philosophy Supplements  ↗
  67. Metatalk: Enabling metalinguistic discussion about writing
    International Journal of Educational Research  ↗
  68. Writing to learn
    Handbook of Writing Research
  69. Teaching and learning argumentative reading and writing: A review of research
    Reading Research Quarterly  ↗
  70. Argumentation, dialogue theory, and probability modeling: Alternative frameworks for argu…
    Educational Psychologist  ↗
  71. It takes two to tango: How scientific reasoning and self-regulation processes impact argu…
    Journal of the Learning Sciences  ↗
  72. Estimating the reproducibility of psychological science
    Science  ↗
  73. Journal of Writing Research
  74. Assessing Writing
  75. A socio-cultural theorisation of formative assessment
    Oxford Review of Education  ↗
  76. Automated writing evaluation for formative assessment of second language writing: investi…
    Educational Psychology  ↗
  77. Student thought and classroom language: Examining the mechanisms of change in dialogic teaching
    Educational Psychologist  ↗
  78. Measuring production and comprehension of written arguments in upper-elementary grades
    Studia Paedagogica  ↗
  79. Roschelle, J., Lester, J. & Fusco, J. (Eds.) (2020). AI and the future of learning: Expert panel report [Repo…
     ↗
  80. Formative assessment and the design of instructional systems
    Instructional Science  ↗
  81. Beyond feedback: Developing student capability in complex appraisal
    Approaches to assessment that enhance learning in higher education
  82. Knowledge telling and knowledge transforming in written composition
    Advances in Applied Psycholinguistics
  83. The language of schooling: A functional linguistics perspective
  84. Commentary: Evaluating the validity of formative and interim assessment
    Educational Measurement: Issues and Practice  ↗
  85. Classroom assessment to support teaching and learning
    The ANNALS of the American Academy of Political and Social Science  ↗
  86. Focus on formative feedback
    Review of Educational Research  ↗
  87. The challenge of academic language. The Cambridge handbook of literacy
  88. Institutional ecology,translations' and boundary objects: Amateurs and professionals in B…
    Social Studies of Science  ↗
  89. Dialogic feedback and potentialities for student learning
    Learning, Culture and Social Interaction  ↗
  90. Written Communication
  91. Digital support for academic writing: A review of technologies and pedagogies
    Computers & Education  ↗
  92. Building Metacognitive Skills Through Social Learning: Vygotsky’s Framework
  93. Argumentation
  94. Interaction between learning and development
    Mind in society
  95. Exploring the Effects of an Automated Writing Evaluation Tool on Metacognitive Engagement…
    International Journal of Learning, Teaching and Educational Research  ↗
  96. Assessing Writing
  97. Llms are biased teachers: Evaluating llm bias in personalized education
    Findings of the Association for Computational Linguistics: NAACL 2025
  98. Semiotic mediation, dialogue and the construction of knowledge
    Human Development  ↗
  99. The semiotic mediation of mental life: LS Vygotsky and MM Bakhtin
    Semiotic mediation
  100. The primacy of mediated action in sociocultural studies
    Mind, Culture, and Activity
  101. Mind as action
  102. Professional development in classroom discussion to improve argumentation: Teacher and st…
    Learning and Instruction  ↗
  103. Handbook of research on technology tools for real-world skill development
    Using automated Feedback to improve Writing quality: Opportunities and challenges
  104. Automated feedback and automated scoring in the elementary grades: Usage, attitudes, and …
    International Journal of Artificial Intelligence in Education  ↗
  105. Automated writing evaluation and feedback: Multiple metrics of efficacy
    Journal of Educational Computing Research  ↗
  106. Zhang, H., Magooda, A., Litman, D., Correnti, R., Wang, E., Matsmura, L. C., . & Quintana, R. (2019, July). e…
     ↗