Written Communication Oct 2024

Dense and Disconnected: Analyzing the Sedimented Style of ChatGPT-Generated Text at Scale

Ben Markey Carnegie Mellon University ; David West Brown Carnegie Mellon University ; Michael Laudenbach New Jersey Institute of Technology ; Alan Kohler

Abstract

ChatGPT and other LLMs are at the forefront of pedagogical considerations in classrooms across the academy. Many studies have spoken to the technology’s capacity to generate one-off texts in a variety of genres. This study complements those by inquiring into its capacity to generate compelling texts at scale. In this study, we quantitatively and qualitatively analyze a small corpus of generated texts in two genres and gauge it against novice and published academic writers along known dimensions of linguistic variation. Theoretically, we position and historicize ChatGPT as a writing technology and consider the ways in which generated text may not be congruent with established trajectories of writing development in higher education. Our study found that generated texts are more informationally dense than authored texts and often read as dialogically closed, “empty,” and “fluffy.” We close with a discussion of potentially explanatory linguistic features, as well as relevant pedagogical implications.

Journal
Written Communication
Published
2024-10-01
DOI
10.1177/07410883241263528
CompPile
Open Access
Closed
Topics
Export
BibTeX RIS

Citation context

Cited by in this index (3)

  1. Gallagher et al. (2026)
    Crafting and designing AI-simulated audiences in the writing classroom
    Computers and Composition
  2. Si&acirc et al. (2026)
    Curating a Corpus: A Three-Phase Model
    Journal of Writing Analytics
  3. Jiang et al. (2025)
    Does ChatGPT Write Like a Student? Engagement Markers in Argumentative Essays
    Written Communication

References (62) · 14 in this index

  1. Appleman (2023)
    A Brave New World Requires Courage: New Directions for Literacy Research and Teaching
    Research in the Teaching of English
  2. 10.1057/9781137350466
     ↗
  3. Aull (2019)
    Linguistic Markers of Stance and Genre in Upper-Level Student Writing
    Written Communication
  4. Aull L. (2020)
    How students write
  5. Aull et al. (2014)
    Linguistic Markers of Stance in Early and Advanced Academic Writing: A Corpus-b…
    Written Communication
Show all 62 →
  1. Aull et al. (2020)
    From Cow Paths to Conversation
    Pedagogy
  2. 10.1016/j.jeap.2017.01.005
     ↗
  3. 10.37514/PER-B.2013.4791
     ↗
  4. 10.1145/3442188.3445922
     ↗
  5. 10.1017/CBO9780511621024
     ↗
  6. 10.1093/oso/9780195083644.003.0003
     ↗
  7. 10.1017/9781108686136
     ↗
  8. 10.1017/CBO9780511920776
     ↗
  9. 10.5054/tq.2011.244483
     ↗
  10. 10.1093/applin/amu059
     ↗
  11. Clayson (2018)
    Distributed Cognition and Embodiment in Text Planning: A Situated Study of Coll…
    Written Communication
  12. Byrd A. (2023). Truth-telling: Critical inquiries on LLMs and the corpus texts that train them. Composition S…
  13. 10.1016/j.rmal.2023.100068
     ↗
  14. Ching (2018)
    Tools Matter: Mediated Writing Activity in Alternative Digital Environments
    Written Communication
  15. Clayson (2018)
    Distributed Cognition and Embodiment in Text Planning: A Situated Study of Coll…
    Written Communication
  16. Conrad D. (2001)
    Variation in English
  17. 10.1017/CBO9781139107327
     ↗
  18. DeJeu (2024)
    Topoi of Nonprofit Proposal Writing
    Business and Professional Communication Quarterly
  19. Devitt A. J. (2008)
    Writing genres
  20. 10.18653/v1/N19-1423
     ↗
  21. 10.3998/mpub.9708722
     ↗
  22. Eckstein et al. (2022)
    How Does the Language Control of L1 and L2 Writers Develop Over Time in First-Y…
    Written Communication
  23. Espinoza J. (2015, May 26). Erasers are an ‘instrument of the devil’ which should be banned, says academic. T…
  24. Firth J. R. (1957)
    Papers in linguistics, 1934-1951
  25. 10.1177/107769589304700403
     ↗
  26. 10.1016/S0889-4906(02)00017-0
     ↗
  27. 10.1016/j.jslw.2014.09.007
     ↗
  28. Gabrial B. (2007)
    Handbook of research on writing
  29. Gardner S. (2015)
    Corpus linguistics: Abstract book
  30. Gerrard L. (1989)
    Critical perspectives on computers and composition instruction
  31. Giltrow J. (2021)
    Academic writing
  32. Haas (1989)
    How the Writing Medium Shapes the Writing Process: Effects of Word Processing o…
    Research in the Teaching of English
  33. Haas C. (1995)
    Writing technology: Studies on the materiality of literacy
  34. 10.1109/MIS.2009.36
     ↗
  35. Halliday M. A. K. (1976)
    Cohesion in English
  36. 10.3366/cor.2013.0040
     ↗
  37. Holland D., Cole M. (1995). Between discourse and schema: Reformulating a cultural-historical approach to cul…
     ↗
  38. 10.1177/1461445605050365
     ↗
  39. 10.3998/mpub.9426902
     ↗
  40. Lang et al. (2023)
    Computer-Assisted Corpus Analysis: An Introduction to Concepts, Processes, and …
    IEEE Transactions on Professional Communication
  41. 10.1177/0731948715583115
     ↗
  42. 10.4324/9781315197692
     ↗
  43. 10.1162/tacl_a_00548
     ↗
  44. 10.1016/j.jeap.2013.12.001
     ↗
  45. 10.1080/09500789309541347
     ↗
  46. 10.18653/v1/N18-1202
     ↗
  47. Prior P. (1998)
    Writing/disciplinarity
  48. Radford A. (2018)
    Improving language understanding with unsupervised learning
  49. Russell D. R. (1991)
    Writing in the academic disciplines: A curricular history
  50. Selfe (1999)
    Technology and Literacy: A Story about the Perils of Not Paying Attention
    College Composition and Communication
  51. Stanton C. (2023). A dis-facilitated call for more writing studies in the new AI landscape; or, Finding our p…
     ↗
  52. Staples et al. (2016)
    Academic Writing Development at the University Level: Phrasal and Clausal Compl…
    Written Communication
  53. Sterponi et al. (2017)
    Endangered Literacies? Affordances of Paper-Based Literacy in Medical Practice …
    Written Communication
  54. S. V. G. (1908). Do your pupils use erasers? Western Teacher: Devoted to Schoolroom Methods. Practical Aids a…
  55. Vee A. (2023). Large language models write answers. Composition Studies, 51(1), 176–181. https://compositions…
  56. 10.37514/TWR-J.2023.1.1.02
     ↗
  57. Weaver W. (1949)
    Translation
CrossRef global citation count: 18 View in citation network → Build reading path →

Related articles

  1. Business and Professional Communication Quarterly Sep 2026
    Instructional Control for AI-Assisted Learning: Custom AI Tools for Genre-Based Writing ↗
    Junhua Wang
    genre theory writing pedagogy teacher development professional writing artificial intelligence
  2. Journal of Response to Writing Apr 2026
    Examining Automated Writing Evaluation Error Coverage in Relation to Uptake and Retention ↗
    Mehrzad, Mohaddeseh; Rahimi, Mohammad; Link, Stephanie
    genre theory writing pedagogy teacher development revision assessment artificial intelligence multilingual writers grammar and mechanics
  3. Pedagogy Apr 2026
    Amplifying “Language in Action” ↗
    Carly Schnitzler
    genre theory first-year composition writing pedagogy revision artificial intelligence
  4. Journal of Academic Writing Dec 2025
    Student Evaluative Judgements of Writing and Artificial Intelligence: The Disconnect between Structural and Conceptual Knowledge ↗
    Christopher Eaton; Kaitlyn Harris; Erin Vearncombe
    genre theory writing pedagogy graduate education artificial intelligence
  5. Pedagogy Oct 2025
    Contributors ↗
    modern rhetorical theory rhetorical criticism genre theory cultural rhetorics first-year composition writing pedagogy advanced composition creative writing writing across the curriculum graduate education two-year college service learning teacher development technical communication professional writing labor and working conditions archival research multimodality artificial intelligence literacy studies race and writing gender and writing disability studies literary studies editorial matter