Journal of Writing Analytics Jan 2017 Open Access

Applying Natural Language Processing Tools to a Student Academic Writing Corpus: How Large are Disciplinary Differences Across Science and Engineering Fields?

Scott A. Crossley Georgia State University ; David R. Russell Iowa State University ; Kristopher Kyle University of Hawaii System ; Ute R�mer Georgia State University

Abstract

• Background: Researchers have been working towards better understanding differences in professional disciplinary writing (e.g., Ewer & Latorre, 1969; Hu & Cao, 2015; Hyland, 2002; Hyland & Tse, 2007) for decades. Recently, research has taken important steps towards understanding disciplinary variation in student writing. Much of this research is corpus-based and focuses on lexico-grammatical features in student writing as captured in the British Academic Written English (BAWE) corpus and the Michigan Corpus of Upper-level Student Papers (MICUSP). The present study extends this work by analyzing lexical and cohesion differences among disciplines in MICUSP. Critically, we analyze not only linguistic differences in macro-disciplines (science and engineering), but also in micro-disciplines within these macro-disciplines (biology, physics, industrial engineering, and mechanical engineering).\n• Literature Review: Hardy and Römer (2013) used a multidimensional analysis to investigate linguistic differences across four macro-disciplines represented in MICUSP. Durrant (2014, in press) analyzed vocabulary in texts produced by student writers in the BAWE corpus by discipline and level (year) and disciplinary differences in lexical bundles. Ward (2007) examined lexical differences within micro-disciplines of a single discipline.\n• Research Questions: The research questions that guide this study are as follows:\n1. Are there significant lexical and cohesive differences between science and engineering student writing? 2. Are there significant lexical and cohesive differences between micro-disciplines within science and engineering student writing?\n• Research Methodology: To address the research questions, student-produced science and engineering texts from MICUSP were analyzed with regard to lexical sophistication and textual features of cohesion. Specifically, 22 indices of lexical sophistication calculated by the Tool for the Automatic Analysis of Lexical Sophistication (TAALES; Kyle & Crossley, 2015) and 38 cohesion indices calculated by the Tool for the Automatic Analysis of Cohesion (TAACO; Crossley, Kyle, & McNamara, 2016) were used. These features were then compared both across science and engineering texts (addressing Research Question 1) and across micro-disciplines within science and engineering (biology and physics, industrial and mechanical engineering) using discriminate function analyses (DFA).\n• Results: The DFAs revealed significant linguistic differences, not only between student writing in the two macro-disciplines but also between the micro-disciplines. Differences in classification accuracy based on students’ years of study hovered at about 10%. An analysis of accuracies of classification by paper type found they were similar for larger and smaller sample sizes, providing some indication that paper type was not a confounding variable in classification accuracy.\n• Discussion: The findings provide strong support that macro-disciplinary and micro-disciplinary differences exist in student writing in these MICUSP samples and that these differences are likely not related to student level or paper type. These findings have important implications for understanding disciplinary differences. First, they confirm previous research that found the vocabulary used by different macro-disciplines to be “strikingly diverse” (Durrant, 2015), but they also show a remarkable diversity of cohesion features. The findings suggest that the common understanding of the STEM disciplines as “close” bears reconsideration in linguistic terms. Second, the lexical and cohesion differences between micro-disciplines are large enough and consistent enough to suggest that each micro-discipline can be thought of as containing a unique linguistic profile of features. Third, the differences discerned in the NLP analysis are evident at least as early as the final year of undergraduate study, suggesting that students at this level already have a solid understanding of the conventions of the disciplines of which they are aspiring to be members. Moreover, the differences are relatively homogeneous across levels, which confirms findings by Durrant (2015) but, importantly, extends these findings to include cohesion markers.\n• Conclusions: The findings from this study provide evidence that macro-disciplinary and micro-disciplinary differences at the linguistic level exist in student writing, not only in lexical use but also in text cohesion. A number of pedagogical applications of writing analytics are proposed based on the reported findings from TAALES and TAACO. Further studies using different corpora (e.g., BAWE) or purpose assembled corpora are suggested to address limitations in the size and range of text types found within MICUSP. This study also points the way toward studies of disciplinary differences using NLP approaches that capture data which goes beyond the lexical and cohesive features of text, including the use of part-of-speech tags, syntactic parsing, indices related to syntactic complexity and similarity, rhetorical features, or more advanced cohesion metrics (latent semantic analysis, latent Dirichlet allocation, Word2Vec approaches).

Journal
Journal of Writing Analytics
Published
2017-01-01
DOI
10.37514/jwa-j.2017.1.1.04
CompPile
Open Access
OA PDF Gold
Topics
Export
BibTeX RIS

Citation context

Cited by in this index (4)

  1. Yoon et al. (2020)
    Quantifying Disciplinary Voices: An Automated Approach to Interactional Metadis…
    Written Communication
  2. Palmquist (2019)
    Directions in Writing Analytics: Some Suggestions
    Journal of Writing Analytics
  3. Lang et al. (2019)
    A Taxonomy for Writing Analytics
    Journal of Writing Analytics
  4. Jin (2018)
    A Multidimensional Analysis of Research Article Discussion Sections in the Fiel…
    IEEE Transactions on Professional Communication

References (56) · 2 in this index

  1. Ädel, A., & Römer, U. (2012). Research on advanced student writing across disciplines and levels: Introducing…
     ↗
  2. BAZERMAN (1985)
    Physicists Reading Physics: Schema-Laden Purposes and Purpose-Laden Schema
    Written Communication
  3. Bazerman, C., & Paradis, J. (1991). Textual dynamics: Historical & contemporary studies of writing in profess…
  4. Biber, D. (1988). Variation across speech and writing. Cambridge: Cambridge University Press.
     ↗
  5. Biber, D. (2006). Stance in spoken and written university registers. Journal of English for Academic Purposes…
     ↗
Show all 56 →
  1. Biber, D., Conrad, S., Reppen, R., Byrd, P., Helt, M., Clark, V., Cortes, V., Csomay, E., & Urzua, A. (2004).…
  2. Brown, G. D. A. (1984). A frequency count of 190,000 words in the London- Lund Corpus of English Conversation…
     ↗
  3. Brysbaert, M., & New, B. (2009). Moving beyond Kucera and Francis: A critical evaluation of current word freq…
     ↗
  4. Brysbaert, M., Warriner, A. B., & Kuperman, V. (2014). Concreteness ratings for 40 thousand generally known E…
     ↗
  5. Chen, Y.-H., & Baker, P. (2010). Lexical bundles in L1 and L2 academic writing. Language Learning and Technol…
     ↗
  6. Coltheart, M. (1981). The MRC psycholinguistic database. Quarterly Journal of Experimental Psychology, 33A, 4…
     ↗
  7. Cortes, V. (2004). Lexical bundles in published and student writing in history and biology. English for Speci…
     ↗
  8. Coxhead, A. (2000). A new academic word list. TESOL Quarterly, 34(2), 213- 238.
     ↗
  9. Crossley, S. A., Allen, L. K., Kyle, K., & McNamara, D.S. (2014). Analyzing discourse processing using the Si…
     ↗
  10. Crossley, S. A., Cai, Z., & McNamara, D. S. (2012). Syntagmatic, paradigmatic, and automatic n-gram approache…
  11. Crossley, S. A., Kyle, K., & McNamara, D. S. (in press). The Tool for the Automatic Analysis of Text Cohesion…
  12. Crossley, S. A., & McNamara, D. S. (2009). Computationally assessing lexical differences in L1 and L2 writing…
     ↗
  13. Crossley, S. A., & McNamara, D. S. (2013). Applications of text analysis tools for spoken response grading. L…
     ↗
  14. Crossley, S. A., Salsbury, T., McNamara, D. S., & Jarvis, S. (2011). Predicting lexical proficiency in langua…
     ↗
  15. Durrant, P. (2014). Discipline-and level-specificity in university students' written vocabulary. Applied Ling…
     ↗
  16. Ewer, J. R., & Latorre, G. (1969). A course in basic scientific English. London: Longman.
  17. Flowerdew, L. (2014). Learner corpus research in EAP: Some core issues and future pathways. English Language …
     ↗
  18. Gieryn, T. (1983). Boundary-work and the demarcation of science from non- science: Strains and interests in p…
     ↗
  19. Grabowski, L. (2015). Keywords and lexical bundles within English pharmaceutical discourse: A corpus-driven d…
     ↗
  20. Halliday, M. A. K., & Hasan, R. (1976). Cohesion in English. London: Longman.
  21. Hardy, J. A., & Römer, U. (2013). Revealing disciplinary variation in student writing: A multi-dimensional an…
     ↗
  22. Harwood (2006)
    (In)appropriate Personal Pronoun Use in Political Science: A Qualitative Study …
    Written Communication
  23. Hu, G., & Cao, F. (2015). Disciplinary and paradigmatic influences on interactional metadiscourse in research…
     ↗
  24. Hyland, K. (2002). Directives: Argument and engagement in academic writing. Applied Linguistics, 23, 215-239.
     ↗
  25. Hyland, K. (2008). As can be seen: Lexical bundles and disciplinary variation. English for Specific Purposes,…
     ↗
  26. Hyland, K., & Tse, P. (2009). The leading journal in its field: Evaluation in journal descriptions. Discourse…
     ↗
  27. Krueger, D., & Kumar, K. B. (2004). Skill-specific rather than general education: A reason for US-Europe grow…
     ↗
  28. Kucera H., & Francis, W. N. (1967). Computational analysis of present-day American English. Providence, RI: B…
  29. Kuperman, V., Stadthagen-Gonzales, H., & Brysbaert, M. (2012). Age-of- acquisition ratings for 30 thousand En…
     ↗
  30. Kyle, K., & Crossley, S. A. (2015). Automatically assessing lexical sophistication: indices, tools, findings,…
     ↗
  31. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, …
     ↗
  32. Larson-Hall, J. (2010). A guide to doing statistics in second language research using SPSS. New York: Routledge.
     ↗
  33. Louwerse, M.M. (2001). An analytic and cognitive parameterization of coherence relations. Cognitive Linguisti…
     ↗
  34. McCarthy, P.M., & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches t…
     ↗
  35. McNamara, D.S., Graesser, A.C., McCarthy, P., & Cai, Z. (2014). Automated evaluation of text and discourse wi…
     ↗
  36. McNamara, D. S., Kintsch, E., Songer, N. B., & Kintsch, W. (1996). Are good texts always better? Text coheren…
     ↗
  37. McNamara, D.S., Louwerse, M.M., McCarthy, P.M., & Graesser, A.C. (2010). Coh-Metrix: Capturing linguistic fea…
     ↗
  38. Nesi, H., & Gardner, S. (2012). Genres across the disciplines: Student writing in higher education. New York,…
     ↗
  39. O'Donnell, M. B., & Römer, U. (2012). From student hard drive to web corpus (part 2): The annotation and onli…
     ↗
  40. Osborne, J., & Dillon, J. (2008). Science education in Europe: Critical reflections. London: The Nuffield Fou…
  41. Ozturk, I. (2007). The textual organisation of research article introductions in applied linguistics: Variabi…
     ↗
  42. Römer, U., & O'Donnell, M. B. (2011). From student hard drive to web corpus (part 1): The design, compilation…
     ↗
  43. Russell, D. R. (2002). Writing in the academic disciplines: A curricular history. Carbondale IL: Southern Ill…
  44. Simpson-Vlach, R., & Ellis, N. C. (2010). An academic formulas list: New methods in phraseology research. App…
     ↗
  45. Sinclair, J. M. (1999). The lexical item. In E. Weigand (Ed.), Contrastive lexical semantics (pp. 1-24). Amst…
     ↗
  46. Swales, J. M. (1990). Genre analysis. English in academic and research settings. Cambridge, UK: Cambridge Uni…
  47. Tabachnick, B. G., & Fidell, L. S. (2001). Using multivariate statistics, fourth edition. Needham Heights, MA…
  48. Thorndike, E. L., & Lorge, I. (1944). The teacher's word book of 30,000 words. New York: Teachers College, Co…
  49. Toglia, M. P., & Battig, W. R. (1978). Handbook of semantic word norms. New York: Lawrence Erlbaum.
  50. Ward, J. (2007). Collocation and technicality in EAP Engineering. Journal of English for Academic Purposes, 6…
     ↗
  51. Zipf, G. K. (1935). The psycho-biology of language: An introduction to dynamic philology. Cambridge, MA: The …
CrossRef global citation count: 17 View in citation network → Build reading path →

Related articles

  1. Res Rhetorica Jan 2026
    Review/Recenzja: Nancy Organ. 2024. Data Visualization for People of All Ages. Oxon: CRC Press; and Jen Christiansen. 2023. Building Science Graphics: An Illustrated Guide to Communicating Science Through Diagrams and Visualizations. Oxon: CRC Press ↗
    Ewa Modrzejewska
    modern rhetorical theory rhetorical criticism discourse analysis digital rhetoric multimodality grammar and mechanics race and writing book reviews
  2. Technical Communication Quarterly Oct 2024
    Editors’ Use of Comprehensive Style Guides: The Case of Singular They ↗
    Jo Mackiewicz; Shaya Kraut; Allison Durazzi
    rhetorical criticism discourse analysis first-year composition writing pedagogy graduate education teacher development writing centers technical communication professional writing digital rhetoric multilingual writers grammar and mechanics literacy studies race and writing public rhetoric editorial matter
  3. Pedagogy Jan 2024
    The Radical Role of Student Writing in Composition ↗
    Jessica Masterson
    modern rhetorical theory rhetorical criticism genre theory discourse analysis cultural rhetorics first-year composition writing pedagogy graduate education teacher development argument collaborative writing professional writing grammar and mechanics literacy studies race and writing gender and writing
  4. Literacy in Composition Studies May 2023
    Erec Smith's A Critique of Anti-Racism in Rhetoric and Composition : A Book Review ↗
    Joshua Barnes
    rhetorical criticism african american rhetorics first-year composition writing pedagogy teacher development argument grammar and mechanics race and writing book reviews
  5. Pedagogy Jan 2022
    Contributors ↗
    modern rhetorical theory rhetorical criticism genre theory discourse analysis african american rhetorics decolonial rhetorics first-year composition writing pedagogy basic writing writing across the curriculum graduate education teacher development argument collaborative writing transfer assessment portfolios writing program administration writing centers peer tutoring technical communication professional writing archival research digital rhetoric social media grammar and mechanics literacy studies race and writing gender and writing disability studies public rhetoric community literacy literary studies editorial matter