Journal of Writing Analytics Jan 2018 Open Access

Utility-Value Score: A Case Study in System Generalization for Writing Analytics

Beata Beigman Klebanov Educational Testing Service ; Stacy Priniski ; Jill Burstein Educational Testing Service ; Binod Gyawali Educational Testing Service ; Judith Harackiewicz ; Dustin Thoman

Abstract

Collection and analysis of students' writing samples on a large scale is a part of the research agenda of the emerging writing analytics community that promises to deliver an unprecedented insight into characteristics of student writing. Yet with a large scale often comes variability of contexts in which the samples were produced-different institutions, different purposes of writing, different author demographics, to name just a few possible dimensions of variation. What are the implications of such variation for the ability of automated methods to create indices/features based on the writing samples that would be valid and meaningful? This paper presents a case study in system generalization. Building on a system developed to assess the expression of utility value (a social-psychology-based construct) in essays written by first-year biology students at one postsecondary institution, we vary data parameters and observe system performance. From the point of view of social psychology, all these variants represent the same underlying construct (i.e., utility value), and it is thus very tempting to think that an automatically produced utility-value score could provide a meaningful analytic, consistently, on a large collection of essays. However, findings from this research show that there are challenges: Some variations are easier to deal with than others, and some components of the automated system generalize better than others. The findings are then discussed both in the context of the case study and more generally.

Journal
Journal of Writing Analytics
Published
2018-01-01
DOI
10.37514/jwa-j.2018.2.1.13
CompPile
Open Access
OA PDF Gold
Topics
Export
BibTeX RIS

Citation context

Cited by in this index (4)

  1. Donahue et al. (2019)
    Peer Review in Biology: Of Novices, Experts, and Disciplines
    Journal of Writing Analytics
  2. Messina (2019)
    Tracing Fan Uptakes: Tagging, Language, and Ideological Practices in The Legend…
    Journal of Writing Analytics
  3. Anson et al. (2019)
    A Text-Analytic Method for Identifying Text Recycling in STEM Research Reports
    Journal of Writing Analytics
  4. Marcellino (2019)
    Seniority in Writing Studies: A Corpus Analysis
    Journal of Writing Analytics

References (14)

  1. Beigman Klebanov, B., Burstein, J., Harackiewicz, J., Priniski, S., & Mulholland, M. (2016). Enhancing STEM m…
     ↗
  2. Klebanov, B., Burstein, J., Harackiewicz, J., Priniski, S., & Mulholland, M. (2017). Reflective writing about…
     ↗
  3. Blanchard, D., Heilman, M., & Madnani, N. (2013). SciKit-Learn Laboratory. GitHub repository, https://github.…
  4. Breiman, L. (2001). Random forests. Machine Learning, 45, 5-32.
     ↗
  5. Canning, E., Harackiewicz, J., Priniski, S., Hecht, C., Tibbetts, Y., & Hyde, J. (2018). Improving performanc…
     ↗
Show all 14 →
  1. Durik, A. & Harackiewicz, J. (2007). Different strokes for different folks: How personal interest moderates t…
     ↗
  2. Eccles, J., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53, …
     ↗
  3. Harackiewicz, J., Canning, E., Tibbetts, Y., Priniski, S., & Hyde, J. (2016). Closing achievement gaps with a…
     ↗
  4. Hidi, S. & Harackiewicz, J. (2000). Motivating the academically unmotivated: A critical issue for the 21 st c…
     ↗
  5. Hulleman, C., Godes, O., Hendricks, B., & Harackiewicz, J. (2010). Enhancing interest and performance with a …
     ↗
  6. Hulleman, C. & Harackiewicz, J. (2009). Promoting interest and performance in high school science classes. Sc…
     ↗
  7. Hulleman, C., Kosovich, J., Barron, K., & Daniel, D. (2017). Making connections: Replicating and extending th…
     ↗
  8. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P…
  9. Pennebaker, J. W., Booth, R. J., Boyd, R. L., & Francis, M. E. (2015). Linguistic Inquiry and Word Count: LIW…
CrossRef global citation count: 8 View in citation network → Build reading path →

Related articles

  1. Rhetoric Society Quarterly Aug 2026
    Environmental Preservation and the Grey Cliffs Conflict: Negotiating Common Narratives, Values, and Ethos: by Kristin D. Pickering, UP of Colorado, 2024, 268 pp., $33.95 (paperback), ISBN: 978-1-64642-575-4 ↗
    Nathan D. Wood
    argument qualitative research
  2. Argumentation Jul 2026
    Arguments and Proto-arguments in Multimodal Protest ↗
    Lotte van Poppel; Jan Albert van Laar
    argument qualitative research multimodality
  3. Assessing Writing Jul 2026
    Developing a rating scale for written intralinguistic mediation in a local context ↗
    Caroline Shackleton; Nathan Turner; Adolfo Sánchez Cuadrado
    assessment qualitative research
  4. Written Communication Mar 2026
    Faculty and Administrator Perceptions of Interdisciplinary Collaborative Writing: Practices, Challenges, and Support Structures ↗
    Stephen Carradini; Ed Nagelhout
    collaborative writing professional writing labor and working conditions qualitative research
  5. Argumentation Mar 2026
    Is a Contradiction Between Arguments Less Likely to be Noticed When They are Implicit? An Experimental Case Study ↗
    Edoardo Lombardi Vallauri
    argument qualitative research social media