Journal of Writing Analytics Jan 2017 Open Access

(Re)Visualizing Rater Agreement: Beyond Single-Parameter Measures

David Eubanks Furman University

Abstract

Technique Identification: A new graphical technique is presented for visualizing and assessing inter-rater agreement in discrete ordinal or categorical data, such as rubric ratings.  To that aim, a chance-corrected Kappa with two new features is derived. First, it is based on interpreting ratings for each subject as vectors to visualize the data. This is done by creating two-dimensional vectors from a subject-rating summary table, sorting the vectors by their slopes, and plotting them in that order to create a trajectory that displays all the data in context. Second, it presents a graph and accompanying statistics (Kappa, p -value) for each pair of ratings in an organized display so that all useful comparisons of the data are visually displayed and statistically assessed. This information is presented on a logical grid, usually called facets . Kappa is calculated in the usual way, by referencing the actual results with an average of random rating assignments. This average becomes a reference line on each graph as a visual cue, as well. The statistical basis for the Kappa and significance testing are derived, and the test assumptions are specified. Value Contribution: The most commonly used statistics for inter-rater agreement, such as the Cohen Kappa or Inter-Class Correlation, give only a single parameter estimate of reliability from which to make judgments about ratings data. The technique presented here constructs graphs of all the data that allow visual inspection of the ratings versus a reference curve that represents chance-matching. The detailed reports on inter-rater agreement can show how to fine-tune ratings systems, such as understanding which parts of an ordinal scale are working best. This solves a practical problem for researchers who rely on rating-type classification by revealing which overall aspects of the rating system need to be improved and adds to the list of tools available for assessing rating reliability. In creating this approach to analysis of rater data, human usability is emphasized. Specifically, the use of geometry is designed to facilitate interpretability rather than being a mathematical derivation from first principles. Technique Application: Two applications are given, both involving social meaning-making. The first uses data from wine-judging to illustrate how the method can illuminate expertise in that domain. The results reproduce published findings that were based on a classical statistical method. A second sample application uses data from a university assessment of student writing in which ratings on a developmental scale are assigned by course instructors to their students. The rating program is an example of social meaning-making that can be used to generate larger data sets than are typical for classroom-based assessment programs. The analysis shows the strengths and weaknesses of the rating system in terms of reliability and demonstrates how that knowledge leads to improvements in assessment. Directions for Further Research: An argument is made for a public library of inter-rater data for empirical use by researchers. The social aspects of rating are discussed, and there is an illustration of the potential to derive new measures of inter-rater agreement from the meaning-making program that produces the data.

Journal
Journal of Writing Analytics
Published
2017-01-01
DOI
10.37514/jwa-j.2017.1.1.10
CompPile
Open Access
OA PDF Gold
Topics
Export
BibTeX RIS

Citation context

Cited by in this index (2)

  1. Rose et al. (2020)
    A Line Matching Method for Reliable Higher-Order Theme Identification
    Journal of Writing Analytics
  2. Eubanks et al. (2020)
    Divergent Writer Development in College
    Journal of Writing Analytics

References (32) · 1 in this index

  1. Agresti, A. (1988). A model for agreement between ratings on an ordinal scale. Biometrics, 44(2), 539-548.
     ↗
  2. Agresti, A. (2013). Categorical data analysis (3rd ed.). Hoboken, NJ: John Wiley & Sons.
  3. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 2…
     ↗
  4. Conger, A. J. (1980). Integration and generalization of kappas for multiple raters. Psychological Bulletin, 8…
     ↗
  5. Elliot, N., Plata, M., & Zelhart, P. F. (1990). A program development handbook for the holistic assessment of…
Show all 32 →
  1. Eubanks, D. A. (2008). Assessing the general education elephant. Assessment Update, 20(4), 4.
  2. Eubanks, D. A. (2016a) Interrater facets. Github Repository. http://github.com/stanislavzza/Inter-Rater-Facets
  3. Eubanks, D. A. (2016b) Rethinking interrater agreement. Assessment Update, 28(4), 8-14. DOI: 10.1002/au.30065
     ↗
  4. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378.
     ↗
  5. Fleiss, J. L., Cohen, J., & Everitt, B. (1969). Large sample standard errors of kappa and weighted kappa. Psy…
     ↗
  6. Fleiss, J. L., Levin, B., & Paik, M. C. (2003). Statistical methods for rates and proportions (3rd ed.). Hobo…
     ↗
  7. Galton, F. (1892). Finger prints. London: Macmillan and Company.
     ↗
  8. Gamer, M., Lemon, J., & Singh, I. F. P. (2012). Irr: Various coefficients of interrater reliability and agree…
  9. Gwet, K. (2015). Testing the difference of correlated agreement coefficients for statistical significance. Ed…
     ↗
  10. Gwet, K. L. (2014). Handbook of inter-rater reliability:Tthe definitive guide to measuring the extent of agre…
  11. Haertel, E. H. (2006). Reliability. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 65-110). We…
  12. Hodgson, R. T. (2008). An examination of judge reliability at a major US wine competition. Journal of Wine Ec…
     ↗
  13. Kelly Riley, D., & Whithaus, C. (2016). A theory of ethics for writing assessment. [Special issue]. Journal o…
  14. Landis, J. R. and Koch, G. G. (1977) The measurement of observer agreement for categorical data. Biometrics, …
     ↗
  15. Lane, S, & Stone, C. (2006), Performance Assessment. In R. L. Brennan (Ed.), Educational measurement (4th ed.…
  16. Messick, S. (1986). The once and future issues of validity: Assessing the meaning and consequences of measure…
     ↗
  17. Moss, P. A. (2004). The meaning and consequences of "reliability". Journal of Educational and Behavioral Stat…
     ↗
  18. Moss, P. A., Pullin, D. C., Gee, J. P., Haertel, E. H., & Young, L. J. (Eds.). (2008). Assessment, equity, an…
     ↗
  19. National Research Council. (2012). Education for life and work: Developing transferable knowledge and skills …
  20. Powers, D. M. (2012). The problem with kappa. In Proceedings of the 13th Conference of the European Chapter o…
  21. R Core Team. (2015). R: A language and environment for statistical computing [Computer software manual]. Vien…
  22. Rezaei et al. (2010)
    Reliability and validity of rubrics for assessment through writing
    Assessing Writing
  23. Roberts, C. (2008). Modelling patterns of agreement for nominal scales. Statistics in Medicine, 27(6), 810-830.
     ↗
  24. Smeeton, N. C. (1985). Early history of the kappa statistic. International Biometric Society, 41(3).
  25. Stanovich, K. E. (1986). Matthew effects in reading: Some consequences of individual differences in the acqui…
     ↗
  26. Stemler, S. E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating inter…
  27. Wang, J., Engelhard, G., Raczynski, K., Song, T., & Wolfe, E. W. (2017). Evaluating rater accuracy and percep…
CrossRef global citation count: 4 View in citation network → Build reading path →

Related articles

  1. Journal of Technical Writing and Communication Mar 2026
    Canon to Code: Rhetorical Rulemaking for Generative AI Content Audits and Governance ↗
    Miriam F. Williams
    classical rhetoric modern rhetorical theory rhetorical criticism assessment technical communication artificial intelligence
  2. Pedagogy Apr 2024
    Contributors ↗
    modern rhetorical theory rhetorical criticism feminist rhetorics first-year composition writing pedagogy creative writing teacher development collaborative writing assessment technical communication professional writing grammar and mechanics literacy studies gender and writing literary studies book reviews editorial matter
  3. IEEE Transactions on Professional Communication Dec 2023
    The State of UX Pedagogy ↗
    Philip B. Gallagher; Guiseppe Getto
    modern rhetorical theory rhetorical criticism discourse analysis writing pedagogy teacher development argument assessment technical communication book reviews
  4. Journal of Technical Writing and Communication Jan 2022
    Changing Climate, Changing Terrain: The Stasis Metaphor and the Climate Crisis ↗
    Shelton Weech
    classical rhetoric modern rhetorical theory rhetorical criticism argument assessment technical communication
  5. Pedagogy Jan 2022
    Contributors ↗
    modern rhetorical theory rhetorical criticism genre theory discourse analysis african american rhetorics decolonial rhetorics first-year composition writing pedagogy basic writing writing across the curriculum graduate education teacher development argument collaborative writing transfer assessment portfolios writing program administration writing centers peer tutoring technical communication professional writing archival research digital rhetoric social media grammar and mechanics literacy studies race and writing gender and writing disability studies public rhetoric community literacy literary studies editorial matter