Assessing Writing Jul 2026 Open Access

Developing a rating scale for written intralinguistic mediation in a local context

Caroline Shackleton Instituto de Investigación Biosanitaria de Granada ; Nathan Turner Instituto de Investigación Biosanitaria de Granada ; Adolfo Sánchez Cuadrado Universidad de Granada

Abstract

Intralinguistic mediation as a task type offers testing bodies an opportunity to expand the test construct to include authentic ability for use test tasks and is of great relevance as a representation of English lingua franca usage in the higher education context. This study reports on the development of an analytic rating scale for one particular mediation task specification, that of facilitating communication in delicate situations and disagreements , which was co-constructed by raters based on theoretical considerations, example scripts, and CEFR/CV descriptors. Raters used the scale to mark 80 performances across three tasks, and results were analyzed using a many-facet Rasch hybrid partial credit model, which pays particular attention to the functioning of an analytic scale and its categories. Findings show that despite the complexities of this multidimensional construct, it can be operationalized through well-designed tasks and a strategic scale-development process. Following focus group feedback and further reflection on the quantitative results, minor refinements were made to the scale. Findings indicate potential for future scale development for scoring authentic, context-specific task types, and the study has clear implications for other testing bodies hoping to include mediation in their proficiency exams. • European university test re-development project in a lingua franca context. • Developed a rating scale for an intra-linguistic written mediation task specification. • Scale co-constructed by raters using theory, example scripts, and CV descriptors. • Iterative mixed-method approach informed the rating scale development. • New scale shown to be valid, contribution to future mediation assessment.

Journal
Assessing Writing
Published
2026-07-01
DOI
10.1016/j.asw.2026.101049
CompPile
Open Access
OA PDF Hybrid
Topics
Export
BibTeX RIS

Citation context

Cited by in this index (0)

No articles in this index cite this work.

References (85) · 12 in this index

  1. Bachman (2002)
    Some reflections on task-based language performance assessment
    Language Testing  ↗
  2. Bachman (2005)
    Building and supporting a case for test use
    Language Assessment Quarterly  ↗
  3. Berceruelo Pino (2024)
    Developing CEFR-based mediation rating scales for standardized exams at state language sc…
    CEFR Journal - Research and Practice  ↗
  4. Biggs, J., Tang, C., & Kennedy, G. (2022). Teaching for quality learning at university 5e. McGraw-hill educat…
  5. Canagarajah (2006)
    Changing communicative needs, revised assessment objectives: Testing English as an intern…
    Language Assessment Quarterly  ↗
Show all 85 →
  1. Chan et al. (2015)
    Developing rubrics to assess the reading-into-writing skills: A case study
    Assessing Writing
  2. Chan (2023)
    Towards more valid scoring criteria for integrated reading-writing and listening-writing …
    Language Testing  ↗
  3. Chan et al. (2022)
    Integrated writing and its correlates: A meta-analysis
    Assessing Writing
  4. Cheong (2019)
    Effects of intertextual processing on L2 integrated writing
    Journal of Second Language Writing  ↗
  5. Cho et al. (2018)
    Writing from sources: Does audience matter?
    Assessing Writing
  6. Council of Europe, (2020) [2018]. Common European Framework of Reference for Languages: Learning, teaching, a…
  7. Creswell, J.W., & Plano Clark, V.L. (2018). Designing and conducting mixed methods research (3rd ed.). SAGE.
  8. Cumming (2021)
    Assessing academic writing
    Assessing English for Higher Education Admissions
  9. Dendrinos, B. & Karavas, E. (2013). The KPG Handbook: Performance Descriptors and Specifications. Athens: RCe…
  10. Dendrinos (2024)
    Conceptualisation and operationalisation of linguistic mediation as a testing construct: …
    Mediation as Negotiation of meanings, Plurilingualism and Language Education
  11. Deygers (2021)
    The CEFR Companion Volume: Between Research-Based Policy and Policy-Based Research
    Applied Linguistics  ↗
  12. Deygers (2015)
    Determining the scoring validity of a co-constructed CEFR-based rating scale
    Language Testing  ↗
  13. Eckes (2005)
    Examining rater effects in TestDaF writing and speaking performance assessments: A many-F…
    Language Assessment Quarterly  ↗
  14. Eckes (2009)
    Many-facet Rasch measurement
    Reference supplement to the manual for relating language examinations to the Common European Framework of Reference for Languages: Learning, teaching, assessment (Section H)
  15. Elder (2017)
    Interrogating the construct of communicative competence in language assessment contexts: …
    Language & Communication  ↗
  16. Ellis, R. (2003). Task-based language learning and teaching. Oxford, England: Oxford University Press.
     ↗
  17. Fulcher, G. (2003). Testing second language speaking. London: Pearson Longman.
  18. Gadomska (2024)
    Developing mediation skills at the tertiary level of education
    Neofilolog  ↗
  19. Grice (1975)
    Logic and conversation
    Speech acts
  20. Harding (2014)
    Communicative language testing: Current issues and future research
    Language Assessment Quarterly  ↗
  21. Harding (2018)
    Language assessment: The challenge of ELF
    The Routledge handbook of English as a lingua franca
  22. Harsch (2024)
    Usability of CEFR Companion Volume scales for the development of an analytic rating scale…
    CEFR Journal - Research and Practice  ↗
  23. Harsch et al. (2024)
    Validating an integrated reading-into-writing scale with trained university students
    Assessing Writing
  24. Harsch et al. (2012)
    Adapting CEF-descriptors for rating purposes: Validation by a combined rater tr…
    Assessing Writing
  25. Harsch et al. (2020)
    Marrying achievement with proficiency – Developing and validating a local CEFR-…
    Assessing Writing
  26. Huhta (2014)
    Assessing learners’ writing skills in a SLA study: Validating the rating process across t…
    Language Testing  ↗
  27. Karakoҫ et al. (2025)
    Designing a rating scale for an integrated reading-writing test: A needs-orient…
    Assessing Writing
  28. Kane (2013)
    Validating the interpretations and uses of test scores
    Journal of Educational Measurement  ↗
  29. Knoch, U. (2009). Diagnostic writing assessment: the development and validation of a rating scale. Frankfurt:…
  30. Knoch (2021)
    Revisiting rating scale development for rater-mediated language performance assessments: …
    Language Testing  ↗
  31. Knoch et al. (2013)
    A closer look at integrated writing tasks: Towards a more focussed definition f…
    Assessing Writing
  32. Kuiken (2017)
    Functional adequacy in L2 writing: Towards a new rating scale
    Language Testing  ↗
  33. Kuiken (2022)
    Measurement of functional adequacy in different learning contexts: Rationale, key issues,…
    Taskent- Journal on Taskent--Based Language Teaching and Learning  ↗
  34. Lestari (2023)
    Operationalizing the reading-into-writing construct in analytic rating scales: Effects of…
    Language Testing  ↗
  35. Leung (2020)
    Mediating communication – ELF and flexible multilingualism perspectives on the Common Eur…
    Australian Journal of Applied Linguistics  ↗
  36. Linacre (2002)
    Optimizing rating scale category effectiveness
    Journal of Applied Measurement
  37. Linacre, J.M. (2025a). A user’s guide to FACETS: Rasch-model computer programs [Software manual 4.4.0]. 〈http…
  38. Linacre, J.M. (2025b). Winsteps® (Version 5.8.0) [Computer Software]. Portland, Oregon: Winsteps.com. Availab…
  39. Long (2000)
    Task-based language teaching and assessment
    Encyclopedia of language teaching
  40. Lukácsi (2020)
    Developing a level-specific checklist for assessing EFL writing
    Language Testing  ↗
  41. McNamara, T.F. (1996). Measuring second language performance. London and New York: Longman.
  42. McNamara, T.F., Knoch, U., & Fan, J. (2019). Fairness, justice, and language assessment: The role of measurem…
  43. McNamara, T.F. & Roever, C. (2006). Language testing: the social dimension. Blackwell.
  44. Nelson (2023)
    Discourse synthesis: Textual transformations in writing from sources
    Reading and Writing  ↗
  45. Norris (2018)
    Task-based language assessment aligning designs with intended uses and consequences
    JLTA Journal  ↗
  46. North (2003)
    Scales for rating language performance: Descriptive models, formulation styles, and prese…
    Research Monograph Series
  47. North (2020)
    Trolls, unicorns and the CEFR: Precision and professionalism in criticism of the CEFR
    CEFR Journal – Research and Practice  ↗
  48. North (2021)
    The CEFR companion volume—What’s new and what might it imply for teaching/learning and fo…
    CEFR Journal - Research and Practice  ↗
  49. North (2022)
    The CEFR companion volume and the action-oriented approach
    Italiano LinguaDue  ↗
  50. North (2024)
    Developing an action-oriented perspective on mediation. The new CEFR/CV descriptors
    Mediation as negotiation of meanings, plurilingualism and language education
  51. North, B. & Piccardo, E. (2016). Developing illustrative descriptors of aspects of mediation for the CEFR. Ed…
     ↗
  52. North (2022)
    The conceptualisation of mediation in the new CEFR
    Sprachmittlung und Mediation im Fremdsprachenunterricht an Schule und Universität. Sprachen lehren – Sprachen lernen  ↗
  53. O’Grady (2022)
    Developing a rating scale for integrated assessment of reading-into-writing skills
    Language Learning in Higher Education  ↗
  54. Owen (2023)
    Designing and researching an intertextual reading-into-writing summary task
    ECLL Official Conference Proceedings  ↗
  55. Pallotti (2022)
    Holistic and analytic assessment of functional adequacy
    Task  ↗
  56. Park (2019)
    An investigation into rater performance with a holistic scale and a binary, analytic scal…
    Papers in Language Testing and Assessment
  57. Piccardo, E., & North, B. (2019): The Action-oriented Approach: a Dynamic Vision of Language Education. Brist…
     ↗
  58. Piccardo (2019)
    Broadening the Scope of Language Education: Mediation, Plurilingualism, and Collaborative…
    Journal of E-Learning and Knowledge Society
  59. Plakans (2009)
    Discourse synthesis in integrated second language writing assessment
    Language Testing  ↗
  60. Plakans et al. (2017)
    Exploring the relationship of organization and connection with scores in integr…
    Assessing Writing
  61. Safari (2023)
    Developing and evaluating an empirically-based diagnostic checklist for assessing second …
    Journal of Second Language Writing  ↗
  62. Saito (2020)
    Developing an e-portfolio reflecting the concept of mediation for university students
    CEFR Journal - Research and Practice  ↗
  63. Sánchez Cuadrado (2022)
    Procesos de evaluación de la mediación en lenguas extranjeras/segundas lenguas
    Mediación en el aprendizaje de lenguas
  64. Schissel (2018)
    Test Impact and Washback
    The TESOL Encyclopedia of English Language Teaching
  65. Seidlhofer (2004)
    10. Research perspectives on teaching English as a lingua Franca
    Annual Review of Applied Linguistics  ↗
  66. Shackleton (2020)
    Planning for positive washback
    Perspectives on Language Assessment Literacy: Challenges for Improved Student Learning
  67. Shackleton (2021)
    Incorporating English and Spanish mediation tasks into language proficiency exams
    Collated Papers for the ALTE 7th International Conference, Madrid
  68. Stathopoulou (2013)
    Investigating mediation as translanguaging practice in a testing context: Towards the dev…
    Proceedings of the international conference "Language testing in Europe: Time for a new framework?”
  69. Stathopoulou, M. (2015). Cross-Language mediation in foreign language teaching and testing. Multilingual Matters.
     ↗
  70. Stathopoulou (2019)
    The reading-to-write construct across languages: Analysing written mediation tasks and pe…
    Selected Papers on Theoretical and Applied Linguistics
  71. Stathopoulou, M. (2025). The ECML METLA project: Drawing upon the CEFRCV to support learners in reaching thei…
  72. Weigle, S.C. (2002). Assessing writing. Cambridge University Press.
     ↗
  73. Wette, R. (2021) Writing Using Sources for Academic Purposes: Theory, Research and Practice. 1st ed. Routledge.
     ↗
  74. Wind (2020)
    Do raters use rating scale categories consistently across analytic rubric domains in writ…
    Assessing Writing
  75. Wind (2023)
    Detecting rating scale malfunctioning with the partial credit model and generalized parti…
    Educational and Psychological Measurement  ↗
  76. Xie (2023)
    Assessing source use: Summary vs. reading-to-write argumentative essay
    Assessing Writing
  77. Xie (2024)
    Construct representation and predictive validity of integrated writing tasks: A…
    Assessing Writing
  78. Yamanishi (2019)
    Developing a scoring rubric for L2 summary writing: A hybrid approach combining analytic …
    Language Testing in Asia
  79. Yoo (2025)
    Reading-writing connections: A systematic review of second language synthesis writing
    L2 Journal  ↗
  80. Yu (2009)
    The shifting sands in the effects of source text summarizability on summary writing
    Assessing Writing
CrossRef global citation count: 0 View in citation network → Build reading path →

Related articles

  1. Written Communication Jul 2026
    The Dynamics Between the Biographical Factors, Professional Identities, and Teaching Practices of Second Language Writing Instructors ↗
    Young Wha Lee
    first-year composition writing pedagogy graduate education teacher development assessment qualitative research multilingual writers grammar and mechanics
  2. Written Communication May 2026
    Developing an Integrated Logging Method for Process-Based Assessment and Feedback in Technical Writing ↗
    Lin Dong
    genre theory assessment technical communication professional writing qualitative research
  3. IEEE Transactions on Professional Communication Mar 2026
    US Hospital Educators' Technology Needs: A Qualitative Study for Developing Action-Oriented Technology ↗
    Margaret Webb; Sweta Baniya; Alexa Smith; Nadra Rasberry; Ihudiya Ogbonnaya-Ogburu
    modern rhetorical theory teacher development assessment professional writing qualitative research book reviews
  4. IEEE Transactions on Professional Communication Mar 2026
    Surveys as UXR: Using Design Thinking to Shape a Survey-Based UX Assessment for Rural Audiences ↗
    Kaitlin J. Coyle; Derek G. Ross
    assessment technical communication qualitative research literacy studies
  5. Journal of Writing Research Feb 2026
    Using AI to understand students’ self-assessments of their writing ↗
    Madeleine Sorapure; Seth Erickson; Sarah Hirsch; Kenny Smith
    first-year composition assessment qualitative research artificial intelligence