Assessing Writing Jul 2024 Open Access

Construct representation and predictive validity of integrated writing tasks: A study on the writing component of the Duolingo English Test

Qin Xie University of Macau

Abstract

This study examined whether two integrated reading-to-write tasks could broaden the construct representation of the writing component of Duolingo English Test (DET). It also verified whether they could enhance DET’s predictive power of English academic writing in universities. The tasks were (1) writing a summary based on two source texts and (2) writing a reading-to-write essay based on five texts. Both were given to a sample (N = 204) of undergraduates from Hong Kong. Each participant also submitted an academic assignment written for the assessment of a disciplinary course. Three professional raters double-marked all writing samples against detailed analytical rubrics. Raw scores were first processed using Multi-Faceted Rasch Measurement to estimate inter- and intra-rater consistency and generate adjusted (fair) measures. Based on these measures, descriptive analyses, sequential multiple regression, and Structural Equation Modeling were conducted (in that order). The analyses verified the writing tasks’ underlying component constructs and assessed their relative contributions to the overall integrated writing scores. Both tasks were found to contribute to DET’s construct representation and add moderate predictive power to the domain performance. The findings, along with their practical implications, are discussed, especially regarding the complex relations between construct representation and predictive validity. • studied the concepts of construct representation (CR) and predictive validity (PV). • within the context of an AI-facilitated language test (Duolingo English Test). • Revealed the complex relations between CR and PV.

Journal
Assessing Writing
Published
2024-07-01
DOI
10.1016/j.asw.2024.100846
CompPile
Open Access
OA PDF Hybrid
Topics
Export
BibTeX RIS

Citation context

Cited by in this index (2)

  1. Xie et al. (2026)
    Can citations indicate performance levels? An investigation of integrated argum…
    Assessing Writing
  2. Shackleton et al. (2026)
    Developing a rating scale for written intralinguistic mediation in a local context
    Assessing Writing

References (61) · 8 in this index

  1. Abunawas, M. (2014). A meta-analytic investigation of the predictive validity of the Test of English as a For…
  2. Bachman (1991)
    What does language testing have to offer?
    TESOL Quarterly  ↗
  3. Bachman (2007)
    What is the construct? The dialectic of abilities and contexts in defining constructs in …
    Language testing reconsidered
  4. Brenzel, J. & Settles, B. (2017). The Duolingo English Test-Design, Validity, and Value. DET Whitepaper(Short…
     ↗
  5. Bridgeman (2015)
    Predicting grades from an English language assessment: The importance of peeling the onion
    Language Testing  ↗
Show all 61 →
  1. Bruce (2015)
    Opposing tensions of local and international standards for EAP writing programmes: Who ar…
    Journal of English for Academic Purposes  ↗
  2. Chan et al. (2015)
    Developing rubrics to assess the reading-into-writing skills: A case study
    Assessing Writing
  3. Chan et al. (2022)
    Integrated writing and its correlates: A meta-analysis
    Assessing Writing
  4. Cho (2012)
    Relationship of TOEFL iBT scores to academic performance: Some evidence from American uni…
    Language Testing  ↗
  5. Cumming et al. (2005)
    Differences in written discourse in independent and integrated prototype tasks …
    Assessing Writing
  6. Cumming (2016)
    Students' writing from sources for academic purposes: A synthesis of recent research
    Journal of English for Academic Purposes  ↗
  7. Embretson (1983)
    Construct validity: Construct representation versus nomothetic span
    Psychological Bulletin  ↗
  8. Embretson (2006)
    Mixed Rasch models for measurement in cognitive psychology
    Multivariate and mixture distribution Rasch models: Extensions and applications
  9. Fulcher (2007)
    Language testing and assessment: An advance resource book
  10. Galloway (2020)
    The role of cross-disciplinary academic language skills in disciplinary, source-based wri…
    Reading and Writing  ↗
  11. Gardner (2013)
    A classification of genre families in university student writing
    Applied Linguistics  ↗
  12. Gebril (2009)
    Investigating source use, discourse features, and process in integrated writing tests
    Spaan Fellow Working Papers in Second Oregon Foreign Language Assessment
  13. Ginther (2018)
    Interpreting the relationships between TOEFL iBT scores and GPA: Language proficiency, po…
    Language Testing  ↗
  14. Grant (2000)
    Using computer-tagged linguistic features to describe L2 writing differences
    Journal of Second Language Writing  ↗
  15. Haladyna (2004)
    Construct-irrelevant variance in high-stakes testing
    Educational Measurement: Issues and Practice  ↗
  16. Hu (2012)
    Investigating Chinese university students’ knowledge of and attitudes towards plagiarism …
    Language Learning  ↗
  17. Ihlenfeldt (2022)
    A meta-analysis on the predictive validity of English language proficiency assessments fo…
    Language Testing  ↗
  18. Isaacs (2023)
    Examining the predictive validity of the Duolingo English Test: Evidence from a major UK …
    Language Testing  ↗
  19. Ishikawa (2016)
    The Duolingo English Test and academic English (Duolingo Research Report DRR-16-01)
    Duolingo English Test
  20. Jamieson (2013)
    Reading and Engaging Sources: What Students' use of Sources Reveals About Advan…
    Across the Disciplines
  21. Khabbazbashi (2021)
    Opening the black box: Exploring automated speaking evaluation
    Challenges in Language Testing Around the World: Insights for language test users
  22. Kyle (2020)
    The relationship between features of source text use and integrated writing quality
    Assessing Writing
  23. LaFlair, G.T. (2020). Duolingo English Test: Subscores (DRR-20–03). 〈https://englishtest.duolingo.com/research〉.
     ↗
  24. LaFlair, G.T. & Settles, B. (2020). Duolingo English Test: Technical Manual. Duolingo. 〈https://englishtest.d…
     ↗
  25. Llosa (2019)
    Comparability of students’ writing performance on TOEFL iBT and in required university wr…
    Language Testing  ↗
  26. Linacre (2020)
    Facets computer program for many-facet Rasch measurement (Version 3.83.3)
  27. Messick (1989)
    Validity
    Educational measurement
  28. Messick (1995)
    Validity of psychological assessment: Validation of inferences from persons’ responses an…
    American Psychologist  ↗
  29. Messick (1996)
    Validity and washback in language testing
    Language Testing  ↗
  30. Plakans (2009)
    Discourse synthesis in integrated second language writing assessment
    Language Testing  ↗
  31. Plakans (2013)
    Using multiple texts in an integrated writing assessment: Source text use as a predictor …
    Journal of Second Language Writing  ↗
  32. Purpura (1999)
  33. Qian (2014)
    A register analysis of the GEPT advanced level examinees’ written production (LTTC-GEPT Research Report No. RG-04)
  34. Sawaki (2005)
    The generalizability of summarization and free recall ratings in L2 reading assessment
    JLTA Journal  ↗
  35. Sawaki (2013)
    Understanding learner strengths and weaknesses: Assessing performance on an integrated wr…
    Language Assessment Quarterly  ↗
  36. Settles (2020)
    Machine learning driven language assessment
    Transactions of the Association for Computational Linguistics  ↗
  37. Sireci (2006)
    Innovative item formats in computer-based testing: In pursuit of improved construct repre…
    Handbook of test development
  38. Soper, D.S. (2020). A-priori Sample Size Calculator for Structural Equation Models [Software]. 〈http://www.da…
  39. Thorndike (1949)
    Personnel selection; test and measurement techniques
  40. Uccelli (2019)
    The role of language skills in mid-adolescents’ science summaries
    Journal of Literacy Research  ↗
  41. Weir (2013)
    Examining the criterion-related validity of GEPT advanced reading and writing tests: Comparing GEPT with IELTS and real-life academic performance (LTTC-GEPT Research Report No. RG-01)
  42. Wette (2017)
    Source text use by undergraduate post-novice L2 writers in disciplinary assignments: Prog…
    Journal of Second Language Writing  ↗
  43. Wongtrirat, R. (2010). English language proficiency and academic achievement of international students: A met…
  44. Xie (2013)
    Does test preparation work? Implications for score validity
    Language Assessment Quarterly  ↗
  45. Xie (2015)
    Do component weighting and test method affect time management and approaches to test prep…
    System  ↗
  46. Xie (2015)
    “I must impress the raters!” An investigation of Chinese test-takers’ strategie…
    Assessing Writing
  47. Xie (2020)
    Diagnosing linguistic problems in English academic writing of university students: An ite…
    Language Assessment Quarterly  ↗
  48. Xie (2023)
    Assessing source use: Summary vs. reading-to-write argumentative essay
    Assessing Writing
  49. Xie (2013)
    Do test design and uses influence test preparation: Testing a model of washback with Stru…
    Language Testing  ↗
  50. Yang (2014)
    Toward a model of strategies and summary writing performance
    Language Assessment Quarterly  ↗
  51. Yang (2012)
    Second language writers’ strategy use and performance on an integrated reading‐listening‐…
    TESOL Quarterly  ↗
  52. Yin (2022)
    Evidence-centered design in language testing
    The Routledge Handbook of Language Testing
  53. Yu (2009)
    The shifting sands in the effects of source text summarizability on summary writing
    Assessing Writing
  54. Zhang (2013)
    Effects of instruction on ESL students’ synthesis writing
    Journal of Second Language Writing  ↗
  55. Zhu (2016)
    Exploring the relationships between independent listening and listening-reading-writing t…
    Language Assessment Quarterly  ↗
  56. Zhu (2020)
    Secondary school students’ discourse synthesis performance on Chinese (L1) and English (L…
    Reading and Writing
CrossRef global citation count: 1 View in citation network → Build reading path →

Related articles

  1. Assessing Writing Oct 2026
    Replicating and expanding a validity argument for eRevise as a formative assessment: Empirical support for theories of conceptual change ↗
    Richard Correnti; Lindsay Clare Matsumura; Elaine Lin Wang; Diane Litman; Zhexiong Liu; Haoran Zhang; Tianwen Li
    argument threshold concepts assessment artificial intelligence
  2. Assessing Writing Oct 2026
    Effects of varying time constraints on quality, linguistic features, and fluency behaviors in L2 writing ↗
    Jongbong Lee; Charlene Polio
    argument assessment empirical research multilingual writers affect and writing
  3. Philosophy & Rhetoric Dec 2024
    Sophisms and Contempt for Autonomy ↗
    Katharina Stevens
    argument assessment
  4. Assessing Writing Jul 2024
    Examining the direct and indirect impacts of verbatim source use on linguistic complexity in integrated argumentative writing assessment ↗
    Huiying Cai; Xun Yan
    argument assessment
  5. Written Communication Apr 2024
    A Direct Functional Measure of Text Quality: Did the Reader Understand? ↗
    Joachim Grabowski; Moti Mathiebe
    assessment