Operationalising intelligibility within the CEFR-CV framework: A mixed methods approach combining Adaptive Comparative Judgment and Think Aloud Protocol

Abstract

The constructs of intelligibility (listeners’ actual understanding) and comprehensibility (listeners’ perceived ease of understanding) are central in L2 pronunciation research, signalling a shift from emphasising nativeness to recognising English as a global Lingua Franca. Despite advances, precise definitions of these constructs remain elusive. The 2020 Companion Volume to the CEFR (CEFR-CV) redefined intelligibility by merging it with comprehensibility, but effective tools for assessing it are still lacking, which complicates its understanding. This study evaluates the efficacy of Adaptive Comparative Judgment (ACJ) in measuring CEFR-CV intelligibility. ACJ involves judges who compare speech samples in pairs and select those that best align with a holistic criterion. This process is enriched by integrating Think Aloud Protocols (TAPs), which reveal the linguistic features influencing assessments. These insights have guided the development of comprehensive CEFR-CV intelligibility constructs. To establish these constructs across diverse contexts, judges evaluated recordings from both controlled and uncontrolled tasks. Twelve Mandarin-speaking judges assessed 30 Mandarin L1 English L2 learners in the controlled task, while 12 mixed-L1 judges evaluated 30 mixed-L1 learners in the uncontrolled task. The integration of quantitative ACJ data with qualitative TAP analyses provided strong evidence of the effectiveness of ACJ. Analysis of the TAPs resulted in 1,526 and 917 codings from each task respectively, establishing the foundation for the CEFR-CV intelligibility constructs. The findings reveal that CEFR-CV intelligibility not only encompasses traditional pronunciation features but also broadens to include wider speaking assessment criteria. Constructs for controlled tasks include segmentals, suprasegmentals, fluency, grammar, discourse, and acceptability, while those for uncontrolled tasks additionally consider accentedness and lexis. These constructs provide a solid framework for L2 pronunciation assessment. This study not only opens new avenues for employing ACJ in L2 pronunciation assessment but also elucidates the intelligibility construct as defined by the CEFR-CV, offering significant implications for language teaching and testing practice in ELF contexts.

More information

Identifiers

ORCID for Jingwen Wang: orcid.org/0009-0000-7615-2674

ORCID for Ying Zheng: orcid.org/0000-0003-2574-0358

ORCID for Laura Dominguez: orcid.org/0000-0003-2701-2469

Catalogue record

Export record