Assessing intelligibility as conceptualised within the CEFR-Companion Volume (CV) Framework using Adaptive Comparative Judgment

Abstract

Two pivotal constructs, intelligibility (listeners’ actual understanding) and comprehensibility (listeners’ perceived ease of understanding), have dominated second language (L2) pronunciation research, marking a shift away from an emphasis on nativeness. The 2020 Companion Volume to the Common European Framework of Reference for Languages (CEFR-CV) presented a revised phonological scale, integrating both dimensions into a new definition of intelligibility. However, effective measurements to assess this refined construct are still lacking. This study explores the potential of Adaptive Comparative Judgment (ACJ) in measuring intelligibility as conceptualised by the CEFR-CV. ACJ employs judges who evaluate two stimuli based on a holistic criterion, selecting the better one. Through a collection of such binary decisions, judges’ evaluations are statistically analysed, producing standardised estimates for each stimulus. Twelve judges assessed speech samples from 30 Mandarin speakers of English as an L2 learners performing four sentence repetition tasks. Incorporating Think Aloud Protocol (TAP) into the judgment process, the study combined quantitative and qualitative analyses, providing evidence for the efficacy of ACJ in measuring intelligibility. The findings, discussed in the context of existing research on intelligibility and comprehensibility, unveil future research on the use of ACJ in L2 pronunciation assessment in further elucidating the intelligibility construct as defined by the CEFR-CV.

More information

Identifiers

ORCID for Jingwen Wang: orcid.org/0009-0000-7615-2674

ORCID for Ying Zheng: orcid.org/0000-0003-2574-0358

Catalogue record

Export record

Altmetrics