Sylvain Coulange

Abstract:

Oral production proficiency in a foreign language is commonly assessed through the speaker's ability to be understood. The term comprehensibility is frequently used in the field of second language acquisition to refer to the level of effort required by a listener to understand the speaker. Comprehensibility depends on numerous factors, both speaker- and listener-related, among which the position of hesitation markers and speech rhythm are often highlighted as key factors in evaluation scales. In contrast, automatic assessment approaches still predominantly rely on comparisons with a model, often based on read speech, without specific attention to linguistic phenomena likely to disrupt comprehension.

In this thesis, we propose a tool for the automatic evaluation of spontaneous oral production in English, specifically targeting two linguistic phenomena likely to impact speaker comprehensibility: the syntactic distribution of pauses and lexical stress. This tool was used to analyse variation of these phenomena among French- and Japanese-speaking learners of English at B1 and B2 proficiency levels (L2), as well as among native English speakers (L1). We then measured their impact on the perception of real-time comprehension effort in native English-speaking listeners.

The analyses reveal that a major difference between B1 and B2 speakers, as well as between L1 and L2 speakers, lies in the frequency of low-syntactic-level pauses: as proficiency increases, pauses tend to concentrate more at high-syntactic-boundary locations (e.g., between clauses). Based on these observations, a syntactic pause distribution score was developed to quantify this concentration.

Regarding lexical stress, the findings show a strong influence of speakers' native language stress patterns. French speakers typically increase pitch (f0) and lengthen the final syllable of words, while intensity remains stable across syllables. Although B1 and B2 speakers largely overlap, significant improvement is observed between these levels: B2 speakers more frequently stress the expected syllable and produce stronger acoustic contrasts in terms of f0 and intensity. On the other hand, Japanese speakers, whose native language incorporates lexical stress, position stress more accurately and produce greater prosodic contrasts than French speakers.

Finally, to measure the impact of different types of pauses and lexical stress patterns on perceived comprehensibility, we recruited 60 listeners to dynamically evaluate 16 speech excerpts from the aforementioned corpus of French speakers. While listening, participants were asked to click a button whenever they experienced difficulty understanding the speaker, for whatever reason. An analysis of the normalized click patterns indicates a significant increase in perceived difficulty between 0 and 3 seconds following a pause within a phrase, and between 2 and 3 seconds after a word with an unexpected stressed syllable. Conversely, perceived difficulty significantly decreased after a pause between clauses or following a correctly stressed word. These findings support the idea that low-syntactic-level pauses and unexpected lexical stress directly impact the perception of comprehensibility. Our experiment shows that this impact is direct and measurable.

UGA, Website

Event on Automated L2 Speaking Assessment