Fabien Lopez

Neural machine translation systems have reached near-expert performance at the sentence level, producing fluent and locally accurate translations. However, this performance degrades when moving to document-level translation, where discourse-level phenomena come into account, such as referential coherence, terminological consistency, and context-dependent disambiguation. In particular, specific linguistic ambiguities (such as those related to coreference) involve only a very small number of tokens (e.g.: pronouns), yet have a decisive impact on the overall coherence of the translated text. A major limitation of current evaluation practices comes from the fact that standard metrics are largely insensitive to these localised errors or do not require the model to actually generate the translation. As a result, a model may achieve high scores while still failing in practice to correctly translate context-dependent ambiguities. This highlights a gap between measured performance and actual discourse quality and underscores the need for more fine-grained evaluation methods that aim
to better understand the model’s translation process. This thesis addresses this issue by investigating how context-aware Transformer models leverage surrounding sentences through their attention mechanisms. Rather than focusing solely on output evaluation, this work adopts an interpretability-orientated perspective, aiming to determine whether models effectively rely on relevant contextual elements, such as the correct antecedents, when producing translations.

To this end, the thesis proposes a methodological framework combining attention weight analysis and controlled input context. It introduces a set of complementary metrics designed to quantify the extent to which attention distributions align with coreference relations across sentences. These metrics capture different types of attention behaviour, ranging from sharp alignment peaks to more subtle contextual activations. Ultimately, this work contributes to bridging the gap between translation performance and model interpretability by providing tools to assess not only whether a model produces a correct translation but also to questions whether it does so for the right reasons: by effectively leveraging contextual information.