Cross-lingual Word Sense Disambiguation with Tiny Aya

Wed, 15 Jul 2026 00:00:00 +0000

Cohere published a about the projects that came out of Expedition Tiny Aya, and the Word Sense Disambiguation project I have been leading since February is one of them!

Expedition Tiny Aya is a research program that Cohere Labs started shortly after releasing Tiny Aya, its open-weight multilingual model, to get the model into the hands of researchers, students and builders, with mentoring from Cohere. Our team was Alisa Banerjee, Hamza Shezad, Sirichada Wattanasiritanawong, Ahanaf Aziz and me, and our mentor was Ammar Khairi.

We wanted to find out how well a small multilingual model can learn to tell the senses of a word apart in many languages at once. Word Sense Disambiguation is the task of deciding which meaning a word has in a given sentence (I wrote about why it matters for Latin and Ancient Greek ). Most of the research on it has been done in English, while the training and evaluation data for other languages is much smaller.

To train and test the models, we built a new version of XL-WSD (Pasini et al., 2021), a sense-annotated benchmark covering 18 languages, from Basque and Bulgarian to Japanese and Korean, and reformatted it as instruction data for language models. Every instance comes in two forms. In the multiple-choice version, the model reads the sentence and a numbered list of candidate definitions and answers with the number of the correct one; in the generation version, it gets only the sentence and the target word, and has to write the definition itself. We kept only definitions in the same language as the sentence and dropped words with a single sense, so that every example actually requires disambiguation. The dataset is and follows the XL-WSD non-commercial license.

We then trained and evaluated Tiny Aya models across these languages. In the words of Cohere’s summary, we observed strong cross-lingual transfer, “with some configurations achieving performance competitive with substantially larger models”.

Screenshot from on the Cohere blog

Many thanks to my teammates, to Ammar for the mentoring, and to Cohere Labs for running the program.

Cross-Lingual Word Sense Disambiguation with Tiny Aya

Wed, 15 Jul 2026 00:00:00 +0000

A project of Expedition Tiny Aya, the research program run by Cohere Labs around its open-weight multilingual model Tiny Aya. I led a team (Alisa Banerjee, Hamza Shezad, Sirichada Wattanasiritanawong, Ahanaf Aziz and me, mentored by Ammar Khairi) that trained and evaluated Tiny Aya models on Word Sense Disambiguation in 18 languages.

Word Sense Disambiguation asks a model to decide which meaning a word has in context. For most languages other than English, sense-annotated data is scarce, which makes cross-lingual transfer an important question: can a small multilingual model learn the task in some languages and apply it in others?

To find out, we adapted XL-WSD (Pasini et al., 2021), a sense-annotated benchmark in 18 languages, into instruction data for language models. Each instance has two views that share the same gold sense: a multiple-choice question, where the model picks the correct definition from a numbered list, and a generation task, where it writes the definition without seeing any candidates. Candidate definitions are always in the language of the sentence, and words with a single sense are removed, so every example requires disambiguation. The resulting dataset, with train, dev and test splits, is released on under the XL-WSD non-commercial license.

The project was featured in Cohere’s blog post , which reports that the team observed strong cross-lingual transfer, with some configurations competitive with substantially larger models. More details are in .

Multilingual NLP | Michele Ciletti - Personal Website

Cross-lingual Word Sense Disambiguation with Tiny Aya

Cross-Lingual Word Sense Disambiguation with Tiny Aya