How Should We Model the Probability of a Language?
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911435180736512 |
|---|---|
| author | Dent, Rasul Suarez, Pedro Ortiz Clérice, Thibault Sagot, Benoît |
| author_facet | Dent, Rasul Suarez, Pedro Ortiz Clérice, Thibault Sagot, Benoît |
| contents | Of the over 7,000 languages spoken in the world, commercial language identification (LID) systems only reliably identify a few hundred in written form. Research-grade systems extend this coverage under certain circumstances, but for most languages coverage remains patchy or nonexistent. This position paper argues that this situation is largely self-imposed. In particular, it arises from a persistent framing of LID as decontextualized text classification, which obscures the central role of prior probability estimation and is reinforced by institutional incentives that favor global, fixed-prior models. We argue that improving coverage for tail languages requires rethinking LID as a routing problem and developing principled ways to incorporate environmental cues that make languages locally plausible. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_08951 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | How Should We Model the Probability of a Language? Dent, Rasul Suarez, Pedro Ortiz Clérice, Thibault Sagot, Benoît Computation and Language Of the over 7,000 languages spoken in the world, commercial language identification (LID) systems only reliably identify a few hundred in written form. Research-grade systems extend this coverage under certain circumstances, but for most languages coverage remains patchy or nonexistent. This position paper argues that this situation is largely self-imposed. In particular, it arises from a persistent framing of LID as decontextualized text classification, which obscures the central role of prior probability estimation and is reinforced by institutional incentives that favor global, fixed-prior models. We argue that improving coverage for tail languages requires rethinking LID as a routing problem and developing principled ways to incorporate environmental cues that make languages locally plausible. |
| title | How Should We Model the Probability of a Language? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.08951 |