XferBench: a Data-Driven Benchmark for Emergent Language
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913416336113664 |
|---|---|
| author | Boldt, Brendon Mortensen, David |
| author_facet | Boldt, Brendon Mortensen, David |
| contents | In this paper, we introduce a benchmark for evaluating the overall quality of emergent languages using data-driven methods. Specifically, we interpret the notion of the "quality" of an emergent language as its similarity to human language within a deep learning framework. We measure this by using the emergent language as pretraining data for a downstream NLP tasks in human language -- the better the downstream performance, the better the emergent language. We implement this benchmark as an easy-to-use Python package that only requires a text file of utterances from the emergent language to be evaluated. Finally, we empirically test the benchmark's validity using human, synthetic, and emergent language baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_03456 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | XferBench: a Data-Driven Benchmark for Emergent Language Boldt, Brendon Mortensen, David Computation and Language I.2.7; I.6.m In this paper, we introduce a benchmark for evaluating the overall quality of emergent languages using data-driven methods. Specifically, we interpret the notion of the "quality" of an emergent language as its similarity to human language within a deep learning framework. We measure this by using the emergent language as pretraining data for a downstream NLP tasks in human language -- the better the downstream performance, the better the emergent language. We implement this benchmark as an easy-to-use Python package that only requires a text file of utterances from the emergent language to be evaluated. Finally, we empirically test the benchmark's validity using human, synthetic, and emergent language baselines. |
| title | XferBench: a Data-Driven Benchmark for Emergent Language |
| topic | Computation and Language I.2.7; I.6.m |
| url | https://arxiv.org/abs/2407.03456 |