Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2603.04408 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912943029878784 |
|---|---|
| author | Peng, Luzhou Yang, Zhengxin Ji, Honglu Yang, Yikang Fan, Fanda Gao, Wanling Ge, Jiayuan Han, Yilin Zhan, Jianfeng |
| author_facet | Peng, Luzhou Yang, Zhengxin Ji, Honglu Yang, Yikang Fan, Fanda Gao, Wanling Ge, Jiayuan Han, Yilin Zhan, Jianfeng |
| contents | Current evaluation paradigms for large language models (LLMs) characterize models and datasets separately, yielding coarse descriptions: items in datasets are treated as pre-labeled entries, and models are summarized by overall scores such as accuracy, together ignoring the diversity of population-level model behaviors across items with varying properties. To address this gap, this paper conceptualizes LLMs as composed of memes, a notion introduced by Dawkins as cultural genes that replicate knowledge and behavior. Building on this perspective, the Probing Memes paradigm reconceptualizes evaluation as an entangled world of models and data. It centers on a Perception Matrix that captures model-item interactions, enabling Probe Properties for characterizing items and Meme Scores for depicting model behavioral traits. Applied to 9 datasets and 4,507 LLMs, Probing Memes reveals hidden capability structures and quantifies phenomena invisible under traditional paradigms (e.g., elite models failing on problems that most models answer easily). It not only supports more informative and extensible benchmarks but also enables population-based evaluation of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_04408 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Probing Memes in LLMs: A Paradigm for the Entangled Evaluation World Peng, Luzhou Yang, Zhengxin Ji, Honglu Yang, Yikang Fan, Fanda Gao, Wanling Ge, Jiayuan Han, Yilin Zhan, Jianfeng Computation and Language Current evaluation paradigms for large language models (LLMs) characterize models and datasets separately, yielding coarse descriptions: items in datasets are treated as pre-labeled entries, and models are summarized by overall scores such as accuracy, together ignoring the diversity of population-level model behaviors across items with varying properties. To address this gap, this paper conceptualizes LLMs as composed of memes, a notion introduced by Dawkins as cultural genes that replicate knowledge and behavior. Building on this perspective, the Probing Memes paradigm reconceptualizes evaluation as an entangled world of models and data. It centers on a Perception Matrix that captures model-item interactions, enabling Probe Properties for characterizing items and Meme Scores for depicting model behavioral traits. Applied to 9 datasets and 4,507 LLMs, Probing Memes reveals hidden capability structures and quantifies phenomena invisible under traditional paradigms (e.g., elite models failing on problems that most models answer easily). It not only supports more informative and extensible benchmarks but also enables population-based evaluation of LLMs. |
| title | Probing Memes in LLMs: A Paradigm for the Entangled Evaluation World |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2603.04408 |