Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.14520 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914068975058944 |
|---|---|
| author | Chen, Xinyi Yuan, Yifei Li, Jiaang Belongie, Serge de Rijke, Maarten Søgaard, Anders |
| author_facet | Chen, Xinyi Yuan, Yifei Li, Jiaang Belongie, Serge de Rijke, Maarten Søgaard, Anders |
| contents | Language models are often said to face a symbol grounding problem. While some have argued the problem can be solved without resort to other modalities, many have speculated that grounded learning is more efficient. We explore this question in Othello, a simplified, rule-based world that offers a controlled and interpretable testbed for studying world understanding. Building on prior work, we introduce VISOTHELLO, a multi-modal model trained jointly on move sequences and board images. Using the Othello rule understanding task, we examine whether multi-modal learning provides advantages over text-only approaches. We further evaluate robustness under semantically irrelevant perturbations and analyze the consistency of cross-modal alignment. Our results suggest that multi-modal training not only improves performance and robustness but also promotes convergence toward shared internal representations across different model architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_14520 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | What if Othello-Playing Language Models Could See? Chen, Xinyi Yuan, Yifei Li, Jiaang Belongie, Serge de Rijke, Maarten Søgaard, Anders Artificial Intelligence Language models are often said to face a symbol grounding problem. While some have argued the problem can be solved without resort to other modalities, many have speculated that grounded learning is more efficient. We explore this question in Othello, a simplified, rule-based world that offers a controlled and interpretable testbed for studying world understanding. Building on prior work, we introduce VISOTHELLO, a multi-modal model trained jointly on move sequences and board images. Using the Othello rule understanding task, we examine whether multi-modal learning provides advantages over text-only approaches. We further evaluate robustness under semantically irrelevant perturbations and analyze the consistency of cross-modal alignment. Our results suggest that multi-modal training not only improves performance and robustness but also promotes convergence toward shared internal representations across different model architectures. |
| title | What if Othello-Playing Language Models Could See? |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2507.14520 |