The role of positional encodings in the ARC benchmark
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916594730401792 |
|---|---|
| author | Costa, Guilherme H. Bandeira Freire, Miguel Oliveira, Arlindo L. |
| author_facet | Costa, Guilherme H. Bandeira Freire, Miguel Oliveira, Arlindo L. |
| contents | The Abstraction and Reasoning Corpus challenges AI systems to perform abstract reasoning with minimal training data, a task intuitive for humans but demanding for machine learning models. Using CodeT5+ as a case study, we demonstrate how limitations in positional encoding hinder reasoning and impact performance. This work further examines the role of positional encoding across transformer architectures, highlighting its critical influence on models of varying sizes and configurations. Comparing several strategies, we find that while 2D positional encoding and Rotary Position Embedding offer competitive performance, 2D encoding excels in data-constrained scenarios, emphasizing its effectiveness for ARC tasks |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_00174 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The role of positional encodings in the ARC benchmark Costa, Guilherme H. Bandeira Freire, Miguel Oliveira, Arlindo L. Artificial Intelligence Machine Learning The Abstraction and Reasoning Corpus challenges AI systems to perform abstract reasoning with minimal training data, a task intuitive for humans but demanding for machine learning models. Using CodeT5+ as a case study, we demonstrate how limitations in positional encoding hinder reasoning and impact performance. This work further examines the role of positional encoding across transformer architectures, highlighting its critical influence on models of varying sizes and configurations. Comparing several strategies, we find that while 2D positional encoding and Rotary Position Embedding offer competitive performance, 2D encoding excels in data-constrained scenarios, emphasizing its effectiveness for ARC tasks |
| title | The role of positional encodings in the ARC benchmark |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2502.00174 |