Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable Evaluation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915888966402048 |
|---|---|
| author | Habibi, Reza Lee, Darian El-Nasr, Magy Seif |
| author_facet | Habibi, Reza Lee, Darian El-Nasr, Magy Seif |
| contents | Accuracy-based evaluation cannot reliably distinguish genuine generalization from shortcuts like memorization, leakage, or brittle heuristics, especially in small-data regimes. In this position paper, we argue for mechanism-aware evaluation that combines task-relevant symbolic rules with mechanistic interpretability, yielding algorithmic pass/fail scores that show exactly where models generalize versus exploit patterns. We demonstrate this on NL-to-SQL by training two identical architectures under different conditions: one without schema information (forcing memorization), one with schema (enabling grounding). Standard evaluation shows the memorization model achieves 94% field-name accuracy on unseen data, falsely suggesting competence. Our symbolic-mechanistic evaluation reveals this model violates core schema generalization rules, a failure invisible to accuracy metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_23517 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable Evaluation Habibi, Reza Lee, Darian El-Nasr, Magy Seif Machine Learning Artificial Intelligence Computation and Language Symbolic Computation Accuracy-based evaluation cannot reliably distinguish genuine generalization from shortcuts like memorization, leakage, or brittle heuristics, especially in small-data regimes. In this position paper, we argue for mechanism-aware evaluation that combines task-relevant symbolic rules with mechanistic interpretability, yielding algorithmic pass/fail scores that show exactly where models generalize versus exploit patterns. We demonstrate this on NL-to-SQL by training two identical architectures under different conditions: one without schema information (forcing memorization), one with schema (enabling grounding). Standard evaluation shows the memorization model achieves 94% field-name accuracy on unseen data, falsely suggesting competence. Our symbolic-mechanistic evaluation reveals this model violates core schema generalization rules, a failure invisible to accuracy metrics. |
| title | Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable Evaluation |
| topic | Machine Learning Artificial Intelligence Computation and Language Symbolic Computation |
| url | https://arxiv.org/abs/2603.23517 |