Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.23127 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909877802106880 |
|---|---|
| author | Zhuang, Kai Zhang, Jiawei Liu, Yumou Cao, Hanqun Gu, Chunbin Liu, Mengdi Gao, Zhangyang Wang, Zitong Jerry Zhou, Xuanhe Heng, Pheng-Ann Wu, Lijun He, Conghui Tan, Cheng |
| author_facet | Zhuang, Kai Zhang, Jiawei Liu, Yumou Cao, Hanqun Gu, Chunbin Liu, Mengdi Gao, Zhangyang Wang, Zitong Jerry Zhou, Xuanhe Heng, Pheng-Ann Wu, Lijun He, Conghui Tan, Cheng |
| contents | Scientific Large Language Models (Sci-LLMs) have emerged as a promising frontier for accelerating biological discovery. However, these models face a fundamental challenge when processing raw biomolecular sequences: the tokenization dilemma. Whether treating sequences as a specialized language, risking the loss of functional motif information, or as a separate modality, introducing formidable alignment challenges, current strategies fundamentally limit their reasoning capacity. We challenge this sequence-centric paradigm by positing that a more effective strategy is to provide Sci-LLMs with high-level structured context derived from established bioinformatics tools, thereby bypassing the need to interpret low-level noisy sequence data directly. Through a systematic comparison of leading Sci-LLMs on biological reasoning tasks, we tested three input modes: sequence-only, context-only, and a combination of both. Our findings are striking: the context-only approach consistently and substantially outperforms all other modes. Even more revealing, the inclusion of the raw sequence alongside its high-level context consistently degrades performance, indicating that raw sequences act as informational noise, even for models with specialized tokenization schemes. These results suggest that the primary strength of existing Sci-LLMs lies not in their nascent ability to interpret biomolecular syntax from scratch, but in their profound capacity for reasoning over structured, human-readable knowledge. Therefore, we argue for reframing Sci-LLMs not as sequence decoders, but as powerful reasoning engines over expert knowledge. This work lays the foundation for a new class of hybrid scientific AI agents, repositioning the developmental focus from direct sequence interpretation towards high-level knowledge synthesis. The code is available at https://github.com/opendatalab-raiser/CoKE. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_23127 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Lost in Tokenization: Context as the Key to Unlocking Biomolecular Understanding in Scientific LLMs Zhuang, Kai Zhang, Jiawei Liu, Yumou Cao, Hanqun Gu, Chunbin Liu, Mengdi Gao, Zhangyang Wang, Zitong Jerry Zhou, Xuanhe Heng, Pheng-Ann Wu, Lijun He, Conghui Tan, Cheng Artificial Intelligence Scientific Large Language Models (Sci-LLMs) have emerged as a promising frontier for accelerating biological discovery. However, these models face a fundamental challenge when processing raw biomolecular sequences: the tokenization dilemma. Whether treating sequences as a specialized language, risking the loss of functional motif information, or as a separate modality, introducing formidable alignment challenges, current strategies fundamentally limit their reasoning capacity. We challenge this sequence-centric paradigm by positing that a more effective strategy is to provide Sci-LLMs with high-level structured context derived from established bioinformatics tools, thereby bypassing the need to interpret low-level noisy sequence data directly. Through a systematic comparison of leading Sci-LLMs on biological reasoning tasks, we tested three input modes: sequence-only, context-only, and a combination of both. Our findings are striking: the context-only approach consistently and substantially outperforms all other modes. Even more revealing, the inclusion of the raw sequence alongside its high-level context consistently degrades performance, indicating that raw sequences act as informational noise, even for models with specialized tokenization schemes. These results suggest that the primary strength of existing Sci-LLMs lies not in their nascent ability to interpret biomolecular syntax from scratch, but in their profound capacity for reasoning over structured, human-readable knowledge. Therefore, we argue for reframing Sci-LLMs not as sequence decoders, but as powerful reasoning engines over expert knowledge. This work lays the foundation for a new class of hybrid scientific AI agents, repositioning the developmental focus from direct sequence interpretation towards high-level knowledge synthesis. The code is available at https://github.com/opendatalab-raiser/CoKE. |
| title | Lost in Tokenization: Context as the Key to Unlocking Biomolecular Understanding in Scientific LLMs |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2510.23127 |