Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.10648 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916891973386240 |
|---|---|
| author | Zhang, Erica Goto, Ryunosuke Sagan, Naomi Mutter, Jurik Phillips, Nick Alizadeh, Ash Lee, Kangwook Blanchet, Jose Pilanci, Mert Tibshirani, Robert |
| author_facet | Zhang, Erica Goto, Ryunosuke Sagan, Naomi Mutter, Jurik Phillips, Nick Alizadeh, Ash Lee, Kangwook Blanchet, Jose Pilanci, Mert Tibshirani, Robert |
| contents | We introduce LLM-Lasso, a novel framework that leverages large language models (LLMs) to guide feature selection in Lasso $\ell_1$ regression. Unlike traditional methods that rely solely on numerical data, LLM-Lasso incorporates domain-specific knowledge extracted from natural language, enhanced through a retrieval-augmented generation (RAG) pipeline, to seamlessly integrate data-driven modeling with contextual insights. Specifically, the LLM generates penalty factors for each feature, which are converted into weights for the Lasso penalty using a simple, tunable model. Features identified as more relevant by the LLM receive lower penalties, increasing their likelihood of being retained in the final model, while less relevant features are assigned higher penalties, reducing their influence. Importantly, LLM-Lasso has an internal validation step that determines how much to trust the contextual knowledge in our prediction pipeline. Hence it addresses key challenges in robustness, making it suitable for mitigating potential inaccuracies or hallucinations from the LLM. In various biomedical case studies, LLM-Lasso outperforms standard Lasso and existing feature selection baselines, all while ensuring the LLM operates without prior access to the datasets. To our knowledge, this is the first approach to effectively integrate conventional feature selection techniques directly with LLM-based domain-specific reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_10648 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularization Zhang, Erica Goto, Ryunosuke Sagan, Naomi Mutter, Jurik Phillips, Nick Alizadeh, Ash Lee, Kangwook Blanchet, Jose Pilanci, Mert Tibshirani, Robert Machine Learning We introduce LLM-Lasso, a novel framework that leverages large language models (LLMs) to guide feature selection in Lasso $\ell_1$ regression. Unlike traditional methods that rely solely on numerical data, LLM-Lasso incorporates domain-specific knowledge extracted from natural language, enhanced through a retrieval-augmented generation (RAG) pipeline, to seamlessly integrate data-driven modeling with contextual insights. Specifically, the LLM generates penalty factors for each feature, which are converted into weights for the Lasso penalty using a simple, tunable model. Features identified as more relevant by the LLM receive lower penalties, increasing their likelihood of being retained in the final model, while less relevant features are assigned higher penalties, reducing their influence. Importantly, LLM-Lasso has an internal validation step that determines how much to trust the contextual knowledge in our prediction pipeline. Hence it addresses key challenges in robustness, making it suitable for mitigating potential inaccuracies or hallucinations from the LLM. In various biomedical case studies, LLM-Lasso outperforms standard Lasso and existing feature selection baselines, all while ensuring the LLM operates without prior access to the datasets. To our knowledge, this is the first approach to effectively integrate conventional feature selection techniques directly with LLM-based domain-specific reasoning. |
| title | LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularization |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2502.10648 |