Luxical: High-Speed Lexical-Dense Text Embeddings
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915669109374976 |
|---|---|
| author | DatologyAI : Merrick, Luke Fang, Alex Carranza, Aldo Deng, Alvin Abbas, Amro Larsen, Brett Blakeney, Cody Teh, Darren Schwab, David Pan, Fan Mongstad, Haakon Yin, Haoli Urbanek, Jack Lee, Jason Telanoff, Jason Wills, Josh Mentzer, Kaleigh Burstein, Paul Doshi, Parth Burnstein, Paul Maini, Pratyush Monti, Ricardo Adiga, Rishabh Loftin, Scott Joshi, Siddharth Das, Spandan Jiang, Tony Dorna, Vineeth Wang, Zhengping Gaza, Bogdan Morcos, Ari Leavitt, Matthew |
| author_facet | DatologyAI : Merrick, Luke Fang, Alex Carranza, Aldo Deng, Alvin Abbas, Amro Larsen, Brett Blakeney, Cody Teh, Darren Schwab, David Pan, Fan Mongstad, Haakon Yin, Haoli Urbanek, Jack Lee, Jason Telanoff, Jason Wills, Josh Mentzer, Kaleigh Burstein, Paul Doshi, Parth Burnstein, Paul Maini, Pratyush Monti, Ricardo Adiga, Rishabh Loftin, Scott Joshi, Siddharth Das, Spandan Jiang, Tony Dorna, Vineeth Wang, Zhengping Gaza, Bogdan Morcos, Ari Leavitt, Matthew |
| contents | Frontier language model quality increasingly hinges on our ability to organize web-scale text corpora for training. Today's dominant tools trade off speed and flexibility: lexical classifiers (e.g., FastText) are fast but limited to producing classification output scores, while the vector-valued outputs of transformer text embedding models flexibly support numerous workflows (e.g., clustering, classification, and retrieval) but are computationally expensive to produce. We introduce Luxical, a library for high-speed "lexical-dense" text embeddings that aims to recover the best properties of both approaches for web-scale text organization. Luxical combines sparse TF--IDF features, a small ReLU network, and a knowledge distillation training regimen to approximate large transformer embedding models at a fraction of their operational cost. In this technical report, we describe the Luxical architecture and training objective and evaluate a concrete Luxical model in two disparate applications: a targeted webcrawl document retrieval test and an end-to-end language model data curation task grounded in text classification. In these tasks we demonstrate speedups ranging from 3x to 100x over varying-sized neural baselines, and comparable to FastText model inference during the data curation task. On these evaluations, the tested Luxical model illustrates favorable compute/quality trade-offs for large-scale text organization, matching the quality of neural baselines. Luxical is available as open-source software at https://github.com/datologyai/luxical. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_09015 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Luxical: High-Speed Lexical-Dense Text Embeddings DatologyAI : Merrick, Luke Fang, Alex Carranza, Aldo Deng, Alvin Abbas, Amro Larsen, Brett Blakeney, Cody Teh, Darren Schwab, David Pan, Fan Mongstad, Haakon Yin, Haoli Urbanek, Jack Lee, Jason Telanoff, Jason Wills, Josh Mentzer, Kaleigh Burstein, Paul Doshi, Parth Burnstein, Paul Maini, Pratyush Monti, Ricardo Adiga, Rishabh Loftin, Scott Joshi, Siddharth Das, Spandan Jiang, Tony Dorna, Vineeth Wang, Zhengping Gaza, Bogdan Morcos, Ari Leavitt, Matthew Computation and Language Machine Learning Frontier language model quality increasingly hinges on our ability to organize web-scale text corpora for training. Today's dominant tools trade off speed and flexibility: lexical classifiers (e.g., FastText) are fast but limited to producing classification output scores, while the vector-valued outputs of transformer text embedding models flexibly support numerous workflows (e.g., clustering, classification, and retrieval) but are computationally expensive to produce. We introduce Luxical, a library for high-speed "lexical-dense" text embeddings that aims to recover the best properties of both approaches for web-scale text organization. Luxical combines sparse TF--IDF features, a small ReLU network, and a knowledge distillation training regimen to approximate large transformer embedding models at a fraction of their operational cost. In this technical report, we describe the Luxical architecture and training objective and evaluate a concrete Luxical model in two disparate applications: a targeted webcrawl document retrieval test and an end-to-end language model data curation task grounded in text classification. In these tasks we demonstrate speedups ranging from 3x to 100x over varying-sized neural baselines, and comparable to FastText model inference during the data curation task. On these evaluations, the tested Luxical model illustrates favorable compute/quality trade-offs for large-scale text organization, matching the quality of neural baselines. Luxical is available as open-source software at https://github.com/datologyai/luxical. |
| title | Luxical: High-Speed Lexical-Dense Text Embeddings |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2512.09015 |