Cosmos: A CXL-Based Full In-Memory System for Approximate Nearest Neighbor Search

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Ko, Seoyoung, Shim, Hyunjeong, Doh, Wanju, Yun, Sungmin, So, Jinin, Kwon, Yongsuk, Park, Sang-Soo, Roh, Si-Dong, Yoon, Minyong, Song, Taeksang, Ahn, Jung Ho
Format:	Preprint
Published:	2025
Subjects:	Hardware Architecture
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866909619601801216
author	Ko, Seoyoung Shim, Hyunjeong Doh, Wanju Yun, Sungmin So, Jinin Kwon, Yongsuk Park, Sang-Soo Roh, Si-Dong Yoon, Minyong Song, Taeksang Ahn, Jung Ho
author_facet	Ko, Seoyoung Shim, Hyunjeong Doh, Wanju Yun, Sungmin So, Jinin Kwon, Yongsuk Park, Sang-Soo Roh, Si-Dong Yoon, Minyong Song, Taeksang Ahn, Jung Ho
contents	Retrieval-Augmented Generation (RAG) is crucial for improving the quality of large language models by injecting proper contexts extracted from external sources. RAG requires high-throughput, low-latency Approximate Nearest Neighbor Search (ANNS) over billion-scale vector databases. Conventional DRAM/SSD solutions face capacity/latency limits, whereas specialized hardware or RDMA clusters lack flexibility or incur network overhead. We present Cosmos, integrating general-purpose cores within CXL memory devices for full ANNS offload and introducing rank-level parallel distance computation to maximize memory bandwidth. We also propose an adjacency-aware data placement that balances search loads across CXL devices based on inter-cluster proximity. Evaluations on SIFT1B and DEEP1B traces show that Cosmos achieves up to 6.72x higher throughput than the baseline CXL system and 2.35x over a state-of-the-art CXL-based solution, demonstrating scalability for RAG pipelines.
format	Preprint
id	arxiv_https___arxiv_org_abs_2505_16096
institution	arXiv
publishDate	2025
record_format	arxiv
spellingShingle	Cosmos: A CXL-Based Full In-Memory System for Approximate Nearest Neighbor Search Ko, Seoyoung Shim, Hyunjeong Doh, Wanju Yun, Sungmin So, Jinin Kwon, Yongsuk Park, Sang-Soo Roh, Si-Dong Yoon, Minyong Song, Taeksang Ahn, Jung Ho Hardware Architecture Retrieval-Augmented Generation (RAG) is crucial for improving the quality of large language models by injecting proper contexts extracted from external sources. RAG requires high-throughput, low-latency Approximate Nearest Neighbor Search (ANNS) over billion-scale vector databases. Conventional DRAM/SSD solutions face capacity/latency limits, whereas specialized hardware or RDMA clusters lack flexibility or incur network overhead. We present Cosmos, integrating general-purpose cores within CXL memory devices for full ANNS offload and introducing rank-level parallel distance computation to maximize memory bandwidth. We also propose an adjacency-aware data placement that balances search loads across CXL devices based on inter-cluster proximity. Evaluations on SIFT1B and DEEP1B traces show that Cosmos achieves up to 6.72x higher throughput than the baseline CXL system and 2.35x over a state-of-the-art CXL-based solution, demonstrating scalability for RAG pipelines.
title	Cosmos: A CXL-Based Full In-Memory System for Approximate Nearest Neighbor Search
topic	Hardware Architecture
url	https://arxiv.org/abs/2505.16096