From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912741579554816 |
|---|---|
| author | Zhang, Charles Peng, Benji Sun, Xintian Niu, Qian Liu, Junyu Chen, Keyu Li, Ming Feng, Pohsun Bi, Ziqian Liu, Ming Zhang, Yichao Song, Xinyuan Fei, Cheng Yin, Caitlyn Heqi Yan, Lawrence KQ He, Hongyang Wang, Tianyang |
| author_facet | Zhang, Charles Peng, Benji Sun, Xintian Niu, Qian Liu, Junyu Chen, Keyu Li, Ming Feng, Pohsun Bi, Ziqian Liu, Ming Zhang, Yichao Song, Xinyuan Fei, Cheng Yin, Caitlyn Heqi Yan, Lawrence KQ He, Hongyang Wang, Tianyang |
| contents | Word embeddings and language models have transformed natural language processing (NLP) by facilitating the representation of linguistic elements in continuous vector spaces. This review visits foundational concepts such as the distributional hypothesis and contextual similarity, tracing the evolution from sparse representations like one-hot encoding to dense embeddings including Word2Vec, GloVe, and fastText. We examine both static and contextualized embeddings, underscoring advancements in models such as ELMo, BERT, and GPT and their adaptations for cross-lingual and personalized applications. The discussion extends to sentence and document embeddings, covering aggregation methods and generative topic models, along with the application of embeddings in multimodal domains, including vision, robotics, and cognitive science. Advanced topics such as model compression, interpretability, numerical encoding, and bias mitigation are analyzed, addressing both technical challenges and ethical implications. Additionally, we identify future research directions, emphasizing the need for scalable training techniques, enhanced interpretability, and robust grounding in non-textual modalities. By synthesizing current methodologies and emerging trends, this survey offers researchers and practitioners an in-depth resource to push the boundaries of embedding-based language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_05036 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models Zhang, Charles Peng, Benji Sun, Xintian Niu, Qian Liu, Junyu Chen, Keyu Li, Ming Feng, Pohsun Bi, Ziqian Liu, Ming Zhang, Yichao Song, Xinyuan Fei, Cheng Yin, Caitlyn Heqi Yan, Lawrence KQ He, Hongyang Wang, Tianyang Computation and Language Word embeddings and language models have transformed natural language processing (NLP) by facilitating the representation of linguistic elements in continuous vector spaces. This review visits foundational concepts such as the distributional hypothesis and contextual similarity, tracing the evolution from sparse representations like one-hot encoding to dense embeddings including Word2Vec, GloVe, and fastText. We examine both static and contextualized embeddings, underscoring advancements in models such as ELMo, BERT, and GPT and their adaptations for cross-lingual and personalized applications. The discussion extends to sentence and document embeddings, covering aggregation methods and generative topic models, along with the application of embeddings in multimodal domains, including vision, robotics, and cognitive science. Advanced topics such as model compression, interpretability, numerical encoding, and bias mitigation are analyzed, addressing both technical challenges and ethical implications. Additionally, we identify future research directions, emphasizing the need for scalable training techniques, enhanced interpretability, and robust grounding in non-textual modalities. By synthesizing current methodologies and emerging trends, this survey offers researchers and practitioners an in-depth resource to push the boundaries of embedding-based language models. |
| title | From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.05036 |