Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.07557 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914082299314176 |
|---|---|
| author | Bhandarkar, Abhay Mishra, Gaurav Juchani, Khushi Singhal, Harsh |
| author_facet | Bhandarkar, Abhay Mishra, Gaurav Juchani, Khushi Singhal, Harsh |
| contents | This study applies BERTopic, a transformer-based topic modeling technique, to the lmsys-chat-1m dataset, a multilingual conversational corpus built from head-to-head evaluations of large language models (LLMs). Each user prompt is paired with two anonymized LLM responses and a human preference label, used to assess user evaluation of competing model outputs. The main objective is uncovering thematic patterns in these conversations and examining their relation to user preferences, particularly if certain LLMs are consistently preferred within specific topics. A robust preprocessing pipeline was designed for multilingual variation, balancing dialogue turns, and cleaning noisy or redacted data. BERTopic extracted over 29 coherent topics including artificial intelligence, programming, ethics, and cloud infrastructure. We analysed relationships between topics and model preferences to identify trends in model-topic alignment. Visualization techniques included inter-topic distance maps, topic probability distributions, and model-versus-topic matrices. Our findings inform domain-specific fine-tuning and optimization strategies for improving real-world LLM performance and user satisfaction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_07557 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Investigating Thematic Patterns and User Preferences in LLM Interactions using BERTopic Bhandarkar, Abhay Mishra, Gaurav Juchani, Khushi Singhal, Harsh Machine Learning Artificial Intelligence Computers and Society Human-Computer Interaction This study applies BERTopic, a transformer-based topic modeling technique, to the lmsys-chat-1m dataset, a multilingual conversational corpus built from head-to-head evaluations of large language models (LLMs). Each user prompt is paired with two anonymized LLM responses and a human preference label, used to assess user evaluation of competing model outputs. The main objective is uncovering thematic patterns in these conversations and examining their relation to user preferences, particularly if certain LLMs are consistently preferred within specific topics. A robust preprocessing pipeline was designed for multilingual variation, balancing dialogue turns, and cleaning noisy or redacted data. BERTopic extracted over 29 coherent topics including artificial intelligence, programming, ethics, and cloud infrastructure. We analysed relationships between topics and model preferences to identify trends in model-topic alignment. Visualization techniques included inter-topic distance maps, topic probability distributions, and model-versus-topic matrices. Our findings inform domain-specific fine-tuning and optimization strategies for improving real-world LLM performance and user satisfaction. |
| title | Investigating Thematic Patterns and User Preferences in LLM Interactions using BERTopic |
| topic | Machine Learning Artificial Intelligence Computers and Society Human-Computer Interaction |
| url | https://arxiv.org/abs/2510.07557 |