Reasoning before Comparison: LLM-Enhanced Semantic Similarity Metrics for Domain Specialized Text Analysis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910338769747968 |
|---|---|
| author | Xu, Shaochen Wu, Zihao Zhao, Huaqin Shu, Peng Liu, Zhengliang Liao, Wenxiong Li, Sheng Sikora, Andrea Liu, Tianming Li, Xiang |
| author_facet | Xu, Shaochen Wu, Zihao Zhao, Huaqin Shu, Peng Liu, Zhengliang Liao, Wenxiong Li, Sheng Sikora, Andrea Liu, Tianming Li, Xiang |
| contents | In this study, we leverage LLM to enhance the semantic analysis and develop similarity metrics for texts, addressing the limitations of traditional unsupervised NLP metrics like ROUGE and BLEU. We develop a framework where LLMs such as GPT-4 are employed for zero-shot text identification and label generation for radiology reports, where the labels are then used as measurements for text similarity. By testing the proposed framework on the MIMIC data, we find that GPT-4 generated labels can significantly improve the semantic similarity assessment, with scores more closely aligned with clinical ground truth than traditional NLP metrics. Our work demonstrates the possibility of conducting semantic analysis of the text data using semi-quantitative reasoning results by the LLMs for highly specialized domains. While the framework is implemented for radiology report similarity analysis, its concept can be extended to other specialized domains as well. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_11398 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Reasoning before Comparison: LLM-Enhanced Semantic Similarity Metrics for Domain Specialized Text Analysis Xu, Shaochen Wu, Zihao Zhao, Huaqin Shu, Peng Liu, Zhengliang Liao, Wenxiong Li, Sheng Sikora, Andrea Liu, Tianming Li, Xiang Computation and Language Artificial Intelligence In this study, we leverage LLM to enhance the semantic analysis and develop similarity metrics for texts, addressing the limitations of traditional unsupervised NLP metrics like ROUGE and BLEU. We develop a framework where LLMs such as GPT-4 are employed for zero-shot text identification and label generation for radiology reports, where the labels are then used as measurements for text similarity. By testing the proposed framework on the MIMIC data, we find that GPT-4 generated labels can significantly improve the semantic similarity assessment, with scores more closely aligned with clinical ground truth than traditional NLP metrics. Our work demonstrates the possibility of conducting semantic analysis of the text data using semi-quantitative reasoning results by the LLMs for highly specialized domains. While the framework is implemented for radiology report similarity analysis, its concept can be extended to other specialized domains as well. |
| title | Reasoning before Comparison: LLM-Enhanced Semantic Similarity Metrics for Domain Specialized Text Analysis |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.11398 |