Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.17717 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908790116319232 |
|---|---|
| author | Zhang, Kaituo Hu, Mingzhi Le, Hoang Anh Duy Torsha, Fariha Kabir Jiang, Zhimeng Bui, Minh Khai Chang, Chia-Yuan Chuang, Yu-Neng Xiong, Zhen Lin, Ying Wang, Guanchu Zou, Na |
| author_facet | Zhang, Kaituo Hu, Mingzhi Le, Hoang Anh Duy Torsha, Fariha Kabir Jiang, Zhimeng Bui, Minh Khai Chang, Chia-Yuan Chuang, Yu-Neng Xiong, Zhen Lin, Ying Wang, Guanchu Zou, Na |
| contents | Large Language Models (LLMs) have emerged as powerful tools for generating data across various modalities. By transforming data from a scarce resource into a controllable asset, LLMs mitigate the bottlenecks imposed by the acquisition costs of real-world data for model training, evaluation, and system iteration. However, ensuring the high quality of LLM-generated synthetic data remains a critical challenge. Existing research primarily focuses on generation methodologies, with limited direct attention to the quality of the resulting data. Furthermore, most studies are restricted to single modalities, lacking a unified perspective across different data types. To bridge this gap, we propose the \textbf{LLM Data Auditor framework}. In this framework, we first describe how LLMs are utilized to generate data across six distinct modalities. More importantly, we systematically categorize intrinsic metrics for evaluating synthetic data from two dimensions: quality and trustworthiness. This approach shifts the focus from extrinsic evaluation, which relies on downstream task performance, to the inherent properties of the data itself. Using this evaluation system, we analyze the experimental evaluations of representative generation methods for each modality and identify substantial deficiencies in current evaluation practices. Based on these findings, we offer concrete recommendations for the community to improve the evaluation of data generation. Finally, the framework outlines methodologies for the practical application of synthetic data across different modalities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_17717 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | The LLM Data Auditor: A Metric-oriented Survey on Quality and Trustworthiness in Evaluating Synthetic Data Zhang, Kaituo Hu, Mingzhi Le, Hoang Anh Duy Torsha, Fariha Kabir Jiang, Zhimeng Bui, Minh Khai Chang, Chia-Yuan Chuang, Yu-Neng Xiong, Zhen Lin, Ying Wang, Guanchu Zou, Na Artificial Intelligence Machine Learning Large Language Models (LLMs) have emerged as powerful tools for generating data across various modalities. By transforming data from a scarce resource into a controllable asset, LLMs mitigate the bottlenecks imposed by the acquisition costs of real-world data for model training, evaluation, and system iteration. However, ensuring the high quality of LLM-generated synthetic data remains a critical challenge. Existing research primarily focuses on generation methodologies, with limited direct attention to the quality of the resulting data. Furthermore, most studies are restricted to single modalities, lacking a unified perspective across different data types. To bridge this gap, we propose the \textbf{LLM Data Auditor framework}. In this framework, we first describe how LLMs are utilized to generate data across six distinct modalities. More importantly, we systematically categorize intrinsic metrics for evaluating synthetic data from two dimensions: quality and trustworthiness. This approach shifts the focus from extrinsic evaluation, which relies on downstream task performance, to the inherent properties of the data itself. Using this evaluation system, we analyze the experimental evaluations of representative generation methods for each modality and identify substantial deficiencies in current evaluation practices. Based on these findings, we offer concrete recommendations for the community to improve the evaluation of data generation. Finally, the framework outlines methodologies for the practical application of synthetic data across different modalities. |
| title | The LLM Data Auditor: A Metric-oriented Survey on Quality and Trustworthiness in Evaluating Synthetic Data |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2601.17717 |