Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915113688104960 |
|---|---|
| author | Cao, Maosong Zhang, Taolin Li, Mo Zhang, Chuyu Liu, Yunxin Duan, Haodong Zhang, Songyang Chen, Kai |
| author_facet | Cao, Maosong Zhang, Taolin Li, Mo Zhang, Chuyu Liu, Yunxin Duan, Haodong Zhang, Songyang Chen, Kai |
| contents | The quality of Supervised Fine-Tuning (SFT) data plays a critical role in enhancing the conversational capabilities of Large Language Models (LLMs). However, as LLMs become more advanced, the availability of high-quality human-annotated SFT data has become a significant bottleneck, necessitating a greater reliance on synthetic training data. In this work, we introduce Condor, a novel two-stage synthetic data generation framework that incorporates World Knowledge Tree and Self-Reflection Refinement to produce high-quality SFT data at scale. Our experimental results demonstrate that a base model fine-tuned on only 20K Condor-generated samples achieves superior performance compared to counterparts. The additional refinement stage in Condor further enables iterative self-improvement for LLMs at various scales (up to 72B), validating the effectiveness of our approach. Furthermore, our investigation into the scaling for synthetic data in post-training reveals substantial unexplored potential for performance improvements, opening promising avenues for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_12273 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement Cao, Maosong Zhang, Taolin Li, Mo Zhang, Chuyu Liu, Yunxin Duan, Haodong Zhang, Songyang Chen, Kai Computation and Language Artificial Intelligence The quality of Supervised Fine-Tuning (SFT) data plays a critical role in enhancing the conversational capabilities of Large Language Models (LLMs). However, as LLMs become more advanced, the availability of high-quality human-annotated SFT data has become a significant bottleneck, necessitating a greater reliance on synthetic training data. In this work, we introduce Condor, a novel two-stage synthetic data generation framework that incorporates World Knowledge Tree and Self-Reflection Refinement to produce high-quality SFT data at scale. Our experimental results demonstrate that a base model fine-tuned on only 20K Condor-generated samples achieves superior performance compared to counterparts. The additional refinement stage in Condor further enables iterative self-improvement for LLMs at various scales (up to 72B), validating the effectiveness of our approach. Furthermore, our investigation into the scaling for synthetic data in post-training reveals substantial unexplored potential for performance improvements, opening promising avenues for future research. |
| title | Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2501.12273 |