Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.12767 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929635174907904 |
|---|---|
| author | Xie, Liangru Liu, Hui Zeng, Jingying Tang, Xianfeng Han, Yan Luo, Chen Huang, Jing Li, Zhen Wang, Suhang He, Qi |
| author_facet | Xie, Liangru Liu, Hui Zeng, Jingying Tang, Xianfeng Han, Yan Luo, Chen Huang, Jing Li, Zhen Wang, Suhang He, Qi |
| contents | Large Language Models (LLMs) demonstrate remarkable performance in semantic understanding and generation, yet accurately assessing their output reliability remains a significant challenge. While numerous studies have explored calibration techniques, they primarily focus on White-Box LLMs with accessible parameters. Black-Box LLMs, despite their superior performance, pose heightened requirements for calibration techniques due to their API-only interaction constraints. Although recent researches have achieved breakthroughs in black-box LLMs calibration, a systematic survey of these methodologies is still lacking. To bridge this gap, we presents the first comprehensive survey on calibration techniques for black-box LLMs. We first define the Calibration Process of LLMs as comprising two interrelated key steps: Confidence Estimation and Calibration. Second, we conduct a systematic review of applicable methods within black-box settings, and provide insights on the unique challenges and connections in implementing these key steps. Furthermore, we explore typical applications of Calibration Process in black-box LLMs and outline promising future research directions, providing new perspectives for enhancing reliability and human-machine alignment. This is our GitHub link: https://github.com/LiangruXie/Calibration-Process-in-Black-Box-LLMs |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_12767 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Survey of Calibration Process for Black-Box LLMs Xie, Liangru Liu, Hui Zeng, Jingying Tang, Xianfeng Han, Yan Luo, Chen Huang, Jing Li, Zhen Wang, Suhang He, Qi Artificial Intelligence Computation and Language Large Language Models (LLMs) demonstrate remarkable performance in semantic understanding and generation, yet accurately assessing their output reliability remains a significant challenge. While numerous studies have explored calibration techniques, they primarily focus on White-Box LLMs with accessible parameters. Black-Box LLMs, despite their superior performance, pose heightened requirements for calibration techniques due to their API-only interaction constraints. Although recent researches have achieved breakthroughs in black-box LLMs calibration, a systematic survey of these methodologies is still lacking. To bridge this gap, we presents the first comprehensive survey on calibration techniques for black-box LLMs. We first define the Calibration Process of LLMs as comprising two interrelated key steps: Confidence Estimation and Calibration. Second, we conduct a systematic review of applicable methods within black-box settings, and provide insights on the unique challenges and connections in implementing these key steps. Furthermore, we explore typical applications of Calibration Process in black-box LLMs and outline promising future research directions, providing new perspectives for enhancing reliability and human-machine alignment. This is our GitHub link: https://github.com/LiangruXie/Calibration-Process-in-Black-Box-LLMs |
| title | A Survey of Calibration Process for Black-Box LLMs |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2412.12767 |