InfiCoEvalChain: A Blockchain-Based Decentralized Framework for Collaborative LLM Evaluation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914315421876224 |
|---|---|
| author | Yang, Yifan Li, Jinjia Li, Kunxi Zheng, Puhao Wang, Yuanyi Qu, Zheyan Yu, Yang Wu, Jianmin Li, Ming Yang, Hongxia |
| author_facet | Yang, Yifan Li, Jinjia Li, Kunxi Zheng, Puhao Wang, Yuanyi Qu, Zheyan Yu, Yang Wu, Jianmin Li, Ming Yang, Hongxia |
| contents | The rapid advancement of large language models (LLMs) demands increasingly reliable evaluation, yet current centralized evaluation suffers from opacity, overfitting, and hardware-induced variance. Our empirical analysis reveals an alarming inconsistency in existing evaluations: the standard deviation across ten repeated runs of a single model on HumanEval (1.67) actually exceeds the performance gap among the top-10 models on the official leaderboard (0.91), rendering current rankings statistically precarious. To mitigate these instabilities, we propose a decentralized evaluation framework that enables hardware and parameter diversity through large-scale benchmarking across heterogeneous compute nodes. By leveraging the blockchain-based protocol, the framework incentivizes global contributors to act as independent validators, using a robust reward system to ensure evaluation integrity and discourage dishonest participation. This collective verification transforms evaluation from a "centralized black box" into a "decentralized endorsement" where multi-party consensus and diverse inference environments yield a more stable, representative metric. Experimental results demonstrate that the decentralized evaluation framework reduces the standard deviation across ten runs on the same model to 0.28. This significant improvement over conventional frameworks ensures higher statistical confidence in model rankings. We have completely implemented this platform and will soon release it to the community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_08229 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | InfiCoEvalChain: A Blockchain-Based Decentralized Framework for Collaborative LLM Evaluation Yang, Yifan Li, Jinjia Li, Kunxi Zheng, Puhao Wang, Yuanyi Qu, Zheyan Yu, Yang Wu, Jianmin Li, Ming Yang, Hongxia Artificial Intelligence Cryptography and Security Machine Learning The rapid advancement of large language models (LLMs) demands increasingly reliable evaluation, yet current centralized evaluation suffers from opacity, overfitting, and hardware-induced variance. Our empirical analysis reveals an alarming inconsistency in existing evaluations: the standard deviation across ten repeated runs of a single model on HumanEval (1.67) actually exceeds the performance gap among the top-10 models on the official leaderboard (0.91), rendering current rankings statistically precarious. To mitigate these instabilities, we propose a decentralized evaluation framework that enables hardware and parameter diversity through large-scale benchmarking across heterogeneous compute nodes. By leveraging the blockchain-based protocol, the framework incentivizes global contributors to act as independent validators, using a robust reward system to ensure evaluation integrity and discourage dishonest participation. This collective verification transforms evaluation from a "centralized black box" into a "decentralized endorsement" where multi-party consensus and diverse inference environments yield a more stable, representative metric. Experimental results demonstrate that the decentralized evaluation framework reduces the standard deviation across ten runs on the same model to 0.28. This significant improvement over conventional frameworks ensures higher statistical confidence in model rankings. We have completely implemented this platform and will soon release it to the community. |
| title | InfiCoEvalChain: A Blockchain-Based Decentralized Framework for Collaborative LLM Evaluation |
| topic | Artificial Intelligence Cryptography and Security Machine Learning |
| url | https://arxiv.org/abs/2602.08229 |