Uncertainty Makes It Stable: Curiosity-Driven Quantized Mixture-of-Experts
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917360439394304 |
|---|---|
| author | Ordóñez, Sebastián Andrés Cajas Torres, Luis Fernando Torres Meni, Mackenzie J. Paredes, Carlos Andrés Duran Arazo, Eric Bosch, Cristian Carbajo, Ricardo Simon Lai, Yuan Celi, Leo Anthony |
| author_facet | Ordóñez, Sebastián Andrés Cajas Torres, Luis Fernando Torres Meni, Mackenzie J. Paredes, Carlos Andrés Duran Arazo, Eric Bosch, Cristian Carbajo, Ricardo Simon Lai, Yuan Celi, Leo Anthony |
| contents | Deploying deep neural networks on resource-constrained devices faces two critical challenges: maintaining accuracy under aggressive quantization while ensuring predictable inference latency. We present a curiosity-driven quantized Mixture-of-Experts framework that addresses both through Bayesian epistemic uncertainty-based routing across heterogeneous experts (BitNet ternary, 1-16 bit BitLinear, post-training quantization). Evaluated on audio classification benchmarks (ESC-50, Quinn, UrbanSound8K), our 4-bit quantization maintains 99.9 percent of full-precision F1 (0.858 vs 0.859) with 4x compression and 31 percent energy savings versus 8-bit, while both achieve statistical parity with full precision (p > 0.05).
Crucially, curiosity-driven routing simultaneously improves accuracy and stability: on Quinn, F1 increases from 0.802 to 0.809 while cross-fold variance drops by 85 percent (p < 0.001, Levene's test), with reductions of 50 to 94 percent across datasets. The routing is self-organizing, with the high-precision 8-bit expert automatically receiving the most uncertain samples (20 percent lower confidence, p < 0.001), while lightweight experts handle easier inputs. Datasets with already low baseline variance show no artificial stability gain, confirming the mechanism targets genuine epistemic uncertainty rather than overfitting routing decisions.
At 1.2M parameters, the framework provides interpretable, precision-aware routing suitable for safety-sensitive edge deployments where both accuracy and predictability are critical. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_11743 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Uncertainty Makes It Stable: Curiosity-Driven Quantized Mixture-of-Experts Ordóñez, Sebastián Andrés Cajas Torres, Luis Fernando Torres Meni, Mackenzie J. Paredes, Carlos Andrés Duran Arazo, Eric Bosch, Cristian Carbajo, Ricardo Simon Lai, Yuan Celi, Leo Anthony Machine Learning Artificial Intelligence Deploying deep neural networks on resource-constrained devices faces two critical challenges: maintaining accuracy under aggressive quantization while ensuring predictable inference latency. We present a curiosity-driven quantized Mixture-of-Experts framework that addresses both through Bayesian epistemic uncertainty-based routing across heterogeneous experts (BitNet ternary, 1-16 bit BitLinear, post-training quantization). Evaluated on audio classification benchmarks (ESC-50, Quinn, UrbanSound8K), our 4-bit quantization maintains 99.9 percent of full-precision F1 (0.858 vs 0.859) with 4x compression and 31 percent energy savings versus 8-bit, while both achieve statistical parity with full precision (p > 0.05). Crucially, curiosity-driven routing simultaneously improves accuracy and stability: on Quinn, F1 increases from 0.802 to 0.809 while cross-fold variance drops by 85 percent (p < 0.001, Levene's test), with reductions of 50 to 94 percent across datasets. The routing is self-organizing, with the high-precision 8-bit expert automatically receiving the most uncertain samples (20 percent lower confidence, p < 0.001), while lightweight experts handle easier inputs. Datasets with already low baseline variance show no artificial stability gain, confirming the mechanism targets genuine epistemic uncertainty rather than overfitting routing decisions. At 1.2M parameters, the framework provides interpretable, precision-aware routing suitable for safety-sensitive edge deployments where both accuracy and predictability are critical. |
| title | Uncertainty Makes It Stable: Curiosity-Driven Quantized Mixture-of-Experts |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2511.11743 |