Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915317099266048 |
|---|---|
| author | Chehade, Mohamad Ghosal, Soumya Suvra Chakraborty, Souradip Reddy, Avinash Manocha, Dinesh Zhu, Hao Bedi, Amrit Singh |
| author_facet | Chehade, Mohamad Ghosal, Soumya Suvra Chakraborty, Souradip Reddy, Avinash Manocha, Dinesh Zhu, Hao Bedi, Amrit Singh |
| contents | Aligning large language models with humans is challenging due to the inherently multifaceted nature of preference feedback. While existing approaches typically frame this as a multi-objective optimization problem, they often overlook how humans actually make decisions. Research on bounded rationality suggests that human decision making follows satisficing strategies-optimizing primary objectives while ensuring others meet acceptable thresholds. To bridge this gap and operationalize the notion of satisficing alignment, we propose SITAlign: an inference time framework that addresses the multifaceted nature of alignment by maximizing a primary objective while satisfying threshold-based constraints on secondary criteria. We provide theoretical insights by deriving sub-optimality bounds of our satisficing based inference alignment approach. We empirically validate SITAlign's performance through extensive experimentation on multiple benchmarks. For instance, on the PKU-SafeRLHF dataset with the primary objective of maximizing helpfulness while ensuring a threshold on harmlessness, SITAlign outperforms the state-of-the-art multi objective decoding strategy by a margin of 22.3% in terms of GPT-4 win-tie rate for helpfulness reward while adhering to the threshold on harmlessness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23729 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time Chehade, Mohamad Ghosal, Soumya Suvra Chakraborty, Souradip Reddy, Avinash Manocha, Dinesh Zhu, Hao Bedi, Amrit Singh Computation and Language Artificial Intelligence Aligning large language models with humans is challenging due to the inherently multifaceted nature of preference feedback. While existing approaches typically frame this as a multi-objective optimization problem, they often overlook how humans actually make decisions. Research on bounded rationality suggests that human decision making follows satisficing strategies-optimizing primary objectives while ensuring others meet acceptable thresholds. To bridge this gap and operationalize the notion of satisficing alignment, we propose SITAlign: an inference time framework that addresses the multifaceted nature of alignment by maximizing a primary objective while satisfying threshold-based constraints on secondary criteria. We provide theoretical insights by deriving sub-optimality bounds of our satisficing based inference alignment approach. We empirically validate SITAlign's performance through extensive experimentation on multiple benchmarks. For instance, on the PKU-SafeRLHF dataset with the primary objective of maximizing helpfulness while ensuring a threshold on harmlessness, SITAlign outperforms the state-of-the-art multi objective decoding strategy by a margin of 22.3% in terms of GPT-4 win-tie rate for helpfulness reward while adhering to the threshold on harmlessness. |
| title | Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.23729 |