Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2410.14375 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910213397807104 |
|---|---|
| author | Yu, Jialin Zhou, Yuxiang Li, Haoxuan Yu, Junchi Yang, Mengyue He, Yulan Zhang, Nevin L. Torr, Philip Silva, Ricardo |
| author_facet | Yu, Jialin Zhou, Yuxiang Li, Haoxuan Yu, Junchi Yang, Mengyue He, Yulan Zhang, Nevin L. Torr, Philip Silva, Ricardo |
| contents | Adapting to latent confounded shift remains a core challenge in modern AI. This setting is driven by hidden variables that induce spurious correlations between inputs and outputs during training, leading models to rely on non-causal shortcuts. For example, a model may learn to treat metadata (e.g., data source like "Amazon") as a proxy for positive sentiment, causing failure when the source becomes predominantly negative during deployment. To address this latent confounded shift, we introduce Causal Fine-Tuning(CFT). Using a structural causal model as an inductive bias, we derive sufficient identification conditions that motivate a fine-tuning objective for decomposing representations into high-level stable and low-level shift-sensitive components. Instantiating this framework in BERT, we show that learning such causal/spurious representations and adjusting them accordingly yield a more robust predictor. Experiments on spurious correlation injection attacks in text demonstrate that our method outperforms black-box domain generalization baselines, highlighting the benefits of explicitly modeling causal structure. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_14375 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Causal Fine-Tuning under Latent Confounded Shift Yu, Jialin Zhou, Yuxiang Li, Haoxuan Yu, Junchi Yang, Mengyue He, Yulan Zhang, Nevin L. Torr, Philip Silva, Ricardo Machine Learning Computation and Language Adapting to latent confounded shift remains a core challenge in modern AI. This setting is driven by hidden variables that induce spurious correlations between inputs and outputs during training, leading models to rely on non-causal shortcuts. For example, a model may learn to treat metadata (e.g., data source like "Amazon") as a proxy for positive sentiment, causing failure when the source becomes predominantly negative during deployment. To address this latent confounded shift, we introduce Causal Fine-Tuning(CFT). Using a structural causal model as an inductive bias, we derive sufficient identification conditions that motivate a fine-tuning objective for decomposing representations into high-level stable and low-level shift-sensitive components. Instantiating this framework in BERT, we show that learning such causal/spurious representations and adjusting them accordingly yield a more robust predictor. Experiments on spurious correlation injection attacks in text demonstrate that our method outperforms black-box domain generalization baselines, highlighting the benefits of explicitly modeling causal structure. |
| title | Causal Fine-Tuning under Latent Confounded Shift |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2410.14375 |