Optimizing Soft Prompt Tuning via Structural Evolution
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908839816724480 |
|---|---|
| author | Huang, Zhenzhen Zhang, Chaoning Bian, Haoyu Zhang, Songbo Tai, Chi-lok Andy Zhang, Jiaquan Qin, Caiyan Qu, Jingjing Ye, Yalan Yang, Yang Shen, Heng Tao |
| author_facet | Huang, Zhenzhen Zhang, Chaoning Bian, Haoyu Zhang, Songbo Tai, Chi-lok Andy Zhang, Jiaquan Qin, Caiyan Qu, Jingjing Ye, Yalan Yang, Yang Shen, Heng Tao |
| contents | Soft prompt tuning leverages continuous embeddings to capture task-specific information in large pre-trained language models (LLMs), achieving competitive performance in few-shot settings. However, soft prompts rely on high-dimensional, implicit representations and lack explicit semantics and traceable training behaviors, which limits their interpretability. To address this limitation, we propose a soft prompt tuning optimization method based on topological morphological evolution. Specifically, we employ persistent homology from topological data analysis (TDA) to quantify the structural representations of soft prompts in continuous parameter space and their training process evolution. Quantitative analysis shows that topologically stable and compact soft prompts achieve better downstream performance. Based on this empirical observation, we construct a loss function for optimizing soft prompt tuning, termed Topological Soft Prompt Loss (TSLoss). TSLoss guides the model to learn structurally stable adaptations by quantifying inter-parameter connectivity and redundancy. Extensive experiments show that training with TSLoss accelerates convergence and improves tuning performance, providing an interpretable method to understand and optimize soft prompt tuning from structural and topological perspectives. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_16500 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Optimizing Soft Prompt Tuning via Structural Evolution Huang, Zhenzhen Zhang, Chaoning Bian, Haoyu Zhang, Songbo Tai, Chi-lok Andy Zhang, Jiaquan Qin, Caiyan Qu, Jingjing Ye, Yalan Yang, Yang Shen, Heng Tao Computation and Language Soft prompt tuning leverages continuous embeddings to capture task-specific information in large pre-trained language models (LLMs), achieving competitive performance in few-shot settings. However, soft prompts rely on high-dimensional, implicit representations and lack explicit semantics and traceable training behaviors, which limits their interpretability. To address this limitation, we propose a soft prompt tuning optimization method based on topological morphological evolution. Specifically, we employ persistent homology from topological data analysis (TDA) to quantify the structural representations of soft prompts in continuous parameter space and their training process evolution. Quantitative analysis shows that topologically stable and compact soft prompts achieve better downstream performance. Based on this empirical observation, we construct a loss function for optimizing soft prompt tuning, termed Topological Soft Prompt Loss (TSLoss). TSLoss guides the model to learn structurally stable adaptations by quantifying inter-parameter connectivity and redundancy. Extensive experiments show that training with TSLoss accelerates convergence and improves tuning performance, providing an interpretable method to understand and optimize soft prompt tuning from structural and topological perspectives. |
| title | Optimizing Soft Prompt Tuning via Structural Evolution |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.16500 |