Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.17919 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912389653331968 |
|---|---|
| author | Feng, Mingquan Fu, Yifan Zhang, Tongcheng Jiang, Yu Huang, Yixin Yan, Junchi |
| author_facet | Feng, Mingquan Fu, Yifan Zhang, Tongcheng Jiang, Yu Huang, Yixin Yan, Junchi |
| contents | Despite the widely recognized success of residual connections in modern neural networks, their design principles remain largely heuristic. This paper introduces KITINet (Kinetics Theory Inspired Network), a novel architecture that reinterprets feature propagation through the lens of non-equilibrium particle dynamics and partial differential equation (PDE) simulation. At its core, we propose a residual module that models feature updates as the stochastic evolution of a particle system, numerically simulated via a discretized solver for the Boltzmann transport equation (BTE). This formulation mimics particle collisions and energy exchange, enabling adaptive feature refinement via physics-informed interactions. Additionally, we reveal that this mechanism induces network parameter condensation during training, where parameters progressively concentrate into a sparse subset of dominant channels. Experiments on scientific computation (PDE operator), image classification (CIFAR-10/100), and text classification (IMDb/SNLI) show consistent improvements over classic network baselines, with negligible increase of FLOPs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_17919 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | KITINet: Kinetics Theory Inspired Network Architectures with PDE Simulation Approaches Feng, Mingquan Fu, Yifan Zhang, Tongcheng Jiang, Yu Huang, Yixin Yan, Junchi Machine Learning Despite the widely recognized success of residual connections in modern neural networks, their design principles remain largely heuristic. This paper introduces KITINet (Kinetics Theory Inspired Network), a novel architecture that reinterprets feature propagation through the lens of non-equilibrium particle dynamics and partial differential equation (PDE) simulation. At its core, we propose a residual module that models feature updates as the stochastic evolution of a particle system, numerically simulated via a discretized solver for the Boltzmann transport equation (BTE). This formulation mimics particle collisions and energy exchange, enabling adaptive feature refinement via physics-informed interactions. Additionally, we reveal that this mechanism induces network parameter condensation during training, where parameters progressively concentrate into a sparse subset of dominant channels. Experiments on scientific computation (PDE operator), image classification (CIFAR-10/100), and text classification (IMDb/SNLI) show consistent improvements over classic network baselines, with negligible increase of FLOPs. |
| title | KITINet: Kinetics Theory Inspired Network Architectures with PDE Simulation Approaches |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.17919 |