Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.17455 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916263313276928 |
|---|---|
| author | Hasan, Adib Roozbehani, Mardavij Dahleh, Munther |
| author_facet | Hasan, Adib Roozbehani, Mardavij Dahleh, Munther |
| contents | This paper introduces WeatherFormer, a transformer encoder-based model designed to learn robust weather features from minimal observations. It addresses the challenge of modeling complex weather dynamics from small datasets, a bottleneck for many prediction tasks in agriculture, epidemiology, and climate science. WeatherFormer was pretrained on a large pretraining dataset comprised of 39 years of satellite measurements across the Americas. With a novel pretraining task and fine-tuning, WeatherFormer achieves state-of-the-art performance in county-level soybean yield prediction and influenza forecasting. Technical innovations include a unique spatiotemporal encoding that captures geographical, annual, and seasonal variations, adapting the transformer architecture to continuous weather data, and a pretraining strategy to learn representations that are robust to missing weather features. This paper for the first time demonstrates the effectiveness of pretraining large transformer encoder models for weather-dependent applications across multiple domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_17455 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather Representations from Small Datasets Hasan, Adib Roozbehani, Mardavij Dahleh, Munther Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Atmospheric and Oceanic Physics This paper introduces WeatherFormer, a transformer encoder-based model designed to learn robust weather features from minimal observations. It addresses the challenge of modeling complex weather dynamics from small datasets, a bottleneck for many prediction tasks in agriculture, epidemiology, and climate science. WeatherFormer was pretrained on a large pretraining dataset comprised of 39 years of satellite measurements across the Americas. With a novel pretraining task and fine-tuning, WeatherFormer achieves state-of-the-art performance in county-level soybean yield prediction and influenza forecasting. Technical innovations include a unique spatiotemporal encoding that captures geographical, annual, and seasonal variations, adapting the transformer architecture to continuous weather data, and a pretraining strategy to learn representations that are robust to missing weather features. This paper for the first time demonstrates the effectiveness of pretraining large transformer encoder models for weather-dependent applications across multiple domains. |
| title | WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather Representations from Small Datasets |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Atmospheric and Oceanic Physics |
| url | https://arxiv.org/abs/2405.17455 |