Towards Application-Specific Evaluation of Vision Models: Case Studies in Ecology and Biology
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917300922220544 |
|---|---|
| author | Chan, Alex Hoi Hang Brookes, Otto Waldmann, Urs Naik, Hemal Couzin, Iain D. Mirmehdi, Majid Houa, Noël Adiko Normand, Emmanuelle Boesch, Christophe Boesch, Lukas Arandjelovic, Mimi Kühl, Hjalmar Burghardt, Tilo Kano, Fumihiro |
| author_facet | Chan, Alex Hoi Hang Brookes, Otto Waldmann, Urs Naik, Hemal Couzin, Iain D. Mirmehdi, Majid Houa, Noël Adiko Normand, Emmanuelle Boesch, Christophe Boesch, Lukas Arandjelovic, Mimi Kühl, Hjalmar Burghardt, Tilo Kano, Fumihiro |
| contents | Computer vision methods have demonstrated considerable potential to streamline ecological and biological workflows, with a growing number of datasets and models becoming available to the research community. However, these resources focus predominantly on evaluation using machine learning metrics, with relatively little emphasis on how their application impacts downstream analysis. We argue that models should be evaluated using application-specific metrics that directly represent model performance in the context of its final use case. To support this argument, we present two disparate case studies: (1) estimating chimpanzee abundance and density with camera trap distance sampling when using a video-based behaviour classifier and (2) estimating head rotation in pigeons using a 3D posture estimator. We show that even models with strong machine learning performance (e.g., 87% mAP) can yield data that leads to discrepancies in abundance estimates compared to expert-derived data. Similarly, the highest-performing models for posture estimation do not produce the most accurate inferences of gaze direction in pigeons. Motivated by these findings, we call for researchers to integrate application-specific metrics in ecological/biological datasets, allowing for models to be benchmarked in the context of their downstream application and to facilitate better integration of models into application workflows. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_02825 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Application-Specific Evaluation of Vision Models: Case Studies in Ecology and Biology Chan, Alex Hoi Hang Brookes, Otto Waldmann, Urs Naik, Hemal Couzin, Iain D. Mirmehdi, Majid Houa, Noël Adiko Normand, Emmanuelle Boesch, Christophe Boesch, Lukas Arandjelovic, Mimi Kühl, Hjalmar Burghardt, Tilo Kano, Fumihiro Computer Vision and Pattern Recognition Computer vision methods have demonstrated considerable potential to streamline ecological and biological workflows, with a growing number of datasets and models becoming available to the research community. However, these resources focus predominantly on evaluation using machine learning metrics, with relatively little emphasis on how their application impacts downstream analysis. We argue that models should be evaluated using application-specific metrics that directly represent model performance in the context of its final use case. To support this argument, we present two disparate case studies: (1) estimating chimpanzee abundance and density with camera trap distance sampling when using a video-based behaviour classifier and (2) estimating head rotation in pigeons using a 3D posture estimator. We show that even models with strong machine learning performance (e.g., 87% mAP) can yield data that leads to discrepancies in abundance estimates compared to expert-derived data. Similarly, the highest-performing models for posture estimation do not produce the most accurate inferences of gaze direction in pigeons. Motivated by these findings, we call for researchers to integrate application-specific metrics in ecological/biological datasets, allowing for models to be benchmarked in the context of their downstream application and to facilitate better integration of models into application workflows. |
| title | Towards Application-Specific Evaluation of Vision Models: Case Studies in Ecology and Biology |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.02825 |