Evaluating Numerical Reasoning in Text-to-Image Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929699510288384 |
|---|---|
| author | Kajić, Ivana Wiles, Olivia Albuquerque, Isabela Bauer, Matthias Wang, Su Pont-Tuset, Jordi Nematzadeh, Aida |
| author_facet | Kajić, Ivana Wiles, Olivia Albuquerque, Isabela Bauer, Matthias Wang, Su Pont-Tuset, Jordi Nematzadeh, Aida |
| contents | Text-to-image generative models are capable of producing high-quality images that often faithfully depict concepts described using natural language. In this work, we comprehensively evaluate a range of text-to-image models on numerical reasoning tasks of varying difficulty, and show that even the most advanced models have only rudimentary numerical skills. Specifically, their ability to correctly generate an exact number of objects in an image is limited to small numbers, it is highly dependent on the context the number term appears in, and it deteriorates quickly with each successive number. We also demonstrate that models have poor understanding of linguistic quantifiers (such as "a few" or "as many as"), the concept of zero, and struggle with more advanced concepts such as partial quantities and fractional representations. We bundle prompts, generated images and human annotations into GeckoNum, a novel benchmark for evaluation of numerical reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14774 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evaluating Numerical Reasoning in Text-to-Image Models Kajić, Ivana Wiles, Olivia Albuquerque, Isabela Bauer, Matthias Wang, Su Pont-Tuset, Jordi Nematzadeh, Aida Machine Learning Computation and Language Computer Vision and Pattern Recognition Text-to-image generative models are capable of producing high-quality images that often faithfully depict concepts described using natural language. In this work, we comprehensively evaluate a range of text-to-image models on numerical reasoning tasks of varying difficulty, and show that even the most advanced models have only rudimentary numerical skills. Specifically, their ability to correctly generate an exact number of objects in an image is limited to small numbers, it is highly dependent on the context the number term appears in, and it deteriorates quickly with each successive number. We also demonstrate that models have poor understanding of linguistic quantifiers (such as "a few" or "as many as"), the concept of zero, and struggle with more advanced concepts such as partial quantities and fractional representations. We bundle prompts, generated images and human annotations into GeckoNum, a novel benchmark for evaluation of numerical reasoning. |
| title | Evaluating Numerical Reasoning in Text-to-Image Models |
| topic | Machine Learning Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.14774 |