Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.16508 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908344163237888 |
|---|---|
| author | Vayani, Ashmal Dissanayake, Dinura Watawana, Hasindri Ahsan, Noor Sasikumar, Nevasini Thawakar, Omkar Ademtew, Henok Biadglign Hmaiti, Yahya Kumar, Amandeep Kuckreja, Kartik Maslych, Mykola Ghallabi, Wafa Al Mihaylov, Mihail Qin, Chao Shaker, Abdelrahman M Zhang, Mike Ihsani, Mahardika Krisna Esplana, Amiel Gokani, Monil Mirkin, Shachar Singh, Harsh Srivastava, Ashay Hamerlik, Endre Izzati, Fathinah Asma Maani, Fadillah Adamsyah Cavada, Sebastian Chim, Jenny Gupta, Rohit Manjunath, Sanjay Zhumakhanova, Kamila Rabevohitra, Feno Heriniaina Amirudin, Azril Ridzuan, Muhammad Kareem, Daniya More, Ketan Li, Kunyang Shakya, Pramesh Saad, Muhammad Ghasemaghaei, Amirpouya Djanibekov, Amirbek Azizov, Dilshod Jankovic, Branislava Bhatia, Naman Cabrera, Alvaro Obando-Ceron, Johan Otieno, Olympiah Farestam, Fabian Rabbani, Muztoba Baliah, Sanoojan Sanjeev, Santosh Shtanchaev, Abduragim Fatima, Maheen Nguyen, Thao Kareem, Amrin Aremu, Toluwani Xavier, Nathan Bhatkal, Amit Toyin, Hawau Chadha, Aman Cholakkal, Hisham Anwer, Rao Muhammad Felsberg, Michael Laaksonen, Jorma Solorio, Thamar Choudhury, Monojit Laptev, Ivan Shah, Mubarak Khan, Salman Khan, Fahad |
| author_facet | Vayani, Ashmal Dissanayake, Dinura Watawana, Hasindri Ahsan, Noor Sasikumar, Nevasini Thawakar, Omkar Ademtew, Henok Biadglign Hmaiti, Yahya Kumar, Amandeep Kuckreja, Kartik Maslych, Mykola Ghallabi, Wafa Al Mihaylov, Mihail Qin, Chao Shaker, Abdelrahman M Zhang, Mike Ihsani, Mahardika Krisna Esplana, Amiel Gokani, Monil Mirkin, Shachar Singh, Harsh Srivastava, Ashay Hamerlik, Endre Izzati, Fathinah Asma Maani, Fadillah Adamsyah Cavada, Sebastian Chim, Jenny Gupta, Rohit Manjunath, Sanjay Zhumakhanova, Kamila Rabevohitra, Feno Heriniaina Amirudin, Azril Ridzuan, Muhammad Kareem, Daniya More, Ketan Li, Kunyang Shakya, Pramesh Saad, Muhammad Ghasemaghaei, Amirpouya Djanibekov, Amirbek Azizov, Dilshod Jankovic, Branislava Bhatia, Naman Cabrera, Alvaro Obando-Ceron, Johan Otieno, Olympiah Farestam, Fabian Rabbani, Muztoba Baliah, Sanoojan Sanjeev, Santosh Shtanchaev, Abduragim Fatima, Maheen Nguyen, Thao Kareem, Amrin Aremu, Toluwani Xavier, Nathan Bhatkal, Amit Toyin, Hawau Chadha, Aman Cholakkal, Hisham Anwer, Rao Muhammad Felsberg, Michael Laaksonen, Jorma Solorio, Thamar Choudhury, Monojit Laptev, Ivan Shah, Mubarak Khan, Salman Khan, Fahad |
| contents | Existing Large Multimodal Models (LMMs) generally focus on only a few regions and languages. As LMMs continue to improve, it is increasingly important to ensure they understand cultural contexts, respect local sensitivities, and support low-resource languages, all while effectively integrating corresponding visual cues. In pursuit of culturally diverse global multimodal models, our proposed All Languages Matter Benchmark (ALM-bench) represents the largest and most comprehensive effort to date for evaluating LMMs across 100 languages. ALM-bench challenges existing models by testing their ability to understand and reason about culturally diverse images paired with text in various languages, including many low-resource languages traditionally underrepresented in LMM research. The benchmark offers a robust and nuanced evaluation framework featuring various question formats, including true/false, multiple choice, and open-ended questions, which are further divided into short and long-answer categories. ALM-bench design ensures a comprehensive assessment of a model's ability to handle varied levels of difficulty in visual and linguistic reasoning. To capture the rich tapestry of global cultures, ALM-bench carefully curates content from 13 distinct cultural aspects, ranging from traditions and rituals to famous personalities and celebrations. Through this, ALM-bench not only provides a rigorous testing ground for state-of-the-art open and closed-source LMMs but also highlights the importance of cultural and linguistic inclusivity, encouraging the development of models that can serve diverse global populations effectively. Our benchmark is publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_16508 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages Vayani, Ashmal Dissanayake, Dinura Watawana, Hasindri Ahsan, Noor Sasikumar, Nevasini Thawakar, Omkar Ademtew, Henok Biadglign Hmaiti, Yahya Kumar, Amandeep Kuckreja, Kartik Maslych, Mykola Ghallabi, Wafa Al Mihaylov, Mihail Qin, Chao Shaker, Abdelrahman M Zhang, Mike Ihsani, Mahardika Krisna Esplana, Amiel Gokani, Monil Mirkin, Shachar Singh, Harsh Srivastava, Ashay Hamerlik, Endre Izzati, Fathinah Asma Maani, Fadillah Adamsyah Cavada, Sebastian Chim, Jenny Gupta, Rohit Manjunath, Sanjay Zhumakhanova, Kamila Rabevohitra, Feno Heriniaina Amirudin, Azril Ridzuan, Muhammad Kareem, Daniya More, Ketan Li, Kunyang Shakya, Pramesh Saad, Muhammad Ghasemaghaei, Amirpouya Djanibekov, Amirbek Azizov, Dilshod Jankovic, Branislava Bhatia, Naman Cabrera, Alvaro Obando-Ceron, Johan Otieno, Olympiah Farestam, Fabian Rabbani, Muztoba Baliah, Sanoojan Sanjeev, Santosh Shtanchaev, Abduragim Fatima, Maheen Nguyen, Thao Kareem, Amrin Aremu, Toluwani Xavier, Nathan Bhatkal, Amit Toyin, Hawau Chadha, Aman Cholakkal, Hisham Anwer, Rao Muhammad Felsberg, Michael Laaksonen, Jorma Solorio, Thamar Choudhury, Monojit Laptev, Ivan Shah, Mubarak Khan, Salman Khan, Fahad Computer Vision and Pattern Recognition Computation and Language Existing Large Multimodal Models (LMMs) generally focus on only a few regions and languages. As LMMs continue to improve, it is increasingly important to ensure they understand cultural contexts, respect local sensitivities, and support low-resource languages, all while effectively integrating corresponding visual cues. In pursuit of culturally diverse global multimodal models, our proposed All Languages Matter Benchmark (ALM-bench) represents the largest and most comprehensive effort to date for evaluating LMMs across 100 languages. ALM-bench challenges existing models by testing their ability to understand and reason about culturally diverse images paired with text in various languages, including many low-resource languages traditionally underrepresented in LMM research. The benchmark offers a robust and nuanced evaluation framework featuring various question formats, including true/false, multiple choice, and open-ended questions, which are further divided into short and long-answer categories. ALM-bench design ensures a comprehensive assessment of a model's ability to handle varied levels of difficulty in visual and linguistic reasoning. To capture the rich tapestry of global cultures, ALM-bench carefully curates content from 13 distinct cultural aspects, ranging from traditions and rituals to famous personalities and celebrations. Through this, ALM-bench not only provides a rigorous testing ground for state-of-the-art open and closed-source LMMs but also highlights the importance of cultural and linguistic inclusivity, encouraging the development of models that can serve diverse global populations effectively. Our benchmark is publicly available. |
| title | All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2411.16508 |