Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910617850347520 |
|---|---|
| author | Wang, Yifan Stevens, David Shah, Pranay Jiang, Wenwen Liu, Miao Chen, Xu Kuo, Robert Li, Na Gong, Boying Lee, Daniel Hu, Jiabo Zhang, Ning Kamma, Bob |
| author_facet | Wang, Yifan Stevens, David Shah, Pranay Jiang, Wenwen Liu, Miao Chen, Xu Kuo, Robert Li, Na Gong, Boying Lee, Daniel Hu, Jiabo Zhang, Ning Kamma, Bob |
| contents | The growing demand for AI training data has transformed data annotation into a global industry, but traditional approaches relying on human annotators are often time-consuming, labor-intensive, and prone to inconsistent quality. We propose the Model-in-the-Loop (MILO) framework, which integrates AI/ML models into the annotation process. Our research introduces a collaborative paradigm that leverages the strengths of both professional human annotators and large language models (LLMs). By employing LLMs as pre-annotation and real-time assistants, and judges on annotator responses, MILO enables effective interaction patterns between human annotators and LLMs. Three empirical studies on multimodal data annotation demonstrate MILO's efficacy in reducing handling time, improving data quality, and enhancing annotator experiences. We also introduce quality rubrics for flexible evaluation and fine-grained feedback on open-ended annotations. The MILO framework has implications for accelerating AI/ML development, reducing reliance on human annotation alone, and promoting better alignment between human and machine values. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_10702 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMs Wang, Yifan Stevens, David Shah, Pranay Jiang, Wenwen Liu, Miao Chen, Xu Kuo, Robert Li, Na Gong, Boying Lee, Daniel Hu, Jiabo Zhang, Ning Kamma, Bob Human-Computer Interaction Artificial Intelligence Computation and Language Machine Learning The growing demand for AI training data has transformed data annotation into a global industry, but traditional approaches relying on human annotators are often time-consuming, labor-intensive, and prone to inconsistent quality. We propose the Model-in-the-Loop (MILO) framework, which integrates AI/ML models into the annotation process. Our research introduces a collaborative paradigm that leverages the strengths of both professional human annotators and large language models (LLMs). By employing LLMs as pre-annotation and real-time assistants, and judges on annotator responses, MILO enables effective interaction patterns between human annotators and LLMs. Three empirical studies on multimodal data annotation demonstrate MILO's efficacy in reducing handling time, improving data quality, and enhancing annotator experiences. We also introduce quality rubrics for flexible evaluation and fine-grained feedback on open-ended annotations. The MILO framework has implications for accelerating AI/ML development, reducing reliance on human annotation alone, and promoting better alignment between human and machine values. |
| title | Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMs |
| topic | Human-Computer Interaction Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2409.10702 |