On LLM Wizards: Identifying Large Language Models' Behaviors for Wizard of Oz Experiments
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929417544007680 |
|---|---|
| author | Fang, Jingchao Arechiga, Nikos Namaoshi, Keiichi Bravo, Nayeli Hogan, Candice Shamma, David A. |
| author_facet | Fang, Jingchao Arechiga, Nikos Namaoshi, Keiichi Bravo, Nayeli Hogan, Candice Shamma, David A. |
| contents | The Wizard of Oz (WoZ) method is a widely adopted research approach where a human Wizard ``role-plays'' a not readily available technology and interacts with participants to elicit user behaviors and probe the design space. With the growing ability for modern large language models (LLMs) to role-play, one can apply LLMs as Wizards in WoZ experiments with better scalability and lower cost than the traditional approach. However, methodological guidance on responsibly applying LLMs in WoZ experiments and a systematic evaluation of LLMs' role-playing ability are lacking. Through two LLM-powered WoZ studies, we take the first step towards identifying an experiment lifecycle for researchers to safely integrate LLMs into WoZ experiments and interpret data generated from settings that involve Wizards role-played by LLMs. We also contribute a heuristic-based evaluation framework that allows the estimation of LLMs' role-playing ability in WoZ experiments and reveals LLMs' behavior patterns at scale. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_08067 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On LLM Wizards: Identifying Large Language Models' Behaviors for Wizard of Oz Experiments Fang, Jingchao Arechiga, Nikos Namaoshi, Keiichi Bravo, Nayeli Hogan, Candice Shamma, David A. Human-Computer Interaction Artificial Intelligence H.5.m; I.2.7 The Wizard of Oz (WoZ) method is a widely adopted research approach where a human Wizard ``role-plays'' a not readily available technology and interacts with participants to elicit user behaviors and probe the design space. With the growing ability for modern large language models (LLMs) to role-play, one can apply LLMs as Wizards in WoZ experiments with better scalability and lower cost than the traditional approach. However, methodological guidance on responsibly applying LLMs in WoZ experiments and a systematic evaluation of LLMs' role-playing ability are lacking. Through two LLM-powered WoZ studies, we take the first step towards identifying an experiment lifecycle for researchers to safely integrate LLMs into WoZ experiments and interpret data generated from settings that involve Wizards role-played by LLMs. We also contribute a heuristic-based evaluation framework that allows the estimation of LLMs' role-playing ability in WoZ experiments and reveals LLMs' behavior patterns at scale. |
| title | On LLM Wizards: Identifying Large Language Models' Behaviors for Wizard of Oz Experiments |
| topic | Human-Computer Interaction Artificial Intelligence H.5.m; I.2.7 |
| url | https://arxiv.org/abs/2407.08067 |