ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914783334236160 |
|---|---|
| author | Arawjo, Ian Swoopes, Chelse Vaithilingam, Priyan Wattenberg, Martin Glassman, Elena |
| author_facet | Arawjo, Ian Swoopes, Chelse Vaithilingam, Priyan Wattenberg, Martin Glassman, Elena |
| contents | Evaluating outputs of large language models (LLMs) is challenging, requiring making -- and making sense of -- many responses. Yet tools that go beyond basic prompting tend to require knowledge of programming APIs, focus on narrow domains, or are closed-source. We present ChainForge, an open-source visual toolkit for prompt engineering and on-demand hypothesis testing of text generation LLMs. ChainForge provides a graphical interface for comparison of responses across models and prompt variations. Our system was designed to support three tasks: model selection, prompt template design, and hypothesis testing (e.g., auditing). We released ChainForge early in its development and iterated on its design with academics and online users. Through in-lab and interview studies, we find that a range of people could use ChainForge to investigate hypotheses that matter to them, including in real-world settings. We identify three modes of prompt engineering and LLM hypothesis testing: opportunistic exploration, limited evaluation, and iterative refinement. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_09128 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing Arawjo, Ian Swoopes, Chelse Vaithilingam, Priyan Wattenberg, Martin Glassman, Elena Human-Computer Interaction Artificial Intelligence H.5.2; I.2 Evaluating outputs of large language models (LLMs) is challenging, requiring making -- and making sense of -- many responses. Yet tools that go beyond basic prompting tend to require knowledge of programming APIs, focus on narrow domains, or are closed-source. We present ChainForge, an open-source visual toolkit for prompt engineering and on-demand hypothesis testing of text generation LLMs. ChainForge provides a graphical interface for comparison of responses across models and prompt variations. Our system was designed to support three tasks: model selection, prompt template design, and hypothesis testing (e.g., auditing). We released ChainForge early in its development and iterated on its design with academics and online users. Through in-lab and interview studies, we find that a range of people could use ChainForge to investigate hypotheses that matter to them, including in real-world settings. We identify three modes of prompt engineering and LLM hypothesis testing: opportunistic exploration, limited evaluation, and iterative refinement. |
| title | ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing |
| topic | Human-Computer Interaction Artificial Intelligence H.5.2; I.2 |
| url | https://arxiv.org/abs/2309.09128 |