Are Akpans Trick or Treat: Unveiling Helpful Biases in Assistant Systems
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2022
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910852017291264 |
|---|---|
| author | Sun, Jiao Hou, Yu Kim, Jiin Peng, Nanyun |
| author_facet | Sun, Jiao Hou, Yu Kim, Jiin Peng, Nanyun |
| contents | Information-seeking AI assistant systems aim to answer users' queries about knowledge in a timely manner. However, both the human-perceived helpfulness of information-seeking assistant systems and its fairness implication are under-explored. In this paper, we study computational measurements of helpfulness. We collect human annotations on the helpfulness of dialogue responses, develop models for automatic helpfulness evaluation, and then propose to use the helpfulness level of a dialogue system towards different user queries to gauge the fairness of a dialogue system. Experiments with state-of-the-art dialogue systems, including ChatGPT, under three information-seeking scenarios reveal that existing systems tend to be more helpful for questions regarding concepts from highly-developed countries than less-developed countries, uncovering potential fairness concerns underlying the current information-seeking assistant systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2205_12554 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Are Akpans Trick or Treat: Unveiling Helpful Biases in Assistant Systems Sun, Jiao Hou, Yu Kim, Jiin Peng, Nanyun Computation and Language Artificial Intelligence Information-seeking AI assistant systems aim to answer users' queries about knowledge in a timely manner. However, both the human-perceived helpfulness of information-seeking assistant systems and its fairness implication are under-explored. In this paper, we study computational measurements of helpfulness. We collect human annotations on the helpfulness of dialogue responses, develop models for automatic helpfulness evaluation, and then propose to use the helpfulness level of a dialogue system towards different user queries to gauge the fairness of a dialogue system. Experiments with state-of-the-art dialogue systems, including ChatGPT, under three information-seeking scenarios reveal that existing systems tend to be more helpful for questions regarding concepts from highly-developed countries than less-developed countries, uncovering potential fairness concerns underlying the current information-seeking assistant systems. |
| title | Are Akpans Trick or Treat: Unveiling Helpful Biases in Assistant Systems |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2205.12554 |