The Moral Foundations Reddit Corpus
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2022
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911523486564352 |
|---|---|
| author | Trager, Jackson Ziabari, Alireza S. Rahmati, Elnaz Davani, Aida Mostafazadeh Golazizian, Preni Karimi-Malekabadi, Farzan Omrani, Ali Li, Zhihe Kennedy, Brendan Chochlakis, Georgios Reimer, Nils Karl Reyes, Melissa Cheng, Kelsey Wei, Mellow Merrifield, Christina Khosravi, Arta Alvarez, Evans Dehghani, Morteza |
| author_facet | Trager, Jackson Ziabari, Alireza S. Rahmati, Elnaz Davani, Aida Mostafazadeh Golazizian, Preni Karimi-Malekabadi, Farzan Omrani, Ali Li, Zhihe Kennedy, Brendan Chochlakis, Georgios Reimer, Nils Karl Reyes, Melissa Cheng, Kelsey Wei, Mellow Merrifield, Christina Khosravi, Arta Alvarez, Evans Dehghani, Morteza |
| contents | Moral framing and sentiment can affect a variety of online and offline behaviors, including donation, environmental action, political engagement, and protest. Various computational methods in Natural Language Processing (NLP) have been used to detect moral sentiment from textual data, but achieving strong performance in such subjective tasks requires large, hand-annotated datasets. Previous corpora annotated for moral sentiment have proven valuable, and have generated new insights both within NLP and across the social sciences, but have been limited to Twitter. To facilitate improving our understanding of the role of moral rhetoric, we present the Moral Foundations Reddit Corpus, a collection of 16,123 English Reddit comments that have been curated from 12 distinct subreddits, hand-annotated by at least three trained annotators for 8 categories of moral sentiment (i.e., Care, Proportionality, Equality, Purity, Authority, Loyalty, Thin Morality, Implicit/Explicit Morality) based on the updated Moral Foundations Theory (MFT) framework. We evaluate baselines using large language models (Llama3-8B, Ministral-8B) in zero-shot, few-shot, and PEFT (Parameter-Efficient Fine-Tuning) settings, comparing their performance to fine-tuned encoder-only models like BERT (Bidirectional Encoder Representations from Transformers). The results show that LLMs continue to lag behind fine-tuned encoders on this subjective task, underscoring the ongoing need for human-annotated moral corpora for AI alignment evaluation.
Keywords: moral sentiment annotation, moral values, moral foundations theory, multi-label text classification, large language models, benchmark dataset, evaluation and alignment resource |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2208_05545 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | The Moral Foundations Reddit Corpus Trager, Jackson Ziabari, Alireza S. Rahmati, Elnaz Davani, Aida Mostafazadeh Golazizian, Preni Karimi-Malekabadi, Farzan Omrani, Ali Li, Zhihe Kennedy, Brendan Chochlakis, Georgios Reimer, Nils Karl Reyes, Melissa Cheng, Kelsey Wei, Mellow Merrifield, Christina Khosravi, Arta Alvarez, Evans Dehghani, Morteza Computation and Language Computers and Society Machine Learning Moral framing and sentiment can affect a variety of online and offline behaviors, including donation, environmental action, political engagement, and protest. Various computational methods in Natural Language Processing (NLP) have been used to detect moral sentiment from textual data, but achieving strong performance in such subjective tasks requires large, hand-annotated datasets. Previous corpora annotated for moral sentiment have proven valuable, and have generated new insights both within NLP and across the social sciences, but have been limited to Twitter. To facilitate improving our understanding of the role of moral rhetoric, we present the Moral Foundations Reddit Corpus, a collection of 16,123 English Reddit comments that have been curated from 12 distinct subreddits, hand-annotated by at least three trained annotators for 8 categories of moral sentiment (i.e., Care, Proportionality, Equality, Purity, Authority, Loyalty, Thin Morality, Implicit/Explicit Morality) based on the updated Moral Foundations Theory (MFT) framework. We evaluate baselines using large language models (Llama3-8B, Ministral-8B) in zero-shot, few-shot, and PEFT (Parameter-Efficient Fine-Tuning) settings, comparing their performance to fine-tuned encoder-only models like BERT (Bidirectional Encoder Representations from Transformers). The results show that LLMs continue to lag behind fine-tuned encoders on this subjective task, underscoring the ongoing need for human-annotated moral corpora for AI alignment evaluation. Keywords: moral sentiment annotation, moral values, moral foundations theory, multi-label text classification, large language models, benchmark dataset, evaluation and alignment resource |
| title | The Moral Foundations Reddit Corpus |
| topic | Computation and Language Computers and Society Machine Learning |
| url | https://arxiv.org/abs/2208.05545 |