SweEval: Do LLMs Really Swear? A Safety Benchmark for Testing Limits for Enterprise Use
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918031880355840 |
|---|---|
| author | Patel, Hitesh Laxmichand Agarwal, Amit Das, Arion Kumar, Bhargava Panda, Srikant Pattnayak, Priyaranjan Rafi, Taki Hasan Kumar, Tejaswini Chae, Dong-Kyu |
| author_facet | Patel, Hitesh Laxmichand Agarwal, Amit Das, Arion Kumar, Bhargava Panda, Srikant Pattnayak, Priyaranjan Rafi, Taki Hasan Kumar, Tejaswini Chae, Dong-Kyu |
| contents | Enterprise customers are increasingly adopting Large Language Models (LLMs) for critical communication tasks, such as drafting emails, crafting sales pitches, and composing casual messages. Deploying such models across different regions requires them to understand diverse cultural and linguistic contexts and generate safe and respectful responses. For enterprise applications, it is crucial to mitigate reputational risks, maintain trust, and ensure compliance by effectively identifying and handling unsafe or offensive language. To address this, we introduce SweEval, a benchmark simulating real-world scenarios with variations in tone (positive or negative) and context (formal or informal). The prompts explicitly instruct the model to include specific swear words while completing the task. This benchmark evaluates whether LLMs comply with or resist such inappropriate instructions and assesses their alignment with ethical frameworks, cultural nuances, and language comprehension capabilities. In order to advance research in building ethically aligned AI systems for enterprise use and beyond, we release the dataset and code: https://github.com/amitbcp/multilingual_profanity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_17332 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SweEval: Do LLMs Really Swear? A Safety Benchmark for Testing Limits for Enterprise Use Patel, Hitesh Laxmichand Agarwal, Amit Das, Arion Kumar, Bhargava Panda, Srikant Pattnayak, Priyaranjan Rafi, Taki Hasan Kumar, Tejaswini Chae, Dong-Kyu Computation and Language Artificial Intelligence Machine Learning Multiagent Systems I.2.7; I.2.6 Enterprise customers are increasingly adopting Large Language Models (LLMs) for critical communication tasks, such as drafting emails, crafting sales pitches, and composing casual messages. Deploying such models across different regions requires them to understand diverse cultural and linguistic contexts and generate safe and respectful responses. For enterprise applications, it is crucial to mitigate reputational risks, maintain trust, and ensure compliance by effectively identifying and handling unsafe or offensive language. To address this, we introduce SweEval, a benchmark simulating real-world scenarios with variations in tone (positive or negative) and context (formal or informal). The prompts explicitly instruct the model to include specific swear words while completing the task. This benchmark evaluates whether LLMs comply with or resist such inappropriate instructions and assesses their alignment with ethical frameworks, cultural nuances, and language comprehension capabilities. In order to advance research in building ethically aligned AI systems for enterprise use and beyond, we release the dataset and code: https://github.com/amitbcp/multilingual_profanity. |
| title | SweEval: Do LLMs Really Swear? A Safety Benchmark for Testing Limits for Enterprise Use |
| topic | Computation and Language Artificial Intelligence Machine Learning Multiagent Systems I.2.7; I.2.6 |
| url | https://arxiv.org/abs/2505.17332 |