From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistants
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913009355456512 |
|---|---|
| author | Rismani, Shalaleh Blodgett, Su Lin Liao, Q. Vera Olteanu, Alexandra Moon, AJung |
| author_facet | Rismani, Shalaleh Blodgett, Su Lin Liao, Q. Vera Olteanu, Alexandra Moon, AJung |
| contents | AI-based writing assistants are ubiquitous, yet little is known about how users' mental models shape their use. We examine two types of mental models -- functional or related to what the system does, and structural or related to how the system works -- and how they affect control behavior -- how users request, accept, or edit AI suggestions as they write -- and writing outcomes. We primed participants ($N = 48$) with different system descriptions to induce these mental models before asking them to complete a cover letter writing task using a writing assistant that occasionally offered preconfigured ungrammatical suggestions to test whether the mental models affected participants' critical oversight. We find that while participants in the structural mental model condition demonstrate a better understanding of the system, this can have a backfiring effect: while these participants judged the system as more usable, they also produced letters with more grammatical errors, highlighting a complex relationship between system understanding, trust, and control in contexts that require user oversight of error-prone AI outputs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_05166 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistants Rismani, Shalaleh Blodgett, Su Lin Liao, Q. Vera Olteanu, Alexandra Moon, AJung Human-Computer Interaction Artificial Intelligence AI-based writing assistants are ubiquitous, yet little is known about how users' mental models shape their use. We examine two types of mental models -- functional or related to what the system does, and structural or related to how the system works -- and how they affect control behavior -- how users request, accept, or edit AI suggestions as they write -- and writing outcomes. We primed participants ($N = 48$) with different system descriptions to induce these mental models before asking them to complete a cover letter writing task using a writing assistant that occasionally offered preconfigured ungrammatical suggestions to test whether the mental models affected participants' critical oversight. We find that while participants in the structural mental model condition demonstrate a better understanding of the system, this can have a backfiring effect: while these participants judged the system as more usable, they also produced letters with more grammatical errors, highlighting a complex relationship between system understanding, trust, and control in contexts that require user oversight of error-prone AI outputs. |
| title | From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistants |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2604.05166 |