From Feature-Based Models to Generative AI: Validity Evidence for Constructed Response Scoring
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915876190552064 |
|---|---|
| author | Casabianca, Jodi M. McCaffrey, Daniel F. Johnson, Matthew S. Alper, Naim Zubenko, Vladimir |
| author_facet | Casabianca, Jodi M. McCaffrey, Daniel F. Johnson, Matthew S. Alper, Naim Zubenko, Vladimir |
| contents | The rapid advancements in large language models and generative artificial intelligence (AI) capabilities are making their broad application in the high-stakes testing context more likely. Use of generative AI in the scoring of constructed responses is particularly appealing because it reduces the effort required for handcrafting features in traditional AI scoring and might even outperform those methods. The purpose of this paper is to highlight the differences in the feature-based and generative AI applications in constructed response scoring systems and propose a set of best practices for the collection of validity evidence to support the use and interpretation of constructed response scores from scoring systems using generative AI. We compare the validity evidence needed in scoring systems using human ratings, feature-based natural language processing AI scoring engines, and generative AI. The evidence needed in the generative AI context is more extensive than in the feature-based scoring context because of the lack of transparency and other concerns unique to generative AI such as consistency. Constructed response score data from a large corpus of independent argumentative essays written by 6-12th grade students demonstrate the collection of validity evidence for different types of scoring systems and highlight the numerous complexities and considerations when making a validity argument for these scores. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_19280 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | From Feature-Based Models to Generative AI: Validity Evidence for Constructed Response Scoring Casabianca, Jodi M. McCaffrey, Daniel F. Johnson, Matthew S. Alper, Naim Zubenko, Vladimir Computation and Language Artificial Intelligence Computers and Society The rapid advancements in large language models and generative artificial intelligence (AI) capabilities are making their broad application in the high-stakes testing context more likely. Use of generative AI in the scoring of constructed responses is particularly appealing because it reduces the effort required for handcrafting features in traditional AI scoring and might even outperform those methods. The purpose of this paper is to highlight the differences in the feature-based and generative AI applications in constructed response scoring systems and propose a set of best practices for the collection of validity evidence to support the use and interpretation of constructed response scores from scoring systems using generative AI. We compare the validity evidence needed in scoring systems using human ratings, feature-based natural language processing AI scoring engines, and generative AI. The evidence needed in the generative AI context is more extensive than in the feature-based scoring context because of the lack of transparency and other concerns unique to generative AI such as consistency. Constructed response score data from a large corpus of independent argumentative essays written by 6-12th grade students demonstrate the collection of validity evidence for different types of scoring systems and highlight the numerous complexities and considerations when making a validity argument for these scores. |
| title | From Feature-Based Models to Generative AI: Validity Evidence for Constructed Response Scoring |
| topic | Computation and Language Artificial Intelligence Computers and Society |
| url | https://arxiv.org/abs/2603.19280 |