Modeling Writing Traits in a Formative Essay Corpus. Research Report. ETS RR-24-02
Fuente:
ERIC Institute of Education Sciences
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , |
|---|---|
| Formato: | Recurso educativo Open Access |
| Lenguaje: | en |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1867181900134612992 |
|---|---|
| author | Paul Deane Duanli Yan Katherine Castellano Yigal Attali Michelle Lamar Mo Zhang Ian Blood James V. Bruno Chen Li Wenju Cui Chunyi Ruan Colleen Appel Kofi James Rodolfo Long Farah Qureshi |
| author_facet | Paul Deane Duanli Yan Katherine Castellano Yigal Attali Michelle Lamar Mo Zhang Ian Blood James V. Bruno Chen Li Wenju Cui Chunyi Ruan Colleen Appel Kofi James Rodolfo Long Farah Qureshi Paul Deane Duanli Yan Katherine Castellano Yigal Attali Michelle Lamar Mo Zhang Ian Blood James V. Bruno Chen Li Wenju Cui Chunyi Ruan Colleen Appel Kofi James Rodolfo Long Farah Qureshi |
| collection | Education Resources Information Center |
| contents | Modeling Writing Traits in a Formative Essay Corpus. Research Report. ETS RR-24-02 Paul Deane Duanli Yan Katherine Castellano Yigal Attali Michelle Lamar Mo Zhang Ian Blood James V. Bruno Chen Li Wenju Cui Chunyi Ruan Colleen Appel Kofi James Rodolfo Long Farah Qureshi Writing (Composition) Essays Models Elementary School Students Secondary School Students Validity Reliability Natural Language Processing Artificial Intelligence Writing Evaluation Institutional Characteristics Student Characteristics Scoring Automation This paper presents a multidimensional model of variation in writing quality, register, and genre in student essays, trained and tested via confirmatory factor analysis of 1.37 million essay submissions to ETS' digital writing service, Criterion®. The model was also validated with several other corpora, which indicated that it provides a reasonable fit for essay data from 4th grade to college. It includes an analysis of the test-retest reliability of each trait, longitudinal trends by trait, both within the school year and from 4th to 12th grades, and analysis of genre differences by trait, using prompts from the Criterion topic library aligned with the major modes of writing (exposition, argumentation, narrative, description, process, comparison and contrast, and cause and effect). It demonstrates that many of the traits are about as reliable as overall e-rater® scores, that the trait model can be used to build models somewhat more closely aligned with human scores than standard e-rater models, and that there are large, significant trait differences by genre, consistent with genre differences in trait patterns described in the larger literature. Some of the traits demonstrated clear trends between successive revisions. Students using Criterion appear to have consistently improved grammar, usage, and spelling after getting Criterion feedback and to have marginally improved essay organization. Many of the traits also demonstrated clear grade level trends. These features indicate that the trait model could be used to support more detailed scoring and reporting for writing assessments and learning tools. |
| format | Recurso educativo Open Access |
| id | eric_EJ1459602 |
| institution | ERIC Institute of Education Sciences |
| language | en |
| publishDate | 2024 |
| record_format | eric |
| spellingShingle | Modeling Writing Traits in a Formative Essay Corpus. Research Report. ETS RR-24-02 Paul Deane Duanli Yan Katherine Castellano Yigal Attali Michelle Lamar Mo Zhang Ian Blood James V. Bruno Chen Li Wenju Cui Chunyi Ruan Colleen Appel Kofi James Rodolfo Long Farah Qureshi Writing (Composition) Essays Models Elementary School Students Secondary School Students Validity Reliability Natural Language Processing Artificial Intelligence Writing Evaluation Institutional Characteristics Student Characteristics Scoring Automation Modeling Writing Traits in a Formative Essay Corpus. Research Report. ETS RR-24-02 Paul Deane Duanli Yan Katherine Castellano Yigal Attali Michelle Lamar Mo Zhang Ian Blood James V. Bruno Chen Li Wenju Cui Chunyi Ruan Colleen Appel Kofi James Rodolfo Long Farah Qureshi Writing (Composition) Essays Models Elementary School Students Secondary School Students Validity Reliability Natural Language Processing Artificial Intelligence Writing Evaluation Institutional Characteristics Student Characteristics Scoring Automation This paper presents a multidimensional model of variation in writing quality, register, and genre in student essays, trained and tested via confirmatory factor analysis of 1.37 million essay submissions to ETS' digital writing service, Criterion®. The model was also validated with several other corpora, which indicated that it provides a reasonable fit for essay data from 4th grade to college. It includes an analysis of the test-retest reliability of each trait, longitudinal trends by trait, both within the school year and from 4th to 12th grades, and analysis of genre differences by trait, using prompts from the Criterion topic library aligned with the major modes of writing (exposition, argumentation, narrative, description, process, comparison and contrast, and cause and effect). It demonstrates that many of the traits are about as reliable as overall e-rater® scores, that the trait model can be used to build models somewhat more closely aligned with human scores than standard e-rater models, and that there are large, significant trait differences by genre, consistent with genre differences in trait patterns described in the larger literature. Some of the traits demonstrated clear trends between successive revisions. Students using Criterion appear to have consistently improved grammar, usage, and spelling after getting Criterion feedback and to have marginally improved essay organization. Many of the traits also demonstrated clear grade level trends. These features indicate that the trait model could be used to support more detailed scoring and reporting for writing assessments and learning tools. |
| title | Modeling Writing Traits in a Formative Essay Corpus. Research Report. ETS RR-24-02 |
| topic | Writing (Composition) Essays Models Elementary School Students Secondary School Students Validity Reliability Natural Language Processing Artificial Intelligence Writing Evaluation Institutional Characteristics Student Characteristics Scoring Automation |
| url | https://eric.ed.gov/?id=EJ1459602 |