Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.05848 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911573652537344 |
|---|---|
| author | Park, Junsoo Medhat, Youssef Wai, Htet Phyo Thajchayapong, Ploy Goel, Ashok K. |
| author_facet | Park, Junsoo Medhat, Youssef Wai, Htet Phyo Thajchayapong, Ploy Goel, Ashok K. |
| contents | Learner representations play a central role in educational AI systems, yet it is often unclear whether they preserve meaningful differences between students when instructional outcomes are unavailable or highly context-dependent. This work examines how to evaluate learner representations based on whether they retain separation between learners under a shared comparison rule. We introduce distinctiveness, a representation-level measure that evaluates how each learner differs from others in the cohort using pairwise distances, without requiring clustering, labels, or task-specific evaluation. Using student-authored questions collected through a conversational AI agent in an online learning environment, we compare representations based on individual questions with representations that aggregate patterns across a student's interactions over time. Results show that learner-level representations yield higher separation, stronger clustering structure, and more reliable pairwise discrimination than interaction-level representations. These findings demonstrate that learner representations can be evaluated independently of instructional outcomes and provide a practical pre-deployment criterion using distinctiveness as a diagnostic metric for assessing whether a representation supports differentiated modeling or personalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_05848 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Evaluating Learner Representations for Differentiation Prior to Instructional Outcomes Park, Junsoo Medhat, Youssef Wai, Htet Phyo Thajchayapong, Ploy Goel, Ashok K. Computation and Language Artificial Intelligence Learner representations play a central role in educational AI systems, yet it is often unclear whether they preserve meaningful differences between students when instructional outcomes are unavailable or highly context-dependent. This work examines how to evaluate learner representations based on whether they retain separation between learners under a shared comparison rule. We introduce distinctiveness, a representation-level measure that evaluates how each learner differs from others in the cohort using pairwise distances, without requiring clustering, labels, or task-specific evaluation. Using student-authored questions collected through a conversational AI agent in an online learning environment, we compare representations based on individual questions with representations that aggregate patterns across a student's interactions over time. Results show that learner-level representations yield higher separation, stronger clustering structure, and more reliable pairwise discrimination than interaction-level representations. These findings demonstrate that learner representations can be evaluated independently of instructional outcomes and provide a practical pre-deployment criterion using distinctiveness as a diagnostic metric for assessing whether a representation supports differentiated modeling or personalization. |
| title | Evaluating Learner Representations for Differentiation Prior to Instructional Outcomes |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2604.05848 |