Anna Karenina Strikes Again: Pre-Trained LLM Embeddings May Favor High-Performing Learners
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910480923099136 |
|---|---|
| author | Schleifer, Abigail Gurin Klebanov, Beata Beigman Ariely, Moriah Alexandron, Giora |
| author_facet | Schleifer, Abigail Gurin Klebanov, Beata Beigman Ariely, Moriah Alexandron, Giora |
| contents | Unsupervised clustering of student responses to open-ended questions into behavioral and cognitive profiles using pre-trained LLM embeddings is an emerging technique, but little is known about how well this captures pedagogically meaningful information. We investigate this in the context of student responses to open-ended questions in biology, which were previously analyzed and clustered by experts into theory-driven Knowledge Profiles (KPs). Comparing these KPs to ones discovered by purely data-driven clustering techniques, we report poor discoverability of most KPs, except for the ones including the correct answers. We trace this "discoverability bias" to the representations of KPs in the pre-trained LLM embeddings space. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06599 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Anna Karenina Strikes Again: Pre-Trained LLM Embeddings May Favor High-Performing Learners Schleifer, Abigail Gurin Klebanov, Beata Beigman Ariely, Moriah Alexandron, Giora Computation and Language Artificial Intelligence Computers and Society Human-Computer Interaction Information Retrieval Machine Learning Unsupervised clustering of student responses to open-ended questions into behavioral and cognitive profiles using pre-trained LLM embeddings is an emerging technique, but little is known about how well this captures pedagogically meaningful information. We investigate this in the context of student responses to open-ended questions in biology, which were previously analyzed and clustered by experts into theory-driven Knowledge Profiles (KPs). Comparing these KPs to ones discovered by purely data-driven clustering techniques, we report poor discoverability of most KPs, except for the ones including the correct answers. We trace this "discoverability bias" to the representations of KPs in the pre-trained LLM embeddings space. |
| title | Anna Karenina Strikes Again: Pre-Trained LLM Embeddings May Favor High-Performing Learners |
| topic | Computation and Language Artificial Intelligence Computers and Society Human-Computer Interaction Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2406.06599 |