Evaluating Spatiotemporal Consistency in Automatically Generated Sewing Instructions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916976882876416 |
|---|---|
| author | Geiger, Luisa Hartmann, Mareike Sullivan, Michael Koller, Alexander |
| author_facet | Geiger, Luisa Hartmann, Mareike Sullivan, Michael Koller, Alexander |
| contents | In this paper, we propose a novel, automatic tree-based evaluation metric for LLM-generated step-by-step assembly instructions, that more accurately reflects spatiotemporal aspects of construction than traditional metrics such as BLEU and BERT similarity scores. We apply our proposed metric to the domain of sewing instructions, and show that our metric better correlates with manually-annotated error counts as well as human quality ratings, demonstrating our metric's superiority for evaluating the spatiotemporal soundness of sewing instructions. Further experiments show that our metric is more robust than traditional approaches against artificially-constructed counterfactual examples that are specifically constructed to confound metrics that rely on textual similarity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24792 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evaluating Spatiotemporal Consistency in Automatically Generated Sewing Instructions Geiger, Luisa Hartmann, Mareike Sullivan, Michael Koller, Alexander Computation and Language In this paper, we propose a novel, automatic tree-based evaluation metric for LLM-generated step-by-step assembly instructions, that more accurately reflects spatiotemporal aspects of construction than traditional metrics such as BLEU and BERT similarity scores. We apply our proposed metric to the domain of sewing instructions, and show that our metric better correlates with manually-annotated error counts as well as human quality ratings, demonstrating our metric's superiority for evaluating the spatiotemporal soundness of sewing instructions. Further experiments show that our metric is more robust than traditional approaches against artificially-constructed counterfactual examples that are specifically constructed to confound metrics that rely on textual similarity. |
| title | Evaluating Spatiotemporal Consistency in Automatically Generated Sewing Instructions |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2509.24792 |