Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.20118 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915511493722112 |
|---|---|
| author | Naderi, Babak Cutler, Ross |
| author_facet | Naderi, Babak Cutler, Ross |
| contents | In crowdsourced subjective video quality assessment, practitioners often face a choice between Absolute Category Rating (ACR), ACR with Hidden Reference (ACR-HR), and Comparison Category Rating (CCR). We conducted a P.910-compliant, side-by-side comparison across six studies using 15 talking-head sources of good and fair quality, processed with realistic degradations (blur, scaling, compression, freezing, and their combinations), as well as a practical bitrate-ladder task at 720p and 1080p resolutions. We evaluated statistical efficiency (standard deviations), economic efficiency, and decision agreement. Our results show that ACR-HR and ACR correlate strongly at the condition level, while CCR is more sensitive-capturing improvements beyond the reference. ACR-HR, however, exhibits compressed scale use, particularly for videos with fair source quality. ACR-HR is approximately twice as fast and cost-effective, with lower normalized variability, yet the choice of quality measurement method shifts saturation points and bitrate-ladder recommendations. Finally, we provide practical guidance on when to use each test method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_20118 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Comparative Study of Subjective Video Quality Assessment Test Methods in Crowdsourcing for Varied Use Cases Naderi, Babak Cutler, Ross Multimedia In crowdsourced subjective video quality assessment, practitioners often face a choice between Absolute Category Rating (ACR), ACR with Hidden Reference (ACR-HR), and Comparison Category Rating (CCR). We conducted a P.910-compliant, side-by-side comparison across six studies using 15 talking-head sources of good and fair quality, processed with realistic degradations (blur, scaling, compression, freezing, and their combinations), as well as a practical bitrate-ladder task at 720p and 1080p resolutions. We evaluated statistical efficiency (standard deviations), economic efficiency, and decision agreement. Our results show that ACR-HR and ACR correlate strongly at the condition level, while CCR is more sensitive-capturing improvements beyond the reference. ACR-HR, however, exhibits compressed scale use, particularly for videos with fair source quality. ACR-HR is approximately twice as fast and cost-effective, with lower normalized variability, yet the choice of quality measurement method shifts saturation points and bitrate-ladder recommendations. Finally, we provide practical guidance on when to use each test method. |
| title | Comparative Study of Subjective Video Quality Assessment Test Methods in Crowdsourcing for Varied Use Cases |
| topic | Multimedia |
| url | https://arxiv.org/abs/2509.20118 |