| _version_ | 1866901868094947328 |
|---|---|
| author | Martinelli, Marco Silvello, Gianmaria Bonato, Vanessa Di Nunzio, Giorgio Maria Ferro, Nicola Irrera, Ornella Marchesin, Stefano Menotti, Laura Vezzani, Federica Nikolova-Koleva, Ivelina |
| author_facet | Martinelli, Marco Silvello, Gianmaria Bonato, Vanessa Di Nunzio, Giorgio Maria Ferro, Nicola Irrera, Ornella Marchesin, Stefano Menotti, Laura Vezzani, Federica Nikolova-Koleva, Ivelina |
| contents | <p>Deliverable 5.10 presents the GUTBRAINIE evaluation challenge at CLEF 2025, advancing Information Extraction from biomedical literature on the gut-brain axis within the HEREDITARY project. The team behind the challenge developed an expert-curated dataset and annotation framework for biomedical IE tasks, benchmarked state-of-the-art NLP systems, and ensured FAIR compliance and EOSC integration.</p> <p>The gut-brain axis is a rapidly emerging research area with profound implications for neurological and psychiatric disorders. Publications have grown exponentially from approximately 600 articles in 2020 to over 1,800 annually by 2025. The GUTBRAINIE task targeted the extraction of structured information from PubMed abstracts to address the challenge of systematic knowledge discovery from this rapidly expanding literature.</p> <p>Dataset construction followed a rigorous multi-stage workflow. An initial PubMed retrieval yielded 1,647 unique articles. A diverse annotation team comprising 40 layperson annotators and 7 biomedical experts, supported by automated pre-annotation using GLiNER zero-shot NER models, systematically annotated documents. The resulting dataset comprises 1,600+ annotated documents stratified into four quality tiers. The gold-standard evaluation set contains 80 expert-annotated documents. The annotation schema identified 13 entity types and 17 fine-grained relation predicates with standardized URIs.</p> <p>The challenge featured four subtasks of increasing complexity: Named Entity Recognition, Binary Tag-based Relation Extraction, Ternary Tag-based Relation Extraction, and Ternary Mention-based Relation Extraction.</p> <p>International participation included 85 teams (17 submitted the final runs) from 29 countries submitting 391 runs total. Top-performing systems on NER achieved micro-averaged F1-scores above 0.84 using ensemble methods with fine-tuned biomedical transformers (PubMedBERT, BioBERT, BioLinkBERT). Relation extraction performance declined with task complexity: Binary Tag-based (F1 ≈ 0.69), Ternary Tag-based (F1 ≈ 0.69), and Ternary Mention-based (F1 ≈ 0.46). Fine-tuned biomedical models substantially outperformed zero-shot LLM approaches, indicating that supervised domain-specific fine-tuning remains essential.</p> <p>Following FAIR principles, the GutBrainIE dataset, annotation guidelines, and baseline system are publicly available on Zenodo (DOI: 10.5281/zenodo.16845408) in multiple formats (JSON, CSV, TSV). Since Zenodo is part of the EOSC core infrastructure, these resources are automatically indexed and discoverable in EOSC by default. The dataset includes metadata on annotator expertise and quality, enabling reliability-aware training. EOSC integration provides the research community with open access to expert-curated biomedical IE benchmarks supporting reproducible research.</p> <p>The open-source baseline system and Codabench competitions enable community-driven advancement of biomedical NLP methods. This work directly supports HEREDITARY’s objectives of semantic data integration and knowledge discovery.</p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_18184680 |
| institution | Zenodo |
| language | |
| publishDate | 2025 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Deliverable 5.10: First evaluation challenge. Report on the data, results, and integration with EOSC Martinelli, Marco Silvello, Gianmaria Bonato, Vanessa Di Nunzio, Giorgio Maria Ferro, Nicola Irrera, Ornella Marchesin, Stefano Menotti, Laura Vezzani, Federica Nikolova-Koleva, Ivelina <p>Deliverable 5.10 presents the GUTBRAINIE evaluation challenge at CLEF 2025, advancing Information Extraction from biomedical literature on the gut-brain axis within the HEREDITARY project. The team behind the challenge developed an expert-curated dataset and annotation framework for biomedical IE tasks, benchmarked state-of-the-art NLP systems, and ensured FAIR compliance and EOSC integration.</p> <p>The gut-brain axis is a rapidly emerging research area with profound implications for neurological and psychiatric disorders. Publications have grown exponentially from approximately 600 articles in 2020 to over 1,800 annually by 2025. The GUTBRAINIE task targeted the extraction of structured information from PubMed abstracts to address the challenge of systematic knowledge discovery from this rapidly expanding literature.</p> <p>Dataset construction followed a rigorous multi-stage workflow. An initial PubMed retrieval yielded 1,647 unique articles. A diverse annotation team comprising 40 layperson annotators and 7 biomedical experts, supported by automated pre-annotation using GLiNER zero-shot NER models, systematically annotated documents. The resulting dataset comprises 1,600+ annotated documents stratified into four quality tiers. The gold-standard evaluation set contains 80 expert-annotated documents. The annotation schema identified 13 entity types and 17 fine-grained relation predicates with standardized URIs.</p> <p>The challenge featured four subtasks of increasing complexity: Named Entity Recognition, Binary Tag-based Relation Extraction, Ternary Tag-based Relation Extraction, and Ternary Mention-based Relation Extraction.</p> <p>International participation included 85 teams (17 submitted the final runs) from 29 countries submitting 391 runs total. Top-performing systems on NER achieved micro-averaged F1-scores above 0.84 using ensemble methods with fine-tuned biomedical transformers (PubMedBERT, BioBERT, BioLinkBERT). Relation extraction performance declined with task complexity: Binary Tag-based (F1 ≈ 0.69), Ternary Tag-based (F1 ≈ 0.69), and Ternary Mention-based (F1 ≈ 0.46). Fine-tuned biomedical models substantially outperformed zero-shot LLM approaches, indicating that supervised domain-specific fine-tuning remains essential.</p> <p>Following FAIR principles, the GutBrainIE dataset, annotation guidelines, and baseline system are publicly available on Zenodo (DOI: 10.5281/zenodo.16845408) in multiple formats (JSON, CSV, TSV). Since Zenodo is part of the EOSC core infrastructure, these resources are automatically indexed and discoverable in EOSC by default. The dataset includes metadata on annotator expertise and quality, enabling reliability-aware training. EOSC integration provides the research community with open access to expert-curated biomedical IE benchmarks supporting reproducible research.</p> <p>The open-source baseline system and Codabench competitions enable community-driven advancement of biomedical NLP methods. This work directly supports HEREDITARY’s objectives of semantic data integration and knowledge discovery.</p> |
| title | Deliverable 5.10: First evaluation challenge. Report on the data, results, and integration with EOSC |
| url | https://doi.org/10.5281/zenodo.18184680 |