Cover Image

Unsupervised Pretraining for Fact Verification by Language Model Distillation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Bazaga, Adrián, Liò, Pietro, Micklem, Gos
Format:	Preprint
Published:	2023
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Language Model Knowledge Distillation for Efficient Question Answering in Spanish
by: Bazaga, Adrián, et al.
Published: (2023)

HyperBERT: Mixing Hypergraph-Aware Layers with Language Models for Node Classification on Text-Attributed Hypergraphs
by: Bazaga, Adrián, et al.
Published: (2024)

SQLformer: Deep Auto-Regressive Query Graph Generation for Text-to-SQL Translation
by: Bazaga, Adrián, et al.
Published: (2023)

FLUID-LLM: Learning Computational Fluid Dynamics with Spatiotemporal-aware Large Language Models
by: Zhu, Max, et al.
Published: (2024)

Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization
by: Behnia, Tina, et al.
Published: (2025)

Learning to Reason Over Time: Timeline Self-Reflection for Improved Temporal Reasoning in Language Models
by: Bazaga, Adrián, et al.
Published: (2025)

GenRec: Generative Sequential Recommendation with Large Language Models
by: Cao, Panfeng, et al.
Published: (2024)

Unsupervised Contrast-Consistent Ranking with Language Models
by: Stoehr, Niklas, et al.
Published: (2023)

FactLens: Benchmarking Fine-Grained Fact Verification
by: Mitra, Kushan, et al.
Published: (2024)

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
by: Feng, Jiahai, et al.
Published: (2024)

Challenges and Guidelines in Deep Generative Protein Design: Four Case Studies
by: Zheng, Tianyuan, et al.
Published: (2024)

TabMDA: Tabular Manifold Data Augmentation for Any Classifier using Transformers with In-context Subsetting
by: Margeloiu, Andrei, et al.
Published: (2024)

On the Necessity of Learnable Sheaf Laplacians
by: Caralt, Ferran Hernandez, et al.
Published: (2026)

Strong Teacher Not Needed? On Distillation in LLM Pretraining
by: Lu, Taiming, et al.
Published: (2026)

Fact or Fiction? Improving Fact Verification with Knowledge Graphs through Simplified Subgraph Retrievals
by: Opsahl, Tobias A.
Published: (2024)

tinyCLAP: Distilling Constrastive Language-Audio Pretrained Models
by: Paissan, Francesco, et al.
Published: (2023)

On Large Language Models' Hallucination with Regard to Known Facts
by: Jiang, Che, et al.
Published: (2024)

Factual Consistency of Multilingual Pretrained Language Models
by: Fierro, Constanza, et al.
Published: (2022)

FIRE: Fact-checking with Iterative Retrieval and Verification
by: Xie, Zhuohan, et al.
Published: (2024)

STEP: Scientific Time-Series Encoder Pretraining via Cross-Domain Distillation
by: Zhang, Chen, et al.
Published: (2026)

Ensemble Distillation for Unsupervised Constituency Parsing
by: Shayegh, Behzad, et al.
Published: (2023)

From Data to Knowledge: Evaluating How Efficiently Language Models Learn Facts
by: Christoph, Daniel, et al.
Published: (2025)

On the Generalizability of "Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals"
by: Dotsinski, Asen, et al.
Published: (2025)

Procedural Pretraining: Warming Up Language Models with Abstract Data
by: Jiang, Liangze, et al.
Published: (2026)

The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Models
by: Bhaskar, Adithya, et al.
Published: (2024)

ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
by: Bal, Melis Ilayda, et al.
Published: (2025)

Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models
by: Ruis, Laura, et al.
Published: (2024)

Analyzing Similarity Metrics for Data Selection for Language Model Pretraining
by: Sam, Dylan, et al.
Published: (2025)

OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification
by: Zhou, Yuhang, et al.
Published: (2026)

Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
by: Zhao, Siyan, et al.
Published: (2026)

DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation
by: Maekawa, Aru, et al.
Published: (2024)

RePro: Training Language Models to Faithfully Recycle the Web for Pretraining
by: Yu, Zichun, et al.
Published: (2025)

Data Mixing for Large Language Models Pretraining: A Survey and Outlook
by: Chen, Zhuo, et al.
Published: (2026)

Language Models Improve When Pretraining Data Matches Target Tasks
by: Mizrahi, David, et al.
Published: (2025)

Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
by: Grangier, David, et al.
Published: (2024)

Temporal Entailment Pretraining for Clinical Language Models over EHR Data
by: Tanaka, Tatsunori, et al.
Published: (2025)

AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model
by: Yan, Yongyu, et al.
Published: (2022)

How Does Code Pretraining Affect Language Model Task Performance?
by: Petty, Jackson, et al.
Published: (2024)

Are Large Language Models Table-based Fact-Checkers?
by: Zhang, Hanwen, et al.
Published: (2024)

Distill and Align Decomposition for Enhanced Claim Verification
by: Magomere, Jabez, et al.
Published: (2026)