KARL framework helps large language models recognize and abstain from answering questions beyond their knowledge to reduce hallucinations

Researchers propose KARL, a reinforcement learning framework that continuously aligns an LLM's (AI that understands and generates text) abstention behavior with its evolving knowledge boundary, using a Knowledge-Boundary-Aware Reward that performs online knowledge boundary estimation using within-group response statistics.

The framework employs a Two-Stage RL Training Strategy that first explores the knowledge boundary and bypasses the 'abstention trap', then converts incorrect answers beyond the knowledge boundary into abstentions without sacrificing accuracy.

Experiments on multiple benchmarks demonstrate that KARL achieves a superior accuracy-hallucination trade-off, effectively suppressing hallucinations while maintaining high accuracy across both in-distribution and out-of-distribution scenarios.