Amazon Nova uses self-reasoning to improve fine-tuning without costly human annotations

Fine-tuning large language models on domain-specific datasets is a common way to customize them for business use, but it creates a practical and performance problem. Creating high-quality chain-of-thought (reasoning traces) for training data is often impractical and prohibitively expensive, so teams frequently skip reasoning during supervised fine-tuning and train only on input-output pairs. However, the body notes that reasoning is a key capability of Amazon Nova 2 models and significantly enhances prediction performance, often delivering substantial gains on hard problems like coding and math.

When models fine-tune on non-reasoning datasets with reasoning mode active, they lose their ability to reason during inference—even when reasoning mode is explicitly turned on. The body attributes this to the training loss being calculated on both reasoning and output tokens together; because the training data contains only inputs and outputs with no intermediate reasoning steps, the model receives no supervision signal for generating coherent reasoning traces. The loss function effectively penalizes reasoning tokens, training the model to bypass its reasoning mechanisms—a phenomenon called shortcut learning. This causes catastrophic forgetting of general capabilities: the body reports that vanilla supervised fine-tuning on non-reasoning datasets drops math performance from 70% to an average of 6%.

Amazon's established solution has been model merging: weighted interpolation between the fine-tuned model and the base model to recover reasoning and general performance. However, merging introduces a trade-off: restoring general performance often sacrifices gains achieved on the target task. To solve this, Amazon researchers introduced Self-Distilled Reasoning (SDR), which works in three stages. First, the base reasoning model (such as Amazon Nova 2 Lite) generates reasoning traces for each example in the SFT dataset. Next, the training data is augmented by prepending these generated reasoning traces to the original outputs. Finally, the model is fine-tuned with reasoning mode turned on, providing supervision on both reasoning and output tokens. Crucially, SDR requires no human annotation, no separate teacher model, and no post-hoc interventions—it works as in-training regularization by augmenting the dataset with the model's own reasoning.

The body validates SDR across five benchmarks spanning diverse tasks: ToolACE (a tool-calling benchmark with 10K samples, scored on strict match of function calls), CoCoHD (long-context structured extraction from congressional hearing transcripts, 730 samples), GovReport (long-context abstractive summarization of U.S. government reports, 17K samples), Invoice-OCR (multimodal document understanding from scanned invoices, approximately 500 samples), and CaptionGen (video captioning, 1800 samples). Results show that SDR achieves nearly identical math performance to model merging (70% compared to 68%) while improving target performance by over 6.5% on average at the same time—demonstrating that SDR preserves both general and target capabilities more effectively than the merging approach. The body frames this as consistent with growing research showing self-distillation (where a model learns from its own predictions) is a powerful mechanism to mitigate catastrophic forgetting.

The core challenge SDR addresses is the "reasoning suppression problem": when fine-tuning models on datasets containing only input-output pairs (no reasoning steps), the training loss is calculated on both reasoning and output tokens together. Because training data has no supervision signal for generating intermediate reasoning steps, the loss function penalizes reasoning tokens that don't directly produce the final output, training the model to skip its reasoning mechanisms entirely—a form of shortcut learning. This causes catastrophic forgetting of general capabilities; the body shows math performance collapsing from 70% to 6% in vanilla supervised fine-tuning.

Amazon's prior recommendation was model merging: weighted interpolation between the fine-tuned model and the base model to recover reasoning and general performance. However, this introduces a trade-off: restoring general performance often sacrifices gains on the target task. SDR sidesteps this by augmenting the training data with the base model's own reasoning traces (chain-of-thought outputs), then fine-tuning with reasoning enabled. This provides what the body calls "in-training regularization"—the model learns from its own problem-solving process rather than just final outputs, maintaining proximity to the base model's policy while improving domain performance. The body frames this as self-distillation, aligning with recent research showing models can learn effectively from their own predictions.

No comments yet. Be the first to share your thoughts!