DeepSeek's R1 uses verifiable rewards and simplified training to match human-preference-based reasoning models with a fraction of the memory

In January 2025, DeepSeek released R1 using RLVR (Reinforcement Learning with Verifiable Rewards) instead of the human-preference training pipeline. For math problems, the verifier checked if the model's answer matched the known solution; for code, a compiler ran the output and returned pass or fail (binary rewards: 1 for correct, 0 for wrong).

DeepSeek's approach uses GRPO (Group Relative Policy Optimization), which removes the critic model and learned reward model entirely. Instead of four full-size models in memory (policy, reference policy, reward model, critic), the system requires just two (the policy being trained and a reference copy for KL regularization), cutting memory demands substantially.

DeepSeek R1-Zero, trained with GRPO and verifiable rewards with no supervised fine-tuning, went from 15.6% to 77.9% on AIME 2024 math problems; with majority voting, it hit 86.7%, matching OpenAI's o1. The model developed self-verification, reflection, and chain-of-thought reasoning purely from the binary signal.