Researcher RL-trains model to RL-train other models

The practitioner built a system in which Qwen3.6-35B-A3B, a large language model with 35 billion parameters, learns through reinforcement learning to generate complete training configurations for smaller models. The trainer writes three components for each job: a verification environment and rubric (to evaluate the trained model), a dataset, and hyperparameters. These jobs are dispatched to a pool of up to 16 Runpod GPU pods, where a lightweight RL framework called prime-rl runs a GRPO training algorithm on small Qwen models (either 0.6B or 1.7B parameters). After training, each model is evaluated on a hidden benchmark, and its improvement from pre- to post-training becomes the reward signal fed back to the outer RL loop. The trainer agent itself was optimized using Tinker, a technique combining LoRA (a parameter-efficient fine-tuning method) with GRPO (a reinforcement learning algorithm). To test generalization, the practitioner created six task families and withheld one entirely from training, using it only as a test of whether the agent had learned to write training jobs that work on novel problems. The results show episode reward climbing from approximately 0.0 to a peak of approximately 0.63, indicating the agent progressively learned to write better training configurations. The practitioner frames this as an instance of recursive RL—using reinforcement learning to train a system that itself applies reinforcement learning—and emphasizes both the technical novelty and the conceptual appeal of automating the decision-making that normally requires human expertise in machine learning engineering.

This work sits at the intersection of two trends in machine learning: the use of large language models to generate code and configurations, and reinforcement learning applied to meta-learning problems. Rather than treat hyperparameter tuning as a separate search problem, the practitioner embedded it as a task an RL agent learns to solve. The nested structure—RL training an agent that writes RL training jobs—creates a feedback loop where success in the inner loop (the small model improving) directly reinforces the outer agent's decisions. The choice to hold out one task family entirely as a generalization probe is methodologically sound, testing whether the trainer learned general principles rather than task-specific patterns. The reward climbing from approximately 0.0 to approximately 0.63 suggests the agent learned meaningful behavior, though the article does not specify what downstream metrics (accuracy, loss, or other domain-specific measures) the hidden evaluation measured.