Amazon Nova Forge adds custom reward design for multi-turn AI training

Amazon Nova Forge is Amazon's model customization platform, and multi-turn reinforcement learning allows training on agentic tasks where a model acts over several steps—for example, asking questions, calling tools, executing code, or recovering from mistakes. The core technical challenge is designing a reward function (the scoring mechanism) that guides the model toward the right behavior without teaching it the wrong thing by accident.

For multi-turn tasks, Nova Forge offers two paths: a serverless multi-turn RL option (now generally available) for teams that do not want to manage infrastructure, and Bring Your Own Orchestration (BYOO), where teams run the reward logic inside their own environment container. In the BYOO approach, Nova Forge delegates each rollout (a full multi-turn episode or trajectory) to the customer's container, which manages user simulation, code execution, and verification, then returns an aggregate reward score plus optional per-component scores. Training then uses GRPO to rank K model completions within a batch and adjust weights to make higher-reward completions more likely.

The post uses a worked example: training Amazon Nova Lite 2.0 on a collaborative-coding task spanning 500 unique programming tasks. The model receives a brief, under-specified coding request; a user simulator holds the full specification and reveals details only when the model asks a clarifying question. Each turn, the model either asks a question or commits code. If it commits, the code is executed against hidden unit tests to score correctness. The design intent is that guessing produces wrong code, while asking surfaces hidden details and leads to correct code.

The reward is a weighted sum of four components: correctness (1.0 weight: fraction of unit tests passing), asked_before_coding (0.6 weight: 1.0 if asked on turn 1 then committed, 0.6 if asked later then committed, else 0), guessed_immediately (0.4 weight penalty: -1.0 if the first turn is code with no question), and loop_penalty (0.2 weight: -0.5 if the last two turns are more than 80% similar). Two design principles guide this: un-gate the behavior you want (asked_before_coding is credited independently, not conditional on correctness, but does require eventual code commitment to close the "ask forever" loophole), and penalize the failure mode (guessed_immediately makes guessing strictly worse than asking, restoring variation within a GRPO batch). The post demonstrates how to implement these components as scorers inside the environment container and report each through metrics_list.

A critical insight the post surfaces is that a reward component contributes learning only through the variation it creates within a group. If a term takes the same value for every completion, it contributes nothing to the within-group advantage, and therefore nothing to the gradient. The authors encountered a real failure case where the highest-weighted component silently contributed no learning signal at all—a pitfall they explain how to catch by instrumenting each component so teams can trust what training is learning. The infrastructure (Amazon SageMaker HyperPod, an Amazon ECS customer-managed environment, and an S3 bucket for rollout data and checkpoints) and illustrative code are detailed in Part 1 of the series and the aws-samples/sample-nova-multi-turn-rl-infra repository.