Apple researchers propose PORTool, a policy-optimization algorithm that improves LLM tool-use agents by assigning step-level rewards from outcome-only supervision.

PORTool, developed by researchers at Purdue University and Apple, generates a rewarded rollout tree where trajectories share prefixes before branching, enabling direct comparisons among alternative tool-use decisions within the same context.

The algorithm estimates each step's importance using a correctness-dominant signal—whether descendants of that step can ultimately produce a correct final answer—plus an auxiliary term indicating whether the step's tool calls execute successfully.

Experiments show PORTool improves final-answer accuracy while reducing tool-call steps compared with state-of-the-art baselines, with ablation studies confirming the robustness of the proposed step-wise importance estimates.

The paper was accepted at the Fifth Workshop on Natural Language Generation, Evaluation, and Metrics at ACL 2026.