CoFi-PGMA framework corrects learning signals for multi-agent LLM systems using counterfactual policy gradients

Researchers introduced CoFi-PGMA, a framework for training multiple language models that either compete through routing mechanisms or collaborate to produce answers, addressing the problem that standard reinforcement learning objectives become misspecified when learning signals are filtered by system mechanisms.

The approach derives per-agent training objectives based on marginal contribution: for routing systems it applies off-policy corrections for selection-gated feedback, while for collaborative systems it uses leave-one-out difference rewards for credit assignment (a method that measures each agent's individual contribution by comparing performance with and without that agent).

The framework analyzes how softmax routing induces risk-sensitive incentives and provides practical training algorithms that integrate counterfactual estimators, multiturn-aware rewards, and policy optimization methods, with demonstration on a real-world reasoning dataset.