New method enables AI systems to learn from multiple human preferences while balancing performance against safety and fairness constraints

Researchers developed an offline constrained reinforcement learning approach that uses multiple preference oracles to optimize AI behavior based on human feedback

The method ensures protected groups maintain minimum welfare levels while maximizing overall utility, addressing real-world tradeoffs between performance and fairness

Uses maximum likelihood estimation for oracle-specific rewards and formulates the problem as a KL-regularized Lagrangian with a Gibbs policy solution

Provides first finite-sample performance guarantees for offline constrained preference learning with a dual-only algorithm ensuring high-probability constraint satisfaction

Framework extends to handle multiple constraints and general f-divergence regularization, making it adaptable to diverse AI safety applications