ACE-RL Trains One Agent in Three Roles, Beats PPO and GRPO

KAIST AI researchers published ACE-RL on November 13, 2026, introducing a reinforcement learning recipe that co-trains one language model to serve as actor, insight reflector, and playbook curator on top of a fixed Agentic Context Engineering scaffold. The method grades candidate experience artifacts by their downstream effect on future rollouts and outperforms PPO, GRPO, ERL, and end-to-end PPO baselines on held-out agentic benchmark tasks.
Doyoung Kim, Byeongguk Jeon, Kailin Zhang, and Sherry Yang report that gains persist well beyond the training cutoff: a model trained with three attempts per task continues to outperform a GRPO-trained baseline allowed up to eight attempts.
Release date and methodology
Officially released on November 13, 2026, ACE-RL fixes the Agentic Context Engineering harness in place and optimizes role-specific behaviors rather than tuning the scaffold itself. The paper states the system "co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold" and achieves higher held-out performance than all tested baselines.
The framing is direct: agent training should optimize not only the actions in the current rollout but also the model-written experience that future rollouts inherit. This approach treats the playbook as a living artifact that shapes subsequent task attempts rather than a static prompt.
Benchmark performance and persistent gains
ACE-RL beats PPO, GRPO, ERL, and end-to-end PPO on held-out reward metrics across the team's agentic benchmark suite. The reported advantage holds even when training budgets differ: a model trained with three attempts per task under ACE-RL continues to outperform a GRPO-trained agent given up to eight attempts per task.
These results suggest the method's gains come from structural coordination between the three roles rather than raw compute or sample volume. The paper positions the work as a contribution to reinforcement learning recipe research, a domain that has seen steady activity on fine-tuning trackers over the last 90 days.
Ablation results confirm all three roles matter
Ablations in the paper reveal that training only the actor head is insufficient to match full ACE-RL performance. The reflector and curator roles are essential: removing either component degrades held-out reward. This finding underscores the hypothesis that coordinating action generation, insight extraction, and playbook maintenance in a single model yields emergent advantages over narrower training objectives.
- Actor generates task actions in the current rollout
- Reflector extracts insights from rollout outcomes
- Curator edits the playbook artifact based on reflector insights
The fixed scaffold ensures consistency across training runs, letting the RL objective focus on role-specific policy improvements rather than scaffold hyperparameters.
What this means
ACE-RL demonstrates that a single language model can internalize multiple agentic roles without sacrificing performance, offering a simpler architecture than multi-model pipelines. The persistent gains beyond training cutoffs and the necessity of all three roles suggest that future agent systems may benefit from co-training action, reflection, and curation pathways rather than treating them as separate modules. For teams building production agents, the work provides a concrete alternative to standard PPO and GRPO recipes when task performance depends on accumulating and refining experience over repeated attempts.
on Emergent today





