专家前缀要能回滚：城大 MOER，六数学均值 49.1
专家前缀要能回滚：城大 MOER，六数学均值 49.1
View PDF HTML (experimental) Abstract: Mix-policy optimization incorporates off-policy expert data into the reinforcement learning (RL) loop to transcend the capability boundary of on-policy RL. However, existing methods either provide static full-trace guidance that...
MOER 混合策略优化框架 (arXiv:2610.30873)
“View PDF HTML (experimental) Abstract: Mix-policy optimization incorporates off-policy expert data into the reinforcement learning (RL) loop”
Abstract:Mix-policy optimization incorporates off-policy expert data into the reinforcement learning (RL) loop to transcend the capability boundary of on-policy RL. However, existing methods either provide static full-trace guidance that suppresses self-exploration, or adopt expert prefix guidance without correctness guarantees, leading to suboptimal performance or even training collapse. In this paper, we propose MOER, a mix-policy optimization framework that harmonizes expert guidance with self-exploration. MOER injects a decaying expert chain-of-thought prefix into the prompt and lets the model complete the remaining reasoning, with the prefix length progressively annealed to encourage self-exploration. A selective rollback mechanism replaces failed completions with the original full expert trace, guaranteeing the correctness of every off-policy trajectory. Furthermore, a harmonized mix-policy objective optimizes on-policy tokens with a GRPO loss while supervising both expert prefix and completion tokens with a trust-region SFT loss, substantially stabilizing training. Extensive experiments show that MOER achieves the highest average scores in mathematical and general domain reasoning tasks, outperforming all basic and mix-policy baselines, and generalizes consistently across diverse tasks, model scales, and base model categories. Codes are available at this https URL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.30873 [cs.LG] |
| (or arXiv:2610.30873v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30873 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xuyang Zhong [view email]
[v1]
Thu, 22 Oct 2026 07:54:45 UTC (477 KB)
Source: Agent Control Flash News · arxiv.org
Related Products & Analysis
Cursor Agent 架构全景拆解：从影子工作区到多文件推演引擎
全面剖析 Cursor 在 Agent 模式下的底层技术实现，解析影子分支隔离、LSP 上下文剪枝以及预测性补全的核心架构细节与工程实践经验。
Vercel v0 推出全新多 Agent 协作画布与实时全栈部署预览
动态速评：v0 正在从单一组件生成器演进为全栈应用搭建协作台，尤其是与 Neon/Postgres 的直连极大降低了全栈原型验证成本。