EMPO模型：通过效果-边缘策略优化提升LLM代理性能
【CN】同一可执行效果、多段文本在抢信用：天大 Jianye Hao 等 EMPO，ALFWorld 69.5→83.3%
反直觉：多段文本可诱导同一可执行效果，但信用仍绑在各自文本 realization 上（credit–update misalignment）。现有错处：只改细粒度信用、不改更新耦合。作者改动：EMPO——效果层替代目标，无需额外 critic/rollout。硬数字：EMPO-G（Qwen2.5-1.5B）ALFWorld **69.5→83.3%**、WebShop **62.8→70.3%**；EMPO-GiG ALFWorld **90.1→94.0%**；Sokoban VL-3B **72.7→84.4%**。可迁移判断：**训练线专用。不要并进昨版 Self-Credit。**
EMPO模型 (arXiv:2610.29427)
“View PDF HTML (experimental) Abstract: Recent advances in multi-turn reinforcement learning (RL) have substantially improved the performance”
Abstract:Recent advances in multi-turn reinforcement learning (RL) have substantially improved the performance of LLM agents on complex, long-horizon interactive tasks. However, despite progress in fine-grained credit assignment, policy optimization still applies these learning signals through response-specific probability ratios. Different textual responses can induce the same executable effect, yet credit differences arising from subsequent outcomes remain tied to their individual textual realizations. We identify this mismatch as credit-update misalignment. Our analysis reveals that credit-text pairing directly shapes policy-update directions, even when the policy context and executable effect are held fixed. Motivated by this finding, we propose Effect-Marginal Policy Optimization (EMPO), which aligns policy optimization with execution-level equivalence through an effect-level surrogate that couples updates across alternative textual realizations. EMPO preserves the original credit signals, integrates with both trajectory-level and step-level credit estimators, and requires no additional critics or rollouts. Experiments on ALFWorld, WebShop, and Sokoban show that EMPO outperforms strong baselines in both task performance and learning efficiency.
| Comments: | 20 pages, 11 figures |
| Subjects: | Machine Learning (cs.LG); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.29427 [cs.LG] |
| (or arXiv:2610.29427v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29427 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shaojin Ma [view email]
[v1]
Wed, 21 Oct 2026 08:51:14 UTC (1,105 KB)
来源：Agent Control Flash News · arxiv.org
关联产品与深度拆解
Cursor Agent 架构全景拆解：从影子工作区到多文件推演引擎
全面剖析 Cursor 在 Agent 模式下的底层技术实现，解析影子分支隔离、LSP 上下文剪枝以及预测性补全的核心架构细节与工程实践经验。
Vercel v0 推出全新多 Agent 协作画布与实时全栈部署预览
动态速评：v0 正在从单一组件生成器演进为全栈应用搭建协作台，尤其是与 Neon/Postgres 的直连极大降低了全栈原型验证成本。