# A curriculum for post-training agents # 智能体后训练全景学习路线 Post-training refers to the training stages that follow pretraining, such as supervised fine-tuning (SFT), distillation, and reinforcement learning (RL). These stages turn a pretrained base model into one that follows instructions, uses tools, or solves specific domain tasks. While standard post-training typically focuses on single-turn question-answering or chat completions, an agent operates over multi-turn horizons: it plans, invokes external tools, observes execution feedback, and iteratively decides subsequent actions. Agentic post-training adapts these stages specifically for agent systems, ensuring the training objective optimizes across the full sequence of interactive turns rather than isolated outputs. 后训练（Post-training）是指大语言模型在海量语料预训练完成后进行的一系列优化阶段，主要包括监督微调（SFT）、知识蒸馏（Distillation）与强化学习（RL）。这些阶段的目标是让基座模型学会遵循人类指令、调用工具以及解决特定任务。与传统只针对单轮问答或文本对话的后训练不同，智能体（Agent）需要在多轮交互中持续运转——它必须主动规划、调用外部工具、解析环境反馈并决定下一步动作。因此，智能体后训练（Agentic post-training）的核心在于针对完整的多轮交互轨迹与长链路决策进行端到端优化，而不仅是打磨单次回复。 This curriculum is organized into five progressive steps following an introductory setup. Each step outlines the core training methodology, accompanied by architectural diagrams, video tutorials from Hugging Face's *Training Agents* series (using coding agents as a recurring benchmark), technical write-ups, and documentation based on the Hugging Face TRL (Transformer Reinforcement Learning) ecosystem. 本路线图除基础概念准备外，共分为五个循序渐进的阶段。每个阶段均涵盖核心方法解析、训练流程图解、配套视频教程（来自 Hugging Face 的《Training Agents》系列直播，并以编程智能体作为贯穿始终的实战案例）、深度技术解读以及基于 Hugging Face TRL（Transformer Reinforcement Learning）库的实践文档。 --- ## Steps at a glance ## 步骤概览 | Step / 阶段 | The model learns from / 学习来源 | TRL trainer / 对应训练器 | What you need / 准备材料 | | :--- | :--- | :--- | :--- | | **1. SFT (监督微调)** | Agent traces from a stronger model
高能力模型的智能体交互轨迹 | `SFTTrainer` | A dataset of agent traces
多轮工具调用与解决任务的轨迹数据集 | | **2. Distillation (知识蒸馏)** | A teacher's tokens or probabilities
教师模型的输出标记或概率分布（Logits） | `DistillationTrainer` | Prompts and a teacher model
任务提示词与高阶教师模型 | | **3. RL (强化学习)** | A score for each answer
对每次生成答案的标量奖励评分 | `GRPOTrainer` | Prompts and a reward function
任务提示词与可自动判分的奖励函数 | | **4. RL environments (环境强化学习)** | The outcome of acting in an environment
与动态交互环境多轮交互后的最终结果 | `GRPOTrainer` | Tasks and an environment (OpenEnv)
任务集与支持标准交互协议的环境（如 OpenEnv） | | **5. Harnesses and scale (工程扩展与框架集成)** | Full runs of a real agent in sandbox runtimes
智能体在真实沙箱与工程框架中的全流程执行反馈 | `AsyncGRPOTrainer` | Tasks, an environment, and an agent harness
复杂任务集、容器化执行沙箱与智能体调度运行框架 | --- ## 0. Before you start ## 0. 准备工作与核心概念 Before diving into training pipelines, it is crucial to establish a shared vocabulary around modern agentic architectures. In the LLM ecosystem, an **agent** consists of a core model paired with cognitive controls; a **scaffold** defines the code loop, prompt orchestration, and execution logic that coordinates tool use; and a **harness** represents the external testing or execution harness that hosts the sandbox and measures end-to-end task performance. 在正式开始训练之前，必须厘清智能体领域的核心工程概念。在现代大模型体系中，**智能体（Agent）** 是指由语言模型结合自主决策循环构建的系统；**脚手架（Scaffold）** 指驱动智能体运行的代码框架、提示词模板编排及工具调配逻辑；而**测试/运行框架（Harness）** 则是托管外部沙箱、注入测试用例并评测智能体端到端表现的基准运行环境。 * **Read / 深度阅读**: [Harness, Scaffold, and the AI Agent Terms Worth Getting Right](https://huggingface.co/blog/agent-glossary)（《厘清 Harness、Scaffold 等核心 AI 智能体术语》） * **Overview deck / 概览演示**: [Train the agent, not just the model](https://huggingface.co/spaces/sergiopaniego/train-the-agent-slides)（包含 20 张系统幻灯片，覆盖全套训练步骤） * **Prerequisite / 基础储备**: If fine-tuning is new to you, explore the [Hugging Face LLM Course](https://huggingface.co/learn/llm-course)（若缺乏微调经验，建议先研读此大模型公开课）。 --- ## 1. Supervised fine-tuning (SFT) ## 1. 监督微调（SFT） The model learns to emulate high-quality trajectories (agent traces) comprising every step of reasoning, tool call invocation, arguments, and environment observation. These demonstrations are typically gathered from expert human coders or frontier models such as GPT-4. SFT provides a compute-efficient, highly stable starting point that rapidly aligns base models with the expected tool-calling syntax and procedural structure. 在监督微调阶段，模型通过模仿“智能体交互轨迹”（Agent Traces）来学习行为规范。一条完整的轨迹记录了智能体在解决问题过程中的思考链、发起的工具调用、具体参数以及环境返回的结果。这些高质量数据通常采集自人工专家或更强大的前沿闭源模型。SFT 是智能体训练中最经济、最稳定的冷启动方式，能让模型迅速掌握工具调用的特定格式与基本步骤。 However, SFT inherently suffers from distribution shift and exposure bias: because the model is trained exclusively on pristine, successful trajectories, it never encounters its own intermediate failures during training. Consequently, when deployed in dynamic settings, an early error often leads to compounding mistakes, as the model lacks error-recovery behaviors. On-policy distillation and reinforcement learning are explicitly designed to overcome this bottleneck. 然而，SFT 存在固有的“分布偏移”与“暴露偏差”缺陷：模型在训练中只见过完美的成功轨迹，从未面对自己犯错的局面。一旦在实际运行中某个中间步骤调用失误，模型往往缺乏自我纠错和状态恢复的能力，极易导致后续步骤全盘崩溃。后续的同策略（On-policy）蒸馏与强化学习正是为了解决这一痛点。 ![SFT](https://huggingface.co/buckets/merve/post-training-curriculum/resolve/1_sft.png?download=true) * **Watch / 视频教程**: [Training Agents 1: fine-tune a coding agent](https://www.youtube.com/watch?v=rNgUoH7Wbv8)（附带 [代码工程](https://huggingface.co/burtenshaw/gemma4-pi-mono-youtube-livestream-1-scripts)） * **Read / 拓展阅读**: [Training Agents: Class 1, SFT from scratch](https://x.com/ben_burtenshaw/status/2067615361428545566) * **Docs / 官方文档**: [TRL SFT Trainer 文档](https://huggingface.co/docs/trl/sft_trainer) --- ## 2. Distillation ## 2. 知识蒸馏（Distillation） In distillation, a high-capacity teacher model supervises the student model. This can occur either by having the student train directly on teacher-generated outputs or by aligning output probability distributions (token-level logits or KL divergence) against the student's self-generated generations (on-policy distillation). The teacher can be a significantly larger general model, a mixture of specialized domain experts (as demonstrated in Multi-Teacher On-Policy Distillation / MOPD), or even the student model itself augmented with privileged test-time compute, scratchpads, or ground-truth context (self-distillation). 在知识蒸馏阶段，学生模型由更强大的教师模型提供监督信号。这既可以通过让学生模型学习教师生成的优质数据，也可以通过对齐两者的预测概率分布（通过计算 Token 级别的 Logits 差异或 KL 散度）来进行同策略蒸馏（On-policy Distillation）。教师角色既可以是一个超大规模通用模型，也可以是多个垂直领域的专家组合（如多教师同策略蒸馏 MOPD），甚至可以是同一个模型在获得特权提示词、额外思考草稿或环境真值辅助下的自我蒸馏（Self-distillation）。 Because distillation provides token-by-token dense supervision, it exhibits significantly higher sample efficiency than reinforcement learning. The core trade-off lies in upper-bound limitations: the student is inherently bounded by the capabilities and biases of the teacher. Furthermore, on-policy distillation incurs substantial computational costs, as the teacher must execute forward passes synchronously at every training step. 相较于纯强化学习，蒸馏的监督信号是细粒度的逐 Token 损失，因此样本利用效率极高，所需训练数据量更少。但其局限性同样明显：学生模型的性能上限被锁死在教师模型的认知范围内；同时，同策略在线蒸馏需要在每个训练步中实时运行教师模型做前向推理，计算开销十分昂贵。 ![Distillation](https://huggingface.co/buckets/merve/post-training-curriculum/resolve/2_distillation.png?download=true) * **Watch / 视频教程**: [Training Agents 2: distillation](https://www.youtube.com/watch?v=GZT_oNQnLV4)（配套 [资源库](https://huggingface.co/collections/sergiopaniego/training-agents-class-2-distillation-6a47a62b9f09867dccf61e1e)） * **Read / 深度阅读**: * [A brief history of distillation in AI](https://huggingface.co/blog/sergiopaniego/brief-history-of-distillation-in-ai)（《AI 知识蒸馏简史》） * [Distillation in 2026 (so far)](https://huggingface.co/blog/sergiopaniego/distillation-2026)（《2026 年模型蒸馏技术前瞻》） * **Docs / 官方文档**: [TRL Distillation Trainer 文档](https://huggingface.co/docs/trl/distillation_trainer) --- ## 3. Reinforcement learning (RL) ## 3. 强化学习（RL） Reinforcement learning removes reliance on an external teacher model, allowing the student model to learn directly from scalar outcome scores evaluated on its own self-generated attempts. When the model outputs a response, a verifier or programmatic reward function grades it; policy gradient updates then increase the probability of generating traits associated with high-reward responses. 强化学习摆脱了对教师模型的依赖，让模型直接从自己探索出的行为结果中学习。模型针对任务生成回答，由验证器或确定性的奖励函数进行打分，随后的策略梯度优化将提升高分输出在未来被采样的概率。 A prevalent algorithm in modern agent training is Group Relative Policy Optimization (GRPO). Instead of training a resource-intensive secondary critic/value model (as required by traditional PPO), GRPO samples a cohort of candidate completions for each input prompt, calculates their rewards, and normalizes advantages relative to the group mean. Completions outperforming the intra-group baseline receive positive policy reinforcement. 当前智能体训练中最主流的算法之一是分组相对策略优化（GRPO）。与传统 PPO 需要额外维护和训练一个庞大的价值评估模型（Critic）不同，GRPO 会针对每个输入 Prompt 采样生成一组候选输出，计算全组得分后以组内均值作为基准进行归一化。这种组内相对比较机制能够以极低的工程复杂度精准引导策略向优秀样本对齐。 While standard RL provides objective feedback without teacher-induced capacity ceilings, scalar rewards produce a sparse training signal (a single scalar per completion), requiring extensive exploration steps to converge. 尽管基础 RL 无需教师模型即可实现超越基准能力的自我进化，但其反馈信号相对稀疏（通常一个长回答只对应一个标量分数），因此需要大量的采样探索才能稳定收敛。 ![RL](https://huggingface.co/buckets/merve/post-training-curriculum/resolve/3_rl.png?download=true) * **Watch / 视频教程**: [Training Agents 3: reinforcement learning](https://www.youtube.com/watch?v=ztdTed5egrM)（配套 [资源库](https://huggingface.co/collections/sergiopaniego/training-agents-class-3-reinforcement-learning-6ad6aa71050342629dd0e3ec)） * **Read / 深度阅读**: [How frontier models train on outcomes in 2026](https://huggingface.co/blog/sergiopaniego/agentic-rl-2026)（《前沿大模型如何通过结果奖励进行后训练》） * **Projects / 实战案例**: [Fine-tuning a 350M model for structured outputs in 100 GRPO steps](https://huggingface.co/blog/grpo-with-trl-ifstruct)（《基于 TRL 与 GRPO 在 100 步内微调 350M 小模型生成结构化数据》） * **Docs / 官方文档**: [TRL GRPO Trainer 文档](https://huggingface.co/docs/trl/grpo_trainer) --- ## 4. RL environments ## 4. 交互式环境强化学习 In real-world deployment, an agent does not output a solitary response; it interacts iteratively within an external environment that maintains state, registers actions, and returns intermediate observations across multiple turns. The environment runs external tools (e.g., executing shell scripts, running Python runtimes, interacting with APIs), tracks mutation of states, and assesses task completion (such as running test suites in automated software development). 在真实任务中，智能体绝非给出一段单轮回答便宣告结束，而是必须置身于一个具备状态维护能力的动态环境中持续交互。环境负责承接工具执行（如执行终端命令、运行单元测试、调用 API）、维护状态变更并给出即时反馈。最终由环境对整个交互结果进行客观裁定（例如自动化测试是否全数通过）。 Integrating interactive environments directly into the training loop transforms each rollout from a brief token generation process into an extended, multi-step episode. Consequently, rollout generation latency becomes the primary runtime bottleneck of the post-training pipeline. 当引入真正的交互环境后，智能体的每次采样（Rollout）都变成了跨越多个步骤的长周期过程。这使得环境交互与生成轨迹的耗时显著增加，成为整个训练链路中最主要的吞吐量瓶颈。 ![Environments](https://huggingface.co/buckets/merve/post-training-curriculum/resolve/4_environments.png?download=true) * **Watch / 视频与工坊**: * [Training Agents 4: from reward functions to environments](https://www.youtube.com/watch?v=nJV3yUuz6DU)（配套 [资源库](https://huggingface.co/collections/sergiopaniego/training-agents-class-4-rl-environments-6aa27543918888a0f2b5312c)） * [RL for Agents Workshop](https://www.youtube.com/watch?v=cixmqTsi2A4)（智能体强化学习专题工作坊） * **Read / 深度阅读**: [Why RL environments work now (and could not in 2016)](https://huggingface.co/blog/sergiopaniego/rl-environments-story)（《为什么强化学习环境在当下能成功，而在 2016 年却举步维艰》） * **Projects / 实战案例**: * [Paint watercolours with code](https://huggingface.co/blog/train-to-paint-with-code)（通过代码绘制水彩画） * [Autonomous driving with CARLA](https://huggingface.co/blog/sergiopaniego/bringing-carla-to-openenv-trl)（基于 CARLA 模拟器的自动驾驶智能体训练） * **Docs / 官方文档**: * [TRL + OpenEnv 集成指南](https://huggingface.co/docs/trl/openenv) * [OpenEnv 官方文档](https://huggingface.co/docs/openenv) --- ## 5. Harnesses and scale ## 5. 工程框架与大规模训练 At scale, an episode represents an exhaustive execution loop orchestrated through production-grade agent harnesses (such as SWE-bench evaluation runtimes) operating inside isolated remote sandboxes (e.g., Docker or microVMs). Aligning the post-training loop with the actual production harness ensures the policy is trained under the exact prompting protocols, tool schemas, and timeout dynamics it will encounter in production. 在规模化工程场景下，每一次交互轨迹都是智能体在隔离的沙箱环境（如 Docker 容器或轻量级虚拟机）中、借助标准化评测与运行框架（如 SWE-bench 评测体系）进行的完整实战。直接使用生产环境的 Harness 驱动后训练，能确保模型直接适应实际部署时的提示词约束、工具协议以及网络或执行超时等复杂工况。 To optimize hardware utilization, modern setups decouple generation from training across isolated GPU worker pools. Decoupled inference engines (such as vLLM) handle high-concurrency rollout sampling asynchronously, while distributed trainers (via `AsyncGRPOTrainer`) continuously compute parameter updates and broadcast refreshed model weights back to inference workers, ensuring training GPUs never stall waiting for long environment episodes. 由于长链路环境交互极易导致训练节点处于空闲等待状态，前沿系统普遍采用“生成与训练解耦”的分布式异步架构。轨迹采样由基于 vLLM 等高并发推理后端支撑的生成集群负责，参数更新则由专门的训练集群（如 `AsyncGRPOTrainer`）全速执行，训练器随后将最新权重实时同步回推理集群，从而彻底消解多轮智能体训练的吞吐瓶颈。 ![Scale](https://huggingface.co/buckets/merve/post-training-curriculum/resolve/5_scale.png?download=true) * **Slides / 演示幻灯片**: [Training a coding agent through a harness you did not write](https://huggingface.co/spaces/FineEnvs/multi-harness-rl-slides)（《如何借助第三方标准框架训练编程智能体》） * **Read / 深度阅读**: * [The ultimate guide to multi-harness RL](https://huggingface.co/spaces/FineEnvs/multi-harness-rl)（《多框架强化学习终极指南》） * [One sandbox per rollout](https://huggingface.co/blog/sergiopaniego/rl-environments-2026)（《单轨迹单沙箱架构实践》） * [Keep the Tokens Flowing: lessons from 16 open-source RL libraries](https://huggingface.co/blog/async-rl-training-landscape)（《保持 Token 吞吐：解析 16 个开源强化学习库的异步架构演进》） * **Docs / 官方文档**: * [TRL Async GRPO Trainer 文档](https://huggingface.co/docs/trl/async_grpo_trainer) * [OpenEnv Harnesses 适配教程](https://huggingface.co/docs/openenv/tutorials/harnesses)

JetSpec++：通过预测式流水线与自我修正加速大模型推理

从零手写 Agent」一套很适合系统入门 Agent 的中文教程

把智能体真正接到设备上，这个开源项目想做的是“可用的 agent 基础设施”

智能体后训练全景学习路线

智能经济学：通用人工智能（AGI）何时以及如何引爆经济转型？

tiny-llm ，一个从零实现大语言模型推理系统的实战课程

阿里开源的Agent教程 ： AI Agent Handbook

Kimi的官文：如何用 Kimi 做出更好的 3D 作品？

Claude Code 团队如何进行代码审查

在安卓手机上跑 DeepSeek Harness （非官方）

如何从零手写一个Agent？

一套面向团队的 AI 编程 Agent 协作平台

在 Qwen 3.5 内部运行一颗功能完备、确定性的 RISC-V CPU

Impeccable —— 面向 AI 编程智能体的前端设计词汇表与规则检测器

如何把基础大模型训练成会用工具、能自己改错的 AI 智能体

AI 芯片架构深度解析

市面上主流的 AI 浏览器自动化方案

node-llm，一个使用 Node.js + TypeScript 从零构建 GPT 类大语言模型的开源教学项目

工业界怎么训练 Agent：环境、评测与奖励比模型本身更难

渴望胜过资历：给初级工程师的求职与成长建议