# Introducing *FrontierHarness* Eval # 推出 FrontierHarness 评测基准 **Date / 日期:** September 01, 2026 / 2026 年 9 月 1 日 **Authors / 作者:** Shilin Zhu, Shiqi Mei **Tags / 标签:** `agents` `announcement` `benchmark` `harness` `evaluation` / `智能体` `官方发布` `基准测试` `Agent框架` `性能评估` --- ## TL;DR / 核心速览 Run the same tasks through different harnesses and you get very different bills, pass rates, and wall-clock times. *FrontierHarness v1.0* covers software development and terminal-based tasks. *(Background: In modern AI agent systems, a **Harness** (execution framework) handles prompt orchestration, tool execution loops, context truncation, and environment communication around the base foundation model. Even when powered by the exact same underlying LLM, different harnesses can drastically alter runtime cost, execution time, and task success rates.)* 在运行相同任务时，使用不同的 Agent 框架（Harness）会导致账单费用、任务通过率和实际耗时产生巨大差异。*FrontierHarness v1.0* 专注于评估软件开发与终端交互场景。*(背景补充：在 AI 智能体体系中，**Harness（执行框架）** 是指包裹在底层大模型外部的工具调度、Prompt 组装、上下文管理与环境交互层。即使底层使用完全相同的基础模型，不同的 Harness 实现也会极大程度影响任务的最终开销、耗时和成功率。)* Claude Code and DSH (DeepSeek Harness) Creator both landed at a 63% pass rate. Claude Code paid 5.6x more per pass to get there. Claude Code 与 DSH（DeepSeek Harness）Creator 配置均取得了 63% 的任务通过率，但 Claude Code 为此支付的单次通过成本比 DSH 高出 5.6 倍。 If you want a quick decision, pick based on what matters most to you: 如果你需要快速选型，可以根据最看重的维度进行选择：

Quality Leader / 综合质量之选: Codex

66.7% pass rate · $3.47 per pass / 66.7% 通过率 · 单次通过成本 $3.47

Balanced Pick / 均衡优选: Pi

60.0% pass rate · $2.43 per pass / 60.0% 通过率 · 单次通过成本 $2.43

Cost Leader / 成本之选: Exo Harness

$1.05 per completed task · 53.3% pass rate / 任务完成单价 $1.05 · 53.3% 通过率

Speed Leader / 极速之选: DSH Minimal

5m 41s median runtime · 56.7% pass rate / 运行耗时中位数 5分41秒 · 56.7% 通过率

Full leaderboard at [frontierharness.org](https://frontierharness.org/). Explore the [source data and task definitions on GitHub](https://github.com/frontier-harness-eval/eval). 完整排行榜可访问 [frontierharness.org](https://frontierharness.org/)。原始数据与任务定义已开源于 [GitHub 仓库](https://github.com/frontier-harness-eval/eval)。 --- ## Results / 评测结果 ### Pareto Frontier / 帕累托前沿分布

Across all 360 cells: 209 successes, 151 failures. Field-wide pass rate is 58.1%; field-wide token-weighted cache hit rate is 92.4%. 在全部 360 次测试运行中：成功 209 次，失败 151 次。全场平均通过率为 58.1%，全场 Token 加权缓存命中率为 92.4%。 The pass rates for all 12 configurations are close, ranging from 50.0% to 66.7%, just a 17-point difference. But the cost per completed task varies much more, from $1.05 to $18.34 on the same tasks. That means two harnesses can solve the same problem, but one might cost over ten times more than the other. 全部 12 种配置的通过率区间较为接近，分布在 50.0% 到 66.7% 之间，极差仅为 16.7 个百分点。然而，完成相同任务的单次成本差异却极其悬殊（从 $1.05 到 $18.34）。这意味着两个不同的 Harness 虽然能够解决同一个问题，但其中一个的开销可能是另一个的十多倍。 ### Full Leaderboard / 完整评测数据表 | Harness (configuration)
框架（配置） | Median cost per pass
单次通过中位数成本 | Pass rate
通过率 | Cache hit
缓存命中率 (中位数) | Median time
耗时中位数 | | :--- | :--- | :--- | :--- | :--- | | **Codex** | **$3.47** | **66.7%** | 88.0% | 6m 43s | | **DSH Creator** | $3.28 | 63.3% | 84.3% | 6m 44s | | **Claude Code** | $18.34 | 63.3% | 67.8% | 9m 38s | | **Pi** | $2.43 | 60.0% | 79.4% | 7m 33s | | **DSH Standard** | $3.46 | 60.0% | 86.5% | 6m 17s | | **DSH PTC** | $4.58 | 60.0% | 87.2% | 7m 44s | | **Kimi Code** | $3.65 | 56.7% | 88.0% | 7m 56s | | **DSH Minimal** | $4.72 | 56.7% | 84.6% | 5m 41s | | **Oh My Pi** | $4.75 | 56.7% | 82.2% | 6m 46s | | **Exo Harness** | **$1.05** | 53.3% | 70.3% | 6m 17s | | **Hermes** | $2.90 | 50.0% | 85.9% | 6m 58s | | **OpenCode** | $3.24 | 50.0% | 78.4% | 6m 27s | Exo Harness and Pi solve tasks for a fraction of what the rest of the field spends. Codex and DSH Creator show that efficiency and quality are not opposed: both sit near the top of the pass-rate column while spending close to the field median. Claude Code marks the other end. It passes as many tasks as DSH Creator and pays 5.6x more per pass to do it. As discussed below, that result may reflect the interaction between the harness, model, and gateway rather than the harness alone. Exo Harness 和 Pi 仅用行业均值的一小部分成本就完成了任务。Codex 与 DSH Creator 则证明了“高效率”与“高质量”并非对立：二者在保持极高通过率的同时，成本维持在中位数水平。Claude Code 则处于另一个极端：虽然通过率与 DSH Creator 持平，但成本高出 5.6 倍。后文将详细分析，该现象更多源自 Harness、模型以及网关缓存机制之间的兼容适配问题，而非框架本身的单纯缺陷。 Three patterns are worth pulling out: 1. **Finishing a task and finishing it cheaply are separate skills.** Claude Code and DSH Creator both pass 19 tasks. One costs $3.28 per pass, the other $18.34. 2. **Cache hit rate is not cost.** A cached 300-turn failure can still burn more than a short cache miss. Steps and tokens are what you pay for; caching only changes the unit price. 3. **Vendor lineage did not produce an advantage here.** Kimi Code, made by the same lab as the model, tied for seventh on pass rate at 56.7% and sat mid-pack on cost. 值得关注的三个核心结论： 1. **“完成任务”与“低成本完成任务”是两种完全不同的工程能力。** Claude Code 与 DSH Creator 均成功解决了 19 个任务，但单次通过成本分别为 $18.34 与 $3.28。 2. **缓存命中率高并不等同于总成本低。** 一个经历了 300 轮交互但最终失败的高缓存命中会话，其开销依然远高于一次轮数极少但缓存未命中的调用。步数和 Token 消耗量是决定总成本的根源，缓存机制仅降低了单位单价。 3. **同厂血缘并未带来天然加成。** 与底层模型出自同一团队的 Kimi Code，最终通过率仅为 56.7%（并列第七），成本也仅处于中游水平。 > **Note / 注释:** **DSH** (DeepSeek Harness) 共评测了四种模式：Standard（标准版）、Minimal（精简版）、Creator（构建者版）与 Code/PTC（程序式工具调用版）。 --- ## Key Highlights / 评测核心特色

Comprehensive harness evaluation / 全方位的 Harness 对比评测

Covering 9 harnesses and 12 configurations focused on SWE and terminal tasks, keeping model, tasks, and runtime strictly constant.
覆盖 9 种主流框架及 12 种配置，在严格控制模型、任务用例和执行环境完全一致的前提下，针对软件工程与终端运维场景展开评测。

Identical cold start on every run / 严苛的完全冷启动隔离

All 360 trials are fresh restores of the same checkpoint down to memory state. Benchmark tasks were never pre-run to prevent prefix cache pollution.
全部 360 次运行均从精确到内存状态的标准快照（Checkpoint）全新拉起。评测前从未预跑任务，彻底杜绝隐式前缀缓存污染。

Neutral evaluation with no home-field advantage / 中立评测，杜绝主场优势

Kimi K3 was selected as the unified backend model, ensuring that neither OpenAI nor Anthropic native frameworks enjoyed built-in advantages.
统一选用中立的 Kimi K3 作为评测模型，避免测试框架因依赖自家原生模型而产生不公平的主场红利。

--- ## Why We Built FrontierHarness / 为什么构建 FrontierHarness Runta is an agent execution layer designed for token efficiency, secret protection, and governed access. What we care about is how much compute it takes an agent to finish a task it is already capable of finishing, and how much of that is the harness’s doing rather than the model’s. Runta 是一套专为 Token 效率、凭据保护与受控访问而设计的 Agent 执行层。我们最关注的核心问题是：当一个大模型本身已经具备完成某项任务的能力时，到底需要消耗多少计算资源才能达成目标？而这其中又有多少额外开销是由外层的 Harness 架构所引起的，而非模型本身？ Model evaluations establish the baseline capability. The harness supplies the tools and environment, and it determines how much of that capability is spent on the problem versus on overhead. Three questions shaped the evaluation: - **What does a task actually cost?** Tokens and steps, measured directly from each trajectory. This is the number we care most about. - **What does a failure cost?** Agents spend the most money on the problems they cannot solve. A cost metric that only counts successes hides the majority of the bill. - **How efficiently does a harness use tools?** Effective tool calls reduce unnecessary steps, improve accuracy, and save tokens. 模型评测定义了能力的上限基准，而 Harness 则负责提供工具与运行环境，直接决定了模型的能力是有针对性地用在解决问题上，还是被浪费在框架调度开销上。本次评测聚焦于三个核心问题： - **完成一项任务的实际成本是多少？** 直接从每条执行轨迹中精确采集 Token 消耗量与交互步数。 - **任务失败时的试错代价是多少？** Agent 往往在那些无法解决的难题上消耗最多的算力。仅统计成功用例成本的指标会掩盖账单中的真实大头。 - **Harness 的工具调用效率如何？** 简洁精准的工具调度能够大幅削减冗余交互轮次、提升正确率并节省 Token。 --- ## How We Built FrontierHarness / 评测方案的设计与实现 ### Benchmark Scope / 评测基准范围 The benchmark covers 30 tasks (21 Terminal-Bench tasks for command-line execution and 9 DeepSWE tasks for real-world software issue resolution) across 9 harnesses. These run in 12 configurations for 360 total evaluations, with one canonical result for every task and harness-configuration pair. Outcomes use deterministic verifier-based pass/fail scoring, and first-turn cache reads are repriced consistently across harnesses to make costs comparable. 评测共涵盖 30 个典型任务（包括 21 个针对命令行执行的 Terminal-Bench 任务，以及 9 个源于真实开源项目的 DeepSWE 复杂软件工程任务），跨越 9 个主流框架与 12 种运行配置，共计执行 360 次评测。每个“任务-框架配置”对均产生一份标准执行轨迹。测试结果采用确定性验证器（Verifier）进行自动化 Pass/Fail 判定，并对首轮缓存读取进行了统一的费率折算以保证成本口径一致。 ### The Execution Model / 环境执行模型 Every harness and every task environment is installed once and captured as a **golden Runta checkpoint**. Each of the 360 trials is a fresh restore from that checkpoint: identical vCPU, memory, disk size, disk contents, and *memory state*. No trial inherits filesystem drift, a warmed page cache, or a partially-installed dependency from the trial before it. 评测将每个框架与任务环境预先安装并固化为**标准 Runta 检查点（Golden Checkpoint）**。全部 360 次测试均从该镜像检查点完全还原拉起：分配完全相同的 vCPU、内存容量、磁盘空间与文件内容，甚至精确到内存镜像状态。确保没有单次测试会继承上一轮产生的文件系统漂移、热页面缓存或残留依赖。 Execution rules ensuring consistency: - **Benchmark tasks are never run before the formal evaluation.** Implicit prefix caching means a debug run leaves the trajectory warm for hours. All debugging happens on a separate smoke task set. - **All 12 configurations run the same task in the same job**, ensuring environmental parity. - **One attempt per cell.** Retries are warmer by definition, and best-of-N selection would bias quality upward. 确保结果严谨可比的核心规则： - **正式评测前绝不预跑测试任务：** 服务端的隐式前缀缓存（Prefix Caching）会导致预热状态驻留数小时。所有前期调试均在独立的冒烟测试用例集上进行。 - **所有 12 种配置均在同批次作业中执行相同任务：** 确保任务间执行环境的高度一致。 - **单次判定（One Attempt per Cell）：** 避免重试机制带来的缓存红利与结果向上偏差（如 Best-of-N 抽样）。 --- ## Why Kimi K3 / 为什么选用 Kimi K3 作为基础模型 We wanted a model that was not native to one of the leading harnesses in the comparison. Using Claude would favor Claude Code’s native environment; using a GPT model would do the same for Codex. Kimi K3 gave us a more neutral starting point, although it does not remove every compatibility effect. 我们需要一款对参与评测的主流框架均保持中立的基础模型。使用 Claude 系列模型会天然偏向 Claude Code 的原生环境；使用 GPT 系列模型则会对 Codex 产生倾斜。Kimi K3 提供了一个相对客观中立的基准，尽管这无法完全消除所有的协议适配效应。 *(背景补充：**Kimi K3** 具备强大的长上下文理解与代码编写能力，支持隐式 Prompt 缓存。本次评测通过 **Fireworks.ai** 进行统一托管与推理分发，确保所有测试框架均访问同一套模型权重。)* All nine harnesses connect through a single gateway that speaks both the OpenAI Responses API and the Anthropic Messages API, so each harness talks its native protocol to identical weights. 所有 9 款框架均通过统一网关接入，网关同时兼容 OpenAI Responses API 与 Anthropic Messages API 协议规范，使各框架能够以其原生协议直接与相同的模型权重进行通信。 ### Why K3 Gives Kimi Code No Home-Field Advantage / 为什么 Kimi Code 并没有获得主场优势 Kimi Code is built by the same lab as Kimi K3, but it did not have a native-path advantage in this evaluation. Kimi Code’s `kimi` provider type returned a 404 against our gateway, so we configured it as `type = "anthropic"` and routed it over the Messages API. Its stream output also omits usage data, which we recovered from `usage.record` events in the session wire log. 尽管 Kimi Code 与 Kimi K3 出自同一实验室，但在本测试中并未获得通道优势。由于 Kimi Code 的原生 `kimi` Provider 在统一网关中返回 404，测试中将其配置为 `type = "anthropic"` 并通过 Messages API 协议进行请求路由。针对其流式输出中缺失 Usage 数据的问题，我们从会话日志的 `usage.record` 事件中进行了精准还原。 Kimi Code tied for seventh at 56.7%, with a $3.65 median cost per pass and a 7m 56s median runtime. Its 88.0% median cache hit rate tied for the best in the field, but that did not translate into a higher pass rate or lower cost. Kimi Code 最终以 56.7% 的通过率并列第七，单次通过中位数成本为 $3.65，中位数耗时 7 分 56 秒。虽然其 88.0% 的中位数缓存命中率位列榜首，但这并未能转化为更突出的通过率优势或成本削减。 --- ## Finding: Claude Code’s Token-Weighted Cache Hit Rate Is Low ## 核心发现：Claude Code 的 Token 加权缓存命中率偏低 Claude Code’s cache hit rate is **25.0% token-weighted** and **67.8% by median cell**. Every other harness in the field sits between 91.6% and 99.1% weighted. The dashboard plots the median; the total token bill is driven by the weighted number. For Claude Code, the gap between the two is 2.7x. Claude Code 的 **Token 加权缓存命中率仅为 25.0%**，而**按单元格统计的中位数为 67.8%**。相比之下，其他框架的加权缓存命中率普遍在 91.6% 到 99.1% 之间。常规仪表盘展示的是中位数，但实际账单支出是由加权总消耗驱动的。在 Claude Code 上，两者存在 2.7 倍的显著差距。 *(背景补充：**中位数命中率**反映典型简单会话的表现；而**Token 加权命中率**计算的是全部消耗 Token 中享受缓存优惠的实际比例。长会话中缓存一旦失效，单次产生的巨额全量 Token 计费会瞬间拉高账单总额。)* The weighted cache hit rate is heavily skewed: three of Claude Code’s nineteen successful cells account for **88% of its total token usage**, and just one task (`python-statemachine-state-data-scoping`, 24.6M input tokens at a 15.7% hit rate) is responsible for **68%** by itself. In effect, the 25.0% weighted cache rate mostly reflects performance on this single, exceptionally large cell. 这种加权偏差具有极端的聚集性：在 Claude Code 成功通过的 19 个任务中，仅其中 3 个任务就消耗了其 **88% 的总 Token 数量**。单单 `python-statemachine-state-data-scoping` 这一个任务（消耗了 24.6M 输入 Token，缓存命中率仅 15.7%）就占据了其总消耗的 **68%**。因此，25.0% 的加权命中率极大程度上反映了框架在极长交互场景下的性能衰退。 ### Why Is the Weighted Rate Low? / 为什么加权命中率会如此偏低？ Claude Code is designed around Anthropic models and their explicit cache-control semantics (`cache_control: {"type": "ephemeral"}`). K3 exposes an implicit prefix cache. The trajectories show that caching was inconsistent from call to call: Claude Code 原生基于 Anthropic 模型的显式缓存语法（如 `cache_control: {"type": "ephemeral"}`）进行设计，而 K3 / Fireworks 采用的是服务端的隐式前缀缓存机制（Implicit Prefix Caching）。执行日志显示，其请求之间的缓存命中表现出偶发性波动： ```text call fresh cacheRead total_in hit% 1 1174 18432 19606 94.0 4 19381 18432 37813 48.7 5 0 38345 38345 100.0 <-- same conversation, full prefix hit / 同一会话，命中全部前缀 8 23092 18688 41780 44.7 13 0 45277 45277 100.0 25 35474 18944 54418 34.8 26 0 55514 55514 100.0 ``` In an append-only conversation, the cache read stayed near 18.4K tokens (the size of the system prompt) on most calls, while the growing conversation was billed at the fresh rate. On roughly one call in five, the full prefix was hit. This explains why the reported figures reflect the specific harness-model pairing rather than Claude Code in isolation. 在一个纯追加式的对话上下文中，大部分请求仅命中了约 18.4K Token（即系统提示词大小），增长的会话历史均按全新输入计费；而大约每五次调用中只有一次能够完整命中全部前缀。这表明评测数据反映的是当前特定“框架-模型网关”组合下的适配表现，而非 Claude Code 在其原生 Anthropic 生态下的孤立性能。 --- ## Case Study: An Example Task / 深度案例剖析 Task: `datacurve/python-statemachine-state-data-scoping` is a feature request against `fgmacedo/python-statemachine` at a pinned base commit. Official DeepSWE baseline pass rate: 38%. Median historical trajectory: 117 agent steps. Seven of the 12 configurations passed. 案例任务：`datacurve/python-statemachine-state-data-scoping` 是针对 `fgmacedo/python-statemachine` 仓库在特定 Commit 下的一个新功能开发需求。在官方 DeepSWE 基准中历史通过率仅为 38%，历史执行步数中位数为 117 步。在本测试中，12 种配置中有 7 种成功通过。 *(背景补充：该任务要求 Agent 在保持现有单元测试全部通过的前提下，深入重构 Python 元类、回调注入机制、历史状态恢复、序列化 Pickle、SCXML 解析器以及状态机图形渲染器，属于高度复杂的全栈级工程代码修改。)* ### Detailed Execution Matrix / 任务执行详情表 | Harness
框架 | Result
执行结果 | Cost
总成本 | Turns
交互轮数 | Cache
缓存命中率 | Time
总耗时 | | :--- | :--- | :--- | :--- | :--- | :--- | | **Pi** | **Pass (通过)** | **$2.50** | **90** | **98.2%** | 39m | | **Codex** | Pass (通过) | $5.97 | 187 | 99.1% | 37m | | **Hermes** | Pass (通过) | $6.70 | 156 | 99.1% | 29m | | **Kimi Code** | Pass (通过) | $9.24 | 209 | 99.3% | 38m | | **DSH Standard** | Pass (通过) | $10.14 | 227 | 99.3% | 47m | | **DSH Minimal** | Pass (通过) | $10.49 | 259 | 99.1% | 44m | | **Claude Code** | **Pass (通过)** | **$64.36** | **381** | **15.7%** | 60m | | **Exo Harness** | Fail (步数超限) | $1.46 | 51 *(Cap)* | 96.0% | 40m | | **Oh My Pi** | Fail (未通过) | $2.71 | 86 | 98.3% | 42m | | **DSH Creator** | Fail (超时) | $3.88 | 109 | 98.8% | 60m | | **DSH PTC** | Fail (超时) | $12.95 | 285 | 99.5% | 60m | | **OpenCode** | Fail (超时) | n/a | 0 | n/a | 60m | This task is the thesis in miniature. Pi and Claude Code both passed, with a **26x difference in price** ($2.50 vs $64.36). Pi reached the fix in 90 turns; Claude Code took 381, with a 15.7% cache hit rate against Pi’s 98.2%. More steps, each one more expensive. 这个案例生动印证了本研究的核心论点。Pi 与 Claude Code 最终都成功修复了代码并通过了测试，但**调用成本相差了 26 倍**（$2.50 对比 $64.36）。Pi 仅用 90 轮对话就完成了修复；而 Claude Code 消耗了 381 轮，且缓存命中率仅为 15.7%（Pi 为 98.2%）。交互轮数更多，且每次轮次的单价更贵，导致成本呈现级数级放大。 DSH PTC spent $12.95 on a failed run—more than six of the seven configurations that passed. DSH Creator failed at $3.88 while DSH Minimal passed at $10.49. Both used the same harness, model, and runtime, but different presets. 此外，DSH PTC 在一次失败的运行中花费了 $12.95，比 7 个成功通过的配置中的 6 个还要昂贵。DSH Creator 失败仅花费 $3.88，而 DSH Minimal 成功通过花费了 $10.49。两者采用相同的框架内核、模型与运行环境，仅仅预设配置（Preset）不同，便直接改变了执行节奏与最终结果。 --- ## Limitations & Future Work / 局限性与未来规划 ### Limitations / 当前局限性 FrontierHarness v1.0 is currently limited to software development and terminal-based tasks. It does not yet evaluate other types of agentic or knowledge work. Broader coverage is needed before making general conclusions. FrontierHarness v1.0 目前主要覆盖软件工程与终端运维场景，尚未涉及通用信息检索、办公协作等其他类型的知识型 Agent 任务。在得出普适性结论前，仍需扩展测试集覆盖范围。 The benchmark uses each harness strictly "as-shipped" without internal modifications. We did not alter internal caching logic or gateway routing to compensate for compatibility quirks. Reported figures reflect the end-to-end performance of out-of-the-box setups. 评测严格采用各框架开箱即用的默认出厂状态（As-Shipped），未对框架源码或缓存逻辑进行人为打补丁或优化适配。因此数据客观呈现的是各框架在默认状态下的端到端真实表现。 ### Future Work / 未来迭代路线 Version 1.1 will evaluate the full **Harness × Model matrix** across multiple frontier models (Claude, GPT, Kimi, DeepSeek) to isolate model capability from harness overhead, generating an authoritative compatibility matrix. 在随后的 v1.1 版本中，我们将引入多款主流模型（如 Claude、GPT、Kimi、DeepSeek 等），开展全维度的 **Harness × Model 交叉矩阵评测**，以剥离单一模型带来的耦合影响，建立权威的 Agent 框架与模型协同兼容性矩阵。 --- ## Conclusion / 结语 The harness makes a difference. Pass rates in this field cluster within 17 points; median cost per pass spans 17x. - **Codex** is the safe default: highest pass rate (66.7%), cost near the field median ($3.47). - **Pi** is the pick when the same job runs repeatedly and token spend compounds ($2.43 per pass, highly efficient tool looping). - **Exo** is the cheapest per completed task ($1.05), with early-stop mechanics making it ideal for low-cost speculative execution. - **DSH** illustrates how preset tuning alone can cause a 1.4x cost swing and 6.6-point pass rate variance on the exact same model. Harness 框架的选择至关重要。当前主流框架的任务通过率差距在 17 个百分点以内，但单次成功成本的跨度却高达 17 倍： - **Codex** 是综合质量最稳健的默认首选：通过率最高（66.7%），成本接近行业中位数（$3.47）。 - **Pi** 是高频执行场景下的最佳选择：单次通过成本低至 $2.43，具备极其紧凑高效的工具调用与上下文管理能力。 - **Exo Harness** 是极致的成本领跑者：单次完成仅需 $1.05，其主动截断机制非常适合低成本试错与重试流。 - **DSH（DeepSeek Harness）** 证明了提示词与工具预设的重要性：在底层模型和环境完全一致的前提下，仅调整预设配置就能带来 6.6% 的通过率差异与 1.4 倍的成本波动。 The cost spread across the field comes mostly from the harness-model pairing. Cache strategy, tool looping, and prompt design belong to the pair, not to the harness alone. Holding the model, tasks, and runtime constant allowed v1.0 to reveal these dynamics clearly. 整个测试体系中展现出的巨大成本鸿沟，本质上源自“框架与模型”之间的工程契合度。缓存调度策略、工具循环设计与 Prompt 体系是框架与模型共同作用的产物。通过在 v1.0 中严格控制模型、测试用例与执行环境的恒定，我们得以首次清晰解构并量化这些关键工程细节对 Agent 实际落地的深远影响。

评论