Knowledge Retrieval Benchmark

Results at a glance

On 24 real developer questions from a large production full-stack monorepo, Agents answering from Context knowledge fully solved more questions, faster and with far fewer tool calls than Agents using an LLM Wiki. Adding Source Graph for remote code reading cut time and tool calls further.

Metric	LLM Wiki	Context	Context + Source Graph
Answering model	GPT 5.6 SOL	GPT 5.6 SOL	Claude Opus 5.5
Fully solved	19 (79.2%)	21 (87.5%)	23 (95.8%)
Partly solved / substantively wrong	2 / 3	2 / 1	0 / 1
Mean end-to-end time	223.2 s	157.1 s	93.2 s
P50 / P90	137.2 / 464.7 s	136.8 / 256.2 s	89.0 / 123.0 s
Mean tool calls	77.5	24.5	7.8

xychart-beta
  title "Mean tool calls per question"
  x-axis ["LLM Wiki", "Context", "Context + Source Graph"]
  y-axis "Tool calls" 0 --> 80
  bar [77.5, 24.5, 7.8]

LLM Wiki and Context used the same model and are directly comparable. The third configuration also changed the model; the same-model comparison below isolates Source Graph's effect.

The method follows Evaluating Your Knowledge Base; the grading and timing rules are in Evaluation Scoring and Timing Standard.

What was compared

Configuration	Knowledge and retrieval
LLM Wiki	A one-off generated LLM Wiki; the Agent searches and reads files locally
Context	A Context knowledge package; the Agent retrieves locally
Context + Source Graph	A Context knowledge package; code is read remotely at a pinned version through Source Graph

The benchmark measures answering with deployed knowledge. Knowledge production — quality, time, review effort and cost of producing and updating knowledge — has not been benchmarked yet.

Questions and scoring

The 24 questions came from real developer questions and were frozen before any run. Each configuration answered each question once.

Question type	Count
Locating interfaces and pages	7
Business interaction rules	5
Interfaces and parameters	4
Permissions and roles	3
Versions and call relationships	3
Debugging and environments	2

• Each question had criteria written in advance; answers were graded fully solved, partly solved, substantively wrong or unanswered.
• Grading was done by a model (GPT-6 Astra Ultra) blind to the configuration, without human calibration.
• Time is the end-to-end time until the caller saw the complete answer: mean, P50 and P90 (nearest-rank).
• Tool calls count only calls visible to the Agent.

On the 17 questions all three solved

To remove differences in question difficulty, the 17 questions that every configuration fully solved were compared separately:

Metric	LLM Wiki	Context	Context + Source Graph
Mean end-to-end time	232.6 s	160.7 s	93.5 s
P90	534.5 s	258.8 s	123.2 s
Mean tool calls	78.3	24.9	8.0

Source Graph with the same model

With Claude Opus 5.5 in both runs, Context with local retrieval fully solved 23 of 24 questions with a mean of 139.3 s, P90 188.7 s and 11.6 tool calls. Adding Source Graph kept 23 of 24 and reduced mean time by 33.1%: 256 local calls for repository preparation, shell, search and file reads were replaced by 164 remote retrievals. Eight remote calls failed parameter validation and the Agent recovered from all of them.

Under the same knowledge and retrieval, another model (Grok 4.7) also fully solved 23 of 24 but took a mean of 304.0 s, about 2.2 times as long. At equal quality, model choice has a large effect on time.

Three models on the same knowledge setup

With the same Context knowledge package on a managed Agent platform, the same 24 questions, one run each and the same blind model grader:

Metric	GPT 5.6 SOL (first run)	Grok 4.7	Claude Opus 5.5
Fully solved	21 (87.5%)	23 (95.8%)	23 (95.8%)
Partly solved / substantively wrong	2 / 1	0 / 1	0 / 1
Mean / P50 / P90 end-to-end	157.1 / 136.8 / 256.2 s	304.0 / 314.2 / 465.9 s	139.3 / 141.4 / 188.7 s
Mean visible tool calls	24.5	40.3	11.6
Mean observed startup	31.8 s	34.4 s	40.0 s

• On the 20 questions all three fully solved, mean end-to-end time was 162.8 s, 312.7 s and 142.3 s, with 25.4, 41.3 and 12.2 tool calls.
• Against the first run, Claude Opus 5.5 was faster on 18 of 24 questions, with P90 26.3% lower and 52.6% fewer tool calls. Grok 4.7 was slower on all 24 and made 64.0% more tool calls; the longer investigations did not raise the full-resolution rate.
• Equal counts are not equal answers. Each newer model fixed some of the first run's gaps and introduced a different error of its own.
• The same conclusion can come by very different paths. On one question both models were correct: one took 493.7 s and 75 calls, the other 176.4 s and 18.

What the traces showed

• When the host repository did not contain the target address, the knowledge gave the page → remote SDK → interface chain directly, so the Agent did not search blindly across repositories.
• Questions about differences between versions first need "compared with which version"; the knowledge recorded the pinned commits and comparison range of its sources.
• The few errors came from knowledge that missed a component's branch condition and over-generalized. Such gaps should become revision tasks.
• Tool calls fell from 77.5 to 24.5 and then to 7.8: knowledge reduced blind exploration, and remote retrieval replaced local repository preparation and file reads.

Where the errors and overhead came from

• Search scope misjudged. In a sparse checkout the caller's directory was not checked out, yet a local search was taken as proof that nothing in the whole repository referred to the target. Having read source code is not the same as having searched all of it; confirm the checkout scope before any repository-wide negative conclusion.
• A condition generalized. The knowledge stated a region for direct calls and kept field-based calls for dynamic forms; the answer extended the first to every request, and a field for another region in the source was the counterexample. The grade fell because of the wrong claim, not because the process was not shown.
• Targeted source checks worked. On the question where the first run missed a parent-page branch condition, both later models traced the parent's component choice and the history data switch and corrected the conclusion.
• Channel preparation. Called directly, the configuration under test still ran the steps meant for a chat channel's reply cards: 54 visible related calls across the first run's 24 questions, and for another model — which resolved links and checked channel context on every question — 149 deduplicated calls, 15.4% of its tool calls. They do not mean a card was sent, and their time was not separated. Trimming delivery preparation at the entry point would remove steps unrelated to answering.
• Waiting for startup. From submission to first observing "running" took about 32 s on the managed platform, including polling granularity.

Conditional estimate: faster startup

This is a conditional estimate, not a measurement. Replace each first-run question's measured startup with an assumed target and keep every other stage unchanged:

estimated E2E = measured E2E − measured startup + assumed startup target

Assumed startup	Mean	P50	P90
5 s	130.4 s	105.5 s	229.9 s
10 s	135.4 s	110.5 s	234.9 s

Whether such a target is reachable has to be re-measured on the same questions and configuration.

What to improve next

• Verify the checkout scope before concluding that nothing in a repository uses something.
• Keep branch conditions when knowledge is merged or summarized.
• Skip reply-card preparation in direct sessions that have no delivery target.

Estimated cost

Per question, tokens fell from about 567,000 to about 63,000. At standard API prices (input $4 and output $20 per million tokens, no cache discount) that is about $2.45 → $0.27, a reduction of 88.9%. The first and model-switched runs were estimated by replaying requests; the LLM Wiki and latest runs were extrapolated from tool-call counts.

Limits of this comparison

• Each question was run once, so stability cannot be inferred.
• These are whole configurations: LLM Wiki and Context ran in different Agent environments with different system prompts; Context + Source Graph also changed the model and Context version, the run date, concurrency and timeout limits, and is a regression on already-seen questions with a single grader.
• End-to-end time includes environment start-up and polling granularity, not quota queuing.
• Cost is an estimate, as described above.
• In the three-model comparison, the two later runs allowed up to four concurrent questions per model; the first allowed at most two and was partly serial. Dates, Skills, runtime and some read versions also differed, so not every difference belongs to the model.

Size reduction is a different measure: knowledge is about 1/115 of its sources by volume, which says nothing about quality on its own. See Knowledge for Agents.

知识检索测评

结果概览

在来自一个大型生产级全栈 monorepo 的 24 个真实开发者问题上，基于 Context 知识作答的 Agent 比使用 LLM Wiki 的 Agent 完全解决的问题更多、速度更快，工具调用也少得多。再加上 Source Graph 远程读取代码后，耗时和工具调用进一步下降。

指标	LLM Wiki	Context	Context + Source Graph
作答模型	GPT 5.6 SOL	GPT 5.6 SOL	Claude Opus 5.5
完全解决	19（79.2%）	21（87.5%）	23（95.8%）
部分解决 / 实质错误	2 / 3	2 / 1	0 / 1
平均端到端耗时	223.2 s	157.1 s	93.2 s
P50 / P90	137.2 / 464.7 s	136.8 / 256.2 s	89.0 / 123.0 s
平均工具调用次数	77.5	24.5	7.8

xychart-beta
  title "每题平均工具调用次数"
  x-axis ["LLM Wiki", "Context", "Context + Source Graph"]
  y-axis "工具调用次数" 0 --> 80
  bar [77.5, 24.5, 7.8]

LLM Wiki 和 Context 使用同一模型，可以直接比较。第三种配置同时更换了模型；下文的同模型对比单独考察了 Source Graph 的作用。

测评方法遵循评测知识库效果；判分和计时规则见测评评分与计时标准。

对比了什么

配置	知识与检索方式
LLM Wiki	一次性生成的 LLM Wiki；Agent 在本地搜索和读取文件
Context	Context 知识包；Agent 在本地检索
Context + Source Graph	Context 知识包；通过 Source Graph 按固定版本远程读取代码

本测评衡量的是使用已部署知识作答的效果。知识生产，即生产和更新知识的质量、耗时、审核工作量与成本，尚未测评。

问题与评分

这 24 个问题来自真实的开发者提问，并在任何一次运行之前就已固定。每种配置对每个问题各回答一次。

问题类型	数量
定位接口和页面	7
业务交互规则	5
接口与参数	4
权限与角色	3
版本与调用关系	3
调试与环境	2

• 每个问题都预先写好评判标准；答案分为完全解决、部分解决、实质错误和未作答。
• 评分由模型（GPT-6 Astra Ultra）完成，评分时不知道答案来自哪种配置，未经人工校准。
• 耗时是调用方看到完整答案为止的端到端时间，统计平均值、P50 和 P90（最近秩法）。
• 工具调用只统计 Agent 可见的调用。

三种配置都解决的 17 个问题

为排除问题难度的差异，单独比较了所有配置都完全解决的 17 个问题：

指标	LLM Wiki	Context	Context + Source Graph
平均端到端耗时	232.6 s	160.7 s	93.5 s
P90	534.5 s	258.8 s	123.2 s
平均工具调用次数	78.3	24.9	8.0

同一模型下的 Source Graph

两次运行都使用 Claude Opus 5.5 时，本地检索的 Context 完全解决了 24 个问题中的 23 个，平均耗时 139.3 s，P90 为 188.7 s，平均 11.6 次工具调用。加上 Source Graph 后仍完全解决 23 个，平均耗时减少 33.1%：用于仓库准备、Shell、搜索和读文件的 256 次本地调用被 164 次远程检索取代。有 8 次远程调用未通过参数校验，Agent 都自行恢复了。

在相同的知识和检索方式下，另一个模型（Grok 4.7）同样完全解决了 23 个问题，但平均耗时 304.0 s，约为前者的 2.2 倍。在质量相同时，模型选择对耗时影响很大。

同一知识配置下的三个模型

在托管 Agent 平台上使用同一个 Context 知识包、同样的 24 个问题，每题运行一次，并由同一个模型评审匿名判分：

指标	GPT 5.6 SOL（首轮）	Grok 4.7	Claude Opus 5.5
完全解决	21（87.5%）	23（95.8%）	23（95.8%）
部分解决 / 实质错误	2 / 1	0 / 1	0 / 1
平均 / P50 / P90 端到端耗时	157.1 / 136.8 / 256.2 s	304.0 / 314.2 / 465.9 s	139.3 / 141.4 / 188.7 s
平均可见工具调用次数	24.5	40.3	11.6
平均观测启动时间	31.8 s	34.4 s	40.0 s

• 在三个模型都完全解决的 20 个问题上，平均端到端耗时分别为 162.8 s、312.7 s 和 142.3 s，工具调用分别为 25.4、41.3 和 12.2 次。
• 与首轮相比，Claude Opus 5.5 在 24 个问题中的 18 个上更快，P90 降低 26.3%，工具调用减少 52.6%。Grok 4.7 在全部 24 个问题上都更慢，工具调用多 64.0%；更长的调查过程并没有提高完全解决率。
• **完全解决数相同不代表回答相同。**两款较新的模型各自修复了首轮的部分缺口，也各自引入了一个不同的错误。
• **同一个结论可以经由差别很大的路径得到。**某个问题上两个模型都答对了：一个用了 493.7 s 和 75 次调用，另一个用了 176.4 s 和 18 次。

从轨迹中看到的

• 当宿主仓库中不包含目标地址时，知识直接给出了“页面 → 远程 SDK → 接口”这条链路，Agent 无需在多个仓库中盲目搜索。
• 关于版本差异的问题，首先要明确“与哪个版本比较”；知识中记录了其来源的固定提交和比较范围。
• 少数错误源于知识遗漏了某个组件的分支条件并过度概括。这类缺口应转化为修订任务。
• 工具调用从 77.5 次降到 24.5 次，再降到 7.8 次：知识减少了盲目探索，远程检索取代了本地仓库准备和文件读取。

错误和开销从哪里来

• **搜索范围判断错误。**在稀疏检出的工作树里，调用方所在的目录没有被检出，局部搜索的结果却被当成了整个仓库都没有引用目标的证据。读过源码不等于搜遍了源码；在得出任何全仓否定结论之前，先确认检出范围。
• **条件被泛化。**知识写明了直接调用对应的某个地区，也保留了动态表单按字段调用的情况；回答却把前者推广到所有请求，源码中另一个地区的字段就是反例。降档是因为错误的断言，而不是因为没有展示过程。
• **有针对性的源码核查奏效。**在首轮漏掉父页分支条件的那道题上，后两个模型都追踪到了父页的组件选择和历史数据切换，纠正了结论。
• **渠道准备。**被直接调用时，被测配置仍执行了为聊天渠道回复卡片准备的步骤：首轮 24 个问题中共有 54 次可见的相关调用；另一个模型每道题都解析链接并检查渠道上下文，去重后有 149 次调用，占其工具调用的 15.4%。这些调用并不代表发送了卡片，其耗时也没有单独分离。在入口处尽早裁掉投递准备，就能去掉与作答无关的步骤。
• **等待启动。**在托管平台上，从提交到首次观察到“运行中”约需 32 s，其中包含轮询粒度。

条件估算：更快的启动

这是条件估算，不是实测。把首轮每道题实测的启动时间替换为假设的目标值，其余各阶段保持不变：

估算 E2E = 实测 E2E − 实测启动时间 + 假设的启动目标

假设启动时间	平均	P50	P90
5 s	130.4 s	105.5 s	229.9 s
10 s	135.4 s	110.5 s	234.9 s

这样的目标能否达到，需要在相同问题和配置上重新测量。

接下来要改进的

• 在断定仓库里没有任何地方用到某样东西之前，先确认检出范围。
• 合并或概括知识时保留分支条件。
• 在没有投递对象的直接会话中跳过回复卡片的准备。

成本估算

每个问题的 token 用量从约 567,000 降到约 63,000。按标准 API 价格（每百万 token 输入 $4、输出 $20，不计缓存折扣）计算，约为 $2.45 → $0.27，降低 88.9%。首次运行和更换模型的运行通过回放请求估算；LLM Wiki 和最新一次运行根据工具调用次数外推。

这项对比的局限

• 每个问题只运行了一次，无法据此判断稳定性。
• 对比的是整套配置：LLM Wiki 和 Context 运行在不同的 Agent 环境中，系统提示词也不同；Context + Source Graph 还更换了模型和 Context 版本、运行日期、并发和超时限制，而且是在已见过的问题上做的回归，只有一个评分者。
• 端到端耗时包含环境启动和轮询粒度带来的时间，不包含配额排队。
• 成本如上所述，只是估算。
• 在三个模型的对比中，后两轮每个模型最多允许 4 个问题并发；首轮最多 2 个，且部分是串行的。日期、Skill、运行环境以及部分读取的版本也不相同，因此并非所有差异都来自模型本身。

体积缩减是另一种衡量：知识的体积约为其来源的 1/115，但这本身并不能说明质量。参见写给 Agent 的知识。