Evaluating Your Knowledge Base

Answer quality and speed depend on the knowledge — its accuracy, completeness and links to evidence — but also on the model, the runtime, retrieval planning and how tools are executed. A good evaluation compares your knowledge base against an alternative, or against its own earlier version, on real questions, and then uses the run traces to find out why answers were slow or wrong.

This page is the design guide: follow it, or hand it to an Agent, to plan an evaluation for your own business. The exact grading, statistics and timing rules are in Evaluation Scoring and Timing Standard; a complete worked run is Knowledge Retrieval Benchmark.

flowchart LR
  set["1 Questions<br/>and guides"] --> freeze["2 Freeze<br/>conditions"]
  freeze --> run["3 Run and<br/>keep evidence"]
  run --> grade["4 Blind<br/>grading"]
  grade --> review["5 Review and<br/>attribute"]
  review --> report["6 Publish with<br/>limits"]

Decide what you are measuring

• Grade the final answer the user received. Ask two things: is the core conclusion correct, and did it deliver what the question asked? Traces explain retrieval, tools and time; the Agent does not have to show its reasoning or retell how it checked.
• Report quality, end-to-end time and visible tool calls separately. Do not merge them into one score with arbitrary weights.
• You are comparing whole setups. Even with the same model, knowledge organization, system prompt, tools, permissions and runtime can differ, so not every difference belongs to the knowledge base.
• Say what is out of scope. Answering with deployed knowledge is one thing; producing and updating the knowledge — its quality, time, review effort and cost — is another, and needs its own evaluation.

Build and freeze the question set

1. Sample real user questions. Keep their wording, necessary context and terms. Follow-ups that belong together can be merged into one frozen input, but never add answer hints, source paths, extra locating words or grading requirements.
2. Give every question a stable ID and keep its origin. The input sent to an Agent contains only the user's question — no ID, earlier answer, rubric or reference evidence.
3. Turn images into faithful text when they cannot be sent identically to every setup; mark unreadable parts as unknown. Leave out questions that cannot be answered without the missing image, and add them in a later version.
4. Separate rule explanations, code lookups and live troubleshooting. Do not quietly add online permissions, deployment versions or real requests the user never gave.
5. Freeze the list and record its hash. Questions already used to tune prompts or fill knowledge gaps are a regression set; keep unseen held-out questions to check that improvements generalize.
6. Cover the questions people actually ask — for example locating interfaces and pages, interaction rules, parameters, permissions and roles, versions and call relationships, debugging and environments.

Write a guide for every question

Each question gets its own file with:

• the exact test input;
• the date it was checked, the source version (a pinned commit) and what is still unchecked;
• the confirmed reference facts and the conditions under which the conclusion holds;
• what the answer must deliver;
• acceptable equivalent wordings, other valid approaches, and what is not required;
• concrete criteria for each of the four grades, and counterexamples that would overturn a wrong claim;
• where the evidence is, which fact it supports, and known gaps.

The reference answer is for checking facts. It does not prescribe length, headings, number of citations, tool order or a full chain of proof, and it is not a template to reproduce. A conclusion that stayed inside the Agent's process and never reached the user does not count. When a deciding fact cannot be verified, mark it pending rather than borrowing an earlier assistant's answer or a knowledge page as proof. If a guide later turns out to be incomplete, record the correction with evidence instead of failing every answer that disagrees with the old guide.

Freeze the comparison conditions

Fix, and write down, everything that could change the result:

• question text, model and reasoning settings, knowledge and source-code scope, system prompt, tool permissions, region and dates;
• whether a setup answers from knowledge only or knowledge plus source checks — never let only one side read sources without saying so;
• cache and long-term memory state, concurrency, run order and number of repeats.

Start a new task for every question and send every setup exactly the same text. Grading guides never enter the answering context. The default recommendation is three runs per question, concurrency one, alternating setup order, with the order or random seed recorded; state it plainly when you choose otherwise. Each change to the questions, guides or configuration gets a new version and a new results directory; old material is kept, and re-runs never overwrite earlier results.

Run and keep the evidence

For every attempt, keep the start and end timestamps with time zone, task or session IDs and links, the complete final answer, the raw tool events, the status, timing precision and any anomaly. The first progress message is not the answer; even when a platform marks a task done, check that the answer was actually delivered.

Keep the original receipts of platform failures, timeouts, cancellations, quota refusals and retries. An extra run is linked to the attempt it repeats; never replace a result with the best of several. If a question needs the user to supply more information, define one multi-turn protocol for every setup and label any ad hoc follow-up or resume.

If the Agent under test is configured for a chat channel — for example it assembles reply cards — a direct-call evaluation may still run channel checks and card preparation. To measure answering alone, switch to a reply policy suited to direct calls, or record that overhead separately.

Grade blind, then review

• Give each answer a random alias per question and record the seed. First-pass graders see neither the setup nor the time, tool count or earlier scores. Links or wording inside an answer can still reveal its origin, so call this limited anonymity.
• Check material errors first, then whether the question was fully answered, and assign one of four grades — or pending verification when a deciding fact cannot yet be checked. The rules are in the standard.
• Review every material error, pending item, partial resolution and borderline call, and spot-check full resolutions. Graders must give reasons someone else can check; length, style, tool order and whether the process is shown are not reasons on their own.
• Check grader consistency with separate calibration samples: the same facts in a short and a long version, as a paragraph and as a table, and the same pair shown in swapped order should get the same grade. Without human-labelled samples, call it internal consistency review, not human calibration.

Find the causes

Numbers say that something is slow or wrong; traces say why. Only after grading, reveal which setup produced which answer, then read the raw tool trace and classify each gap:

• Wrong or missing knowledge — the knowledge base does not cover the fact, or states it wrongly.
• A key condition left out — for example a component's branch condition, leading to an over-generalized answer. Turn such gaps into revision tasks.
• Search scope misjudged — for example concluding "nothing calls this" from a partial checkout.
• Inference or generalization error, or delivery gap — the Agent knew it but did not tell the user.
• Evidence unreachable, repeated searching, channel overhead or waiting for startup.

A log supports only the behaviour it actually shows, and correlation is not cause. More tool calls do not mean more care, and faster does not mean more accurate.

Publish results with their limits

Show the four grade counts with their denominator, mean, P50 and P90 time, visible tool calls, and the sample and protocol. Compare on all questions and on the subset every setup fully resolved. Link each question to its answer, guide, grading, timings and evidence, and seal the raw package with a manifest and hashes.

State everything a reader needs to interpret the numbers: how many times each question ran (a single run says nothing about stability), differences between environments, system prompts, models, versions, dates, concurrency and timeouts, whether the questions had been seen before, how many graders there were and whether they were calibrated, what the timings include, and how any cost figure was estimated. Label every improvement figure as measured, conditional estimate (with its formula and assumptions) or untested direction.

What the evaluation should produce

For the target business: a question document with origins, a separate scoring standard, one guide per question, a collection plan (configurations, repeats, order, timeout, recorded fields), a raw-records directory and a summary report. Freeze the questions and criteria before running any Agent. A new business checks its own sources, question scope, required conditions and counterexamples; it does not reuse another business's factual judgements.

Have an Agent design it

An Agent can draft the whole plan if it reads this page and the standard first. Copy the prompt below, fill in the brackets, and review the result before any run:

Read the "Evaluation" group of the Context documentation at
https://context4ai.org/context/llms.txt — "Evaluating Your Knowledge Base",
"Evaluation Scoring and Timing Standard" and "Knowledge Retrieval Benchmark".
Then design an evaluation of [knowledge base or assistant] against [baseline or
earlier version] for [business area].

Produce, without running anything yet:
1. A question document: 20–30 real user questions sampled from [where], each
   with a stable ID, its origin, the frozen input text and its question type.
2. A scoring standard adapted from the documented one: two primary metrics,
   material errors, four grades and pending verification.
3. One guide per question: checked facts with the pinned source version,
   required deliverables, acceptable alternatives, what is not required, grade
   criteria, counterexamples and evidence locations. Mark anything unverified.
4. A collection plan: configurations to freeze, repeats, order or seed,
   concurrency, timeout, timing events and every field to record.
5. A blind-grading and review plan, and the report outline with its limits.

Do not reuse facts from the documentation's examples, do not add hints to the
questions, and list every decision you need from me.

评测知识库效果

回答的质量和速度取决于知识本身——是否准确、完整、能链接到证据——也取决于模型、运行环境、检索规划和工具的执行方式。好的测评会在真实问题上，把你的知识库与另一种方案或它自己的早期版本做比较，再借助运行轨迹找出回答慢或错的原因。

本页是设计指南：可以照着做，也可以交给 Agent，为你自己的业务规划一次测评。具体的判分、统计和计时规则见测评评分与计时标准；完整的实测示例见知识检索测评。

flowchart LR
  set["1 题集<br/>与逐题指南"] --> freeze["2 冻结<br/>对照条件"]
  freeze --> run["3 运行并<br/>保存证据"]
  run --> grade["4 匿名<br/>判分"]
  grade --> review["5 复核<br/>与归因"]
  review --> report["6 连同局限<br/>一起发布"]

先确定测什么

• **评价用户实际收到的最终回答。**问两件事：核心结论是否正确？是否交付了问题要求的内容？轨迹用来解释检索、工具和耗时；不要求 Agent 展示推理或复述查证过程。
• **质量、端到端耗时和可见工具调用次数分别报告。**不要用随意的权重把它们合成一个分数。
• **比较的是整体方案。**即使模型相同，知识组织、系统提示、工具、权限和运行环境也可能不同，所以并非所有差异都来自知识库。
• **说明不在范围内的部分。**用已部署的知识作答是一回事；生产和更新知识——它的质量、耗时、审核工作量和成本——是另一回事，需要单独测评。

建立并冻结题集

1. **从真实用户问题中抽样。**保留原始措辞、必要上下文和术语。相连的追问可以合并成一次冻结输入，但绝不能加入答案提示、来源路径、额外的定位词或评分要求。
2. **每题给一个稳定 ID 并保留出处。**发给 Agent 的输入只包含用户的问题——不带 ID、历史答案、评分表或参考证据。
3. 图片无法同样地发给每种方案时，转成忠实的文字；看不清的部分标为未知。缺了图片就无法作答的题先排除，在后续版本中再补入。
4. **区分规则说明、代码查证和现场排障。**不要暗中补上用户从未给出的线上权限、部署版本或真实请求。
5. **冻结题单并记录其哈希。**已经被用来调提示词或补知识缺口的题属于回归集；另外保留没见过的保留题，检查改进能否泛化。
6. 覆盖人们实际会问的问题类型——例如定位接口和页面、交互规则、参数、权限与角色、版本与调用关系、调试与环境。

为每道题写一份指南

每道题单独一个文件，包含：

• 原样的测试输入；
• 核实日期、来源版本（固定的提交）以及尚未核实的范围；
• 已确认的参考事实，以及结论成立的条件；
• 回答必须交付的内容；
• 可接受的等价表述、其他有效解法，以及不强求的内容；
• 四档结果各自的具体判据，以及能推翻错误断言的反例；
• 证据所在位置、它支持哪项事实，以及已知缺口。

参考答案用于核实事实。它不规定篇幅、标题、引用数量、工具顺序或完整的查证链，也不是要照抄的模板。只停留在 Agent 过程里、没有交付给用户的结论不算数。决定性事实无法核实时，标为待核验，不要拿之前某个助手的回答或某个知识页当作证据。如果后来发现指南不完整，带着证据记录勘误，而不是把所有与旧指南不一致的回答都判错。

冻结对照条件

把所有可能影响结果的因素固定下来并记录：

• 题面、模型与推理设置、知识与源码范围、系统提示、工具权限、运行区域和日期；
• 方案是只用知识作答，还是知识加源码核查——绝不能只让一方读源码而不说明；
• 缓存与长期记忆状态、并发、运行顺序和重复次数。

每道题都新开一个任务，并给每种方案发送完全相同的文本。判分指南绝不进入作答上下文。默认建议每题运行三次、并发为一、交错各方案的运行顺序，并记录顺序或随机种子；选择其他做法时要明确说明。题集、指南或配置每改动一次，就使用新版本和新的结果目录；旧材料保留，复跑不覆盖之前的结果。

运行并保存证据

每次尝试都要保存：带时区的开始和结束时间戳、任务或会话 ID 与链接、完整的最终回答、原始工具事件、状态、计时精度以及任何异常。第一条进度消息不是答案；即使平台把任务标为完成，也要确认答案确实已经交付。

保留平台失败、超时、取消、配额拒绝和重试的原始回执。额外的运行要关联到它所重复的那次尝试；绝不能从多次结果中挑最好的一次替换。如果某道题需要用户补充信息，为所有方案定义同一套多轮协议，并标注任何临时追问或恢复。

如果被测 Agent 是为聊天渠道配置的——例如它会拼装回复卡片——直接调用测评时它仍可能执行渠道检测和卡片准备。要单独测量作答，改用适合直接调用的回复策略，或把这部分开销单独记录。

匿名判分，再复核

• 每道题给每个回答一个随机别名，并记录随机种子。初评者既看不到方案，也看不到耗时、工具次数或之前的分数。回答里的链接或措辞仍可能暴露来源，所以应称之为有限匿名。
• 先排查实质错误，再判断问题是否被完整回答，然后给出四档之一——决定性事实暂时无法核实时，标为待核验。规则见标准。
• 复核全部实质错误、待核验、部分解决和边界判定，并抽查完全解决。评审必须给出别人可以复核的理由；篇幅、文风、工具顺序和是否展示过程本身都不能作为理由。
• 用单独的校准样例检查评审一致性：同样的事实写成短版和长版、段落和表格，以及同一对回答交换展示顺序，都应得到相同的判定。没有人工标注样本时，应称之为内部一致性复核，而不是人工校准。

找出原因

数字说明存在慢或错；轨迹说明为什么。判分完成之后，才揭示每个回答来自哪种方案，再阅读原始工具轨迹，给每个差距归类：

• 知识有误或缺失——知识库没有覆盖该事实，或写错了。
• 遗漏了关键条件——例如组件的分支条件，导致回答过度概括。这类缺口应转为修订任务。
• 搜索范围判断错误——例如根据局部检出就得出“没有任何地方调用它”的结论。
• 推断或归纳错误，或漏交付——Agent 知道，却没有告诉用户。
• 证据不可达、重复搜索、渠道开销或等待启动。

日志只能支持它实际显示的行为，相关也不等于因果。工具调用多不代表更认真，更快也不代表更准确。

连同局限一起发布结果

给出四档数量及其分母、平均与 P50、P90 耗时、可见工具调用次数，以及样本和协议。既在全部题上比较，也在各方案都完全解决的子集上比较。把每道题链接到它的回答、指南、判分、计时和证据，并用清单和哈希封存原始数据包。

写明读者解读这些数字所需的一切：每道题运行了几次（只运行一次无法说明稳定性），环境、系统提示、模型、版本、日期、并发和超时的差异，题目是否事先见过，有几位评审、是否经过校准，计时包含什么，以及成本数字是如何估算的。每个改进数字都要标明是实测、条件估算（附公式与假设）还是待验证方向。

测评应产出什么

针对目标业务：一份带出处的问题文档、一份独立的评分标准、每题一份指南、一份采集计划（配置、重复次数、顺序、超时、记录字段）、一个原始记录目录，以及一份汇总报告。在运行任何 Agent 之前先冻结题目和判据。新的业务要自行核实来源、题目范围、必要条件和反例，不沿用其他业务的事实判断。

交给 Agent 设计

只要先读本页和标准，Agent 就能起草整份方案。复制下面的提示词，填好方括号里的内容，并在任何运行之前审阅结果：

Read the "Evaluation" group of the Context documentation at
https://context4ai.org/context/llms.txt — "Evaluating Your Knowledge Base",
"Evaluation Scoring and Timing Standard" and "Knowledge Retrieval Benchmark".
Then design an evaluation of [knowledge base or assistant] against [baseline or
earlier version] for [business area].

Produce, without running anything yet:
1. A question document: 20–30 real user questions sampled from [where], each
   with a stable ID, its origin, the frozen input text and its question type.
2. A scoring standard adapted from the documented one: two primary metrics,
   material errors, four grades and pending verification.
3. One guide per question: checked facts with the pinned source version,
   required deliverables, acceptable alternatives, what is not required, grade
   criteria, counterexamples and evidence locations. Mark anything unverified.
4. A collection plan: configurations to freeze, repeats, order or seed,
   concurrency, timeout, timing events and every field to record.
5. A blind-grading and review plan, and the report outline with its limits.

Do not reuse facts from the documentation's examples, do not add hints to the
questions, and list every decision you need from me.