Evaluation Scoring and Timing Standard

This page is the rulebook referred to by Evaluating Your Knowledge Base. Use it as written, or adapt it and say what you changed. It covers how one answer is graded, how grades become rates, and how time and tool calls are measured. Knowledge Retrieval Benchmark applies it.

Industry methods it draws on

The standard borrows ideas from published work. The four grades, sampling and recording conventions are choices made for this project, not an industry-wide standard.

Source	What was borrowed
SimpleQA (OpenAI)	Classify answers as correct, incorrect or not attempted, and count "not attempted" separately so refusals are neither errors nor successes.
FACTS Grounding (Google DeepMind)	First judge whether the answer addresses the request, then check its factual grounding — two separate judgements.
Demystifying evals for AI agents (Anthropic)	Grade an Agent by its outcome rather than a fixed tool order or path; keep the full trace to explain the result.
Ragas Factual Correctness and Faithfulness	Distinguish "agrees with the reference facts" from "agrees with the retrieved material": material can be stale or miss a condition, so faithfulness is not correctness.
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena	Model judges show position, verbosity and self-preference bias; swap order, anonymize and check with calibration samples.
Nearest-rank percentile	P90 is the ⌈0.9 × n⌉-th sorted sample, so the figure is always a real, checkable run.

Two primary metrics and material errors

Metric	What is judged	Recorded value
Core-conclusion correctness	Do the key conclusions given hold within the question's scope? Is a condition missing that makes them misleading?	correct / material error / no gradable conclusion / pending
Request fulfilment	Was what the user asked for actually delivered? Judged only against the question and its necessary context.	full / partial / not met / pending

Material errors are recorded separately as yes, no or pending. For a yes, record the exact claim, the evidence against it and its effect on the user's judgement or action. Any one of these, once confirmed, is a material error:

1. the core conclusion is wrong;
2. a key recommendation would lead the user to do the wrong thing;
3. deciding evidence is fabricated;
4. an unverified live situation is asserted as confirmed fact;
5. a necessary condition is left out, turning the answer into a wrong unconditional statement.

Typos and minor wording issues that change no judgement or action are only noted. A correct main conclusion with a materially wrong side recommendation is still a material error. Correct but irrelevant extra information earns nothing.

Four grades and the grading order

Grade	Rule
Fully resolved (resolved)	Core conclusion correct, request fully met, nothing materially misleading.
Partially resolved (partially_resolved)	Useful, correct content delivered, but part of the request or a needed fact is missing; no confirmed material error.
Material error (material_error)	The core conclusion or a key recommendation is materially wrong, including misleading unconditional generalizations and fabricated deciding evidence.
Not resolved (unresolved)	No substantive answer — only a plan, only a request for more information, or "cannot answer". A wrong refusal that itself contains a confirmed key error is still a material error.

Order: first look for confirmed material errors, then judge coverage of the request. Correct content does not offset a key error.

Pending verification (pending_verification) is a review state, not a fifth grade. When a deciding fact cannot be checked yet, keep the check open and do not grade; never record it as pass or fail. If existing evidence already confirms a material error, grade it directly. Needs more information is an interaction state that can coexist with a grade: only asking questions is not resolved; answering part and then asking can be partially resolved.

Fairness rules

• Accept semantically equivalent short answers, long answers, lists, tables and different valid evidence paths. Check meaning and facts; do not match the reference text, fixed wording, headings or tool order. Form gains or loses nothing unless it actually hinders understanding, hides a key condition or creates ambiguity.
• A condition is necessary only if it changes the conclusion or what the user does. Do not universally require version fields, full call chains, parameter flows, edge-case lists or next steps.
• For "how do I debug this", the actionable method, prerequisites and steps are the deliverable; how the Agent searched need not appear.
• For "list all", check the complete set within the question's scope — a few correct examples may be only partial. Side topics the user did not ask about are not omissions.
• The user's assumptions, counts or terms may be wrong. Correcting the premise accurately and answering the real question can be fully resolved.
• Separate static rules from live facts. Honestly admitting an evidence gap is not an error, but the unfinished part of the request is still recorded.
• Missing citations, an unshown process or a short reply are not grounds for calling a conclusion wrong. Sources are a deliverable only when the user or the shared test protocol asks for them, told identically to every setup. Volunteered citations are checked for existence and support, not counted for points; a temporarily unreachable link is not fabrication. Local evidence existing, a link path being correct, and the link opening for the user's account are three different checks.

Using counterexamples

The grader asks: within the question's scope, is there an evidenced case that overturns a deciding claim, or shows that an omission would change the user's judgement? The counterexample must belong to the question's scope and affect the conclusion — do not widen the question on the fly. The answer need not contain a "counterexamples" section; a correct condition or scope statement is enough. Not mentioning a branch is not itself an error. An answer that keeps its scope and generalizes nothing wrongly but skips part of the request is partially resolved.

Three calibration patterns

These patterns are rewritten from real grading decisions with every business detail removed.

A definition existing is not a call happening. Where does the front end call an old endpoint? The code keeps a request wrapper and a monitoring list registers the address, but nothing in the checked scope calls it; pages now use a newer endpoint.

• Saying "no business call was found in the checked scope" fully resolves it.
• Listing the wrapper without judging whether it is called is partial.
• Treating the monitoring entry as a request, or claiming the endpoint is absent from the code entirely, is a material error.
• Only suggesting searches is not resolved.

Lesson: grade the real meaning and the user's intent, allow correcting the premise, and never score by how often an address appears.

Role differences are also gated by state and permission. What can an owner do on a list page that a non-owner cannot? In reality each action has its own conditions — role, workflow state, item type — and a detail entry is further controlled by a switch and an organization grant, so non-owners may also reach it.

• Stating the differences with their conditions fully resolves it.
• "Non-owners see no actions" or "owners can always act" is a material error.

Lesson: counterexamples test unconditional generalizations. Necessary conditions come from the question's scope; the answer need not show the grader's whole control chain.

An outer choice and an inner data switch are different layers. When the current workflow is the new version, does viewing an old snapshot bring back the old interface? The parent page picks a component from the current workflow; switching history inside it only changes the data. Only a different component renders history by its own version, and a revision number is not a version generation.

• Explaining both layers fully resolves it.
• Asserting that history always restores the old interface, or treating the revision number as the generation, is a material error.

Lesson: when existing knowledge misses the parent-level condition, go back to the source and correct the criteria — do not treat a knowledge page as the answer key.

Grader calibration and review

• Give each answer a random alias per question and record the seed. First-pass grading hides setup, time, tool count and earlier scores. Links or wording inside an answer can still reveal its origin, so state that anonymity is limited.
• Check consistency with separate calibration samples: the same facts short and long, as a paragraph and as a table, should get the same grade, and swapping the display order of two answers should not change the judgement. Store calibration samples apart from the real questions.
• Review every material error, pending item, partial resolution and borderline call, and spot-check full resolutions. Graders give reasons someone else can check; length, style, tool order and whether the process is shown are not universal plus or minus points.
• With only model graders and no human-labelled samples, call it an internal consistency review, never human calibration.

Denominators and statistics

• With Q frozen questions and R repeats, each configuration plans N = Q × R runs. Every grade rate divides by N; a partial resolution is never counted as half.
• Grade counts + pending + not run = N. Graded coverage = runs placed in one of the four grades ÷ N.
• A delivery failure or timeout with no answer is graded not resolved, and the platform, tool or cancellation reason is kept; it is not automatically a knowledge error.
• The default is three runs per question. A question counts toward the stable-resolution count only if all three runs are fully resolved; with single runs, do not report a stable-resolution rate.
• Compare performance on all questions and on the common subset every setup fully resolved, showing each denominator, so setups are not compared on different, easier questions.
• Label every improvement figure as measured, conditional estimate (with formula and assumptions) or untested direction.

Timing

• Primary end-to-end (E2E) time = when the tester received the complete final answer − when the question was submitted. It includes network, platform initialization, retrieval, tools, polling and delivery. Unit: seconds.
• Keep full timestamps with time zone. Across systems, state clock skew and precision; a missing value is never 0.
• Startup, first progress, first token, model text finished, runner finished and user received the final answer are different events. Name each by what was actually captured and never substitute one for another. A first progress message is not the final answer, and a task marked done still needs its answer confirmed as delivered.
• If only platform creation and final send receipts exist, report a separate platform E2E marked as a proxy; never mix it with receiver-side E2E.
• Agree the timeout before running; 600 seconds is a reasonable default. It is an execution limit, not a business target. Record the actual stop time for a timeout and give a re-run its own run ID; a timeout is never a successful 600-second sample. Failures without a final answer are not latency samples of successful delivery, but their rate and waiting time must be shown.
• Mean = sum of same-definition times ÷ sample count. P50 = median (average of the middle two for an even count). P90 = nearest-rank, the ⌈0.9 × n⌉-th sorted value. Always show the sample count; with repeats, also report each question's median.

Tool calls and channel overhead

• Count tool calls by unique call ID and state the coverage — events the platform exposes; internal calls of unexposed sub-Agents are not counted. One shell call may run several commands. Tool calls describe the execution path; they are not model calls or tokens. The span from first to last tool event includes reasoning, waiting and concurrency, so do not call it pure tool time.
• Keep original receipts for platform failures, timeouts, cancellations, quota refusals and retries. Link an extra run to the attempt it repeats; never replace a result with the best one. When a question needs the user to supply more information, define one multi-turn protocol for all setups and label ad hoc follow-ups and resumes.
• Channel overhead: an Agent configured for a chat channel — for example one that assembles reply cards — may still detect the channel, read card instructions and resolve links when called directly. For pure question answering, switch to a reply policy suited to direct calls, or record and exclude that overhead.

测评评分与计时标准

本页是评测知识库效果所引用的规则手册。可以原样采用，也可以调整后注明改了什么。它涵盖单个回答如何判分、判分如何变成比率，以及耗时和工具调用如何测量。知识检索测评就采用了这套标准。

参考的业界方法

本标准借鉴了已发表的工作。四档结果、采样和记录约定是本项目自己的选择，并非业界统一标准。

来源	借鉴之处
SimpleQA（OpenAI）	把回答分为正确、错误和未作答，并单独统计“未作答”，使拒答既不算错误也不算成功。
FACTS Grounding（Google DeepMind）	先判断回答是否回应了请求，再核验其事实依据——两项分开判定。
Demystifying evals for AI agents（Anthropic）	按结果评价 Agent，而不是按固定的工具顺序或路径；保留完整轨迹用于解释结果。
Ragas Factual Correctness 与 Faithfulness	区分“与参考事实一致”和“与检索到的材料一致”：材料可能过时或漏掉条件，所以忠实于材料不等于正确。
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena	模型评审存在位置偏差、冗长偏差和自我偏好；需要交换顺序、匿名化并用校准样例检查。
最近秩法（nearest-rank）百分位	P90 取排序后的第 ⌈0.9 × n⌉ 个样本，因此这个数字总是一次真实、可复核的运行。

两个主指标与实质错误

指标	判定内容	记录值
核心结论正确性	给出的关键结论在题目范围内是否成立？是否缺少会让结论产生误导的条件？	正确 / 实质错误 / 无可判定结论 / 待核验
原问需求满足程度	用户要求的内容是否真正交付？只按问题本身及其必要上下文判断。	完全 / 部分 / 未满足 / 待核验

实质错误单独记录为有、无或待核验。记为有时，记下具体断言、反驳证据，以及它对用户判断或操作的影响。以下任一情况一经确认，即为实质错误：

1. 核心结论错误；
2. 关键建议会让用户做错事；
3. 捏造决定性证据；
4. 把未经证实的现场情况断言为已确认事实；
5. 遗漏必要条件，使回答变成错误的无条件结论。

不改变任何判断或操作的笔误和次要措辞问题只做备注。主结论正确、但附带建议有实质错误的，仍属实质错误。正确但无关的额外信息不加分。

四档结果与判定顺序

结果	规则
完全解决（resolved）	核心结论正确，需求完全满足，没有实质性误导。
部分解决（partially_resolved）	交付了有用且正确的内容，但缺少部分请求或必要事实；没有已确认的实质错误。
实质错误（material_error）	核心结论或关键建议有实质错误，包括误导性的无条件概括和捏造决定性证据。
未解决（unresolved）	没有实质回答——只给计划、只要求补充信息，或说“无法回答”。错误的拒答本身包含已确认的关键错误时，仍记为实质错误。

**顺序：**先排查已确认的实质错误，再判断对需求的覆盖。正确内容不能抵消关键错误。

待核验（pending_verification）是评审状态，不是第五档。决定性事实暂时无法核实时，保留检查项、暂不定档；绝不能记为通过或失败。如果已有证据确认了实质错误，直接定档。需要补充信息是交互状态，可以与某一档并存：只提问题为未解决；先回答一部分再追问，可以是部分解决。

公平规则

• 语义等价的短答、长答、列表、表格以及不同的有效证据路径都接受。核对语义和事实，不去匹配参考文本、固定措辞、标题或工具顺序。形式本身不加分也不减分，除非它确实妨碍理解、掩盖了关键条件或造成歧义。
• 只有会改变结论或用户操作的条件才是必要条件。不统一要求版本字段、完整调用链、参数流、边界情况清单或下一步建议。
• 对于“怎样调试”，可执行的方法、前提和步骤就是要交付的内容；Agent 如何搜索不必出现在答案里。
• 对于“列出全部”，核对题目范围内的完整集合——给出几个正确示例可能只是部分解决。用户没问到的旁支话题不算遗漏。
• 用户的假设、数量或术语可能有误。准确纠正前提并回答真正的问题，可以是完全解决。
• 区分静态规则与现场事实。如实承认证据缺口不算错误，但请求中未完成的部分仍要记录。
• 没有引用、没有展示过程或回复较短，都不能作为判定结论错误的理由。只有当用户或统一的测试协议要求提供来源，并且对每种方案都同样告知时，来源才是交付要求。主动给出的引用核对其存在性和支持度，不按数量计分；链接暂时打不开不等于捏造。本地证据存在、链接路径正确、链接能用用户账号打开，是三种不同的检查。

反例的用法

评审要问：在题目范围内，是否存在有证据的情形能推翻某个决定性断言，或证明某处遗漏会改变用户的判断？反例必须属于题目范围并影响结论——不要临时扩大题目。答案不必专门写一节“反例”；正确的条件或范围表述就足够。没有提到某个分支本身不是错误。保持了范围、没有错误概括、但漏掉部分请求的回答，判为部分解决。

三类校准示例

这些示例改写自真实的判分决定，已去除所有业务细节。

**“定义存在”不等于“发生调用”。***前端在哪里调用了某个旧接口？*代码里保留了请求封装，监控清单里也登记了这个地址，但在核对范围内没有任何地方调用它；页面已改用较新的接口。

• 说明“核对范围内未发现业务调用”即为完全解决。
• 只列出封装、却不判断是否被调用，为部分解决。
• 把监控登记当作发起请求，或断言代码里完全不存在这个接口，为实质错误。
• 只建议怎么搜索，为未解决。

启示：按真实语义和用户意图判分，允许纠正前提，绝不按地址出现的次数打分。

**角色差异还受状态和权限控制。***在某个列表页上，负责人能做哪些非负责人不能做的操作？*实际上每项操作都有各自的条件——角色、流程状态、条目类型——某个详情入口还受开关和组织授权控制，因此非负责人也可能进入。

• 说明差异及其条件，即为完全解决。
• “非负责人看不到任何操作”或“负责人始终可以操作”为实质错误。

启示：反例用来检验无条件概括。必要条件来自题目范围；答案不必展示评审掌握的整条控制链。

**外层选择与内层数据切换是不同的层。**当前流程为新版时，查看旧快照会不会换回旧界面？父页按当前流程选择组件；在组件内切换历史只改变数据。只有另一种组件才会按历史自身的版本渲染，而且修订序号并不等于版本代际。

• 说明这两层，即为完全解决。
• 断言查看历史一定会换回旧界面，或把修订序号当作代际，为实质错误。

启示：已有知识漏掉父层条件时，要回到源码修正判据——不要把知识页当作标准答案。

评审校准与复核

• 每道题给每个回答一个随机别名并记录随机种子。初评隐藏方案、耗时、工具次数和之前的分数。回答中的链接或措辞仍可能暴露来源，因此要说明匿名是有限的。
• 用单独的校准样例检查一致性：同样的事实写成短版和长版、段落和表格，应得到相同的判定；交换两份回答的展示顺序，判断也不应改变。校准样例与正式题分开存放。
• 复核全部实质错误、待核验、部分解决和边界判定，并抽查完全解决。评审给出别人可以复核的理由；篇幅、文风、工具顺序和是否展示过程不作为统一的加减分项。
• 只有模型评审、没有人工标注样本时，称之为内部一致性复核，绝不称为人工校准。

分母与统计

• 设冻结题数为 Q、重复次数为 R，每种配置计划运行 N = Q × R 次。每档比率都除以 N；部分解决绝不折算为半个。
• 四档数量 + 待核验 + 未运行 = N。已评覆盖率 = 归入四档之一的运行数 ÷ N。
• 交付失败或超时且没有回答的，记为未解决，并保留平台、工具或取消原因；不会自动算作知识错误。
• 默认每题运行三次。只有三次都完全解决的题才计入稳定解决数；只有单次运行时，不报告稳定解决率。
• 性能既在全部题上比较，也在各方案都完全解决的共同子集上比较，并显示各自的分母，避免在不同的、更容易的题上比较各方案。
• 每个改进数字都要标明是实测、条件估算（附公式与假设）还是待验证方向。

计时

• 主端到端（E2E）耗时 = 测试端收到完整最终回答的时间 − 提交问题的时间。它包括网络、平台初始化、检索、工具、轮询和交付。单位：秒。
• 保存带时区的完整时间戳。跨系统时，说明时钟偏差和精度；缺失值绝不能记为 0。
• 启动、首条进度、首个 token、模型正文完成、运行器结束和用户收到最终回答是不同的事件。按实际采集到的事件命名，绝不互相替代。首条进度消息不是最终回答；任务被标为完成后，仍要确认回答已经交付。
• 如果只有平台创建回执和最终发送回执，就单独报告平台 E2E，并标明是代理口径；绝不与接收端 E2E 混在一起。
• 运行前先约定超时上限；600 秒是合理的默认值。它是执行上限，不是业务达标线。超时要记录实际停止时间，重跑使用新的运行 ID；超时绝不能算作一次成功的 600 秒样本。没有最终回答的失败不作为成功交付的延迟样本，但必须呈现其比率和等待时间。
• 平均 = 同口径耗时之和 ÷ 样本数。P50 = 中位数（偶数个样本取中间两项的平均）。P90 = 最近秩法，取排序后的第 ⌈0.9 × n⌉ 个值。始终显示样本数；有重复运行时，还要报告每道题的中位数。

工具调用与渠道开销

• 按唯一调用 ID 统计工具调用，并说明覆盖范围——即平台公开的事件；未公开的子 Agent 的内部调用不计入。一次 shell 调用可能运行多条命令。工具调用描述的是执行路径，不等于模型调用次数或 token。从第一个到最后一个工具事件的时间跨度包含推理、等待和并发，因此不要称之为工具纯耗时。
• 保留平台失败、超时、取消、配额拒绝和重试的原始回执。额外的运行关联到它所重复的那次尝试；绝不用最好的一次替换结果。当某道题需要用户补充信息时，为所有方案定义同一套多轮协议，并标注临时追问和恢复。
• **渠道开销：**为聊天渠道配置的 Agent——例如会拼装回复卡片的 Agent——被直接调用时，仍可能检测渠道、读取卡片说明并解析链接。若要比较纯问答，改用适合直接调用的回复策略，或把这部分开销记录下来并排除。