Ancha CLI 0.4.0, offline. The example is Casey, Slack's open-source IT help-desk agent (bolt-python, OpenAI Agents SDK), and 61 production traces of simulated employees. Ancha CLI 0.4.0，离线模式。例子是 Casey：Slack 开源的 IT 客服 agent（bolt-python，OpenAI Agents SDK），61 条生产 trace，员工是模拟的。 From a company's real conversations with its AI agent, to tests that replay them, to a fix you can count. The example is Casey, an IT help-desk agent on Slack. 从一家公司和自己 AI agent 的真实对话，到能重放这些对话的测试，再到能数出来的修复效果。例子是 Casey，Slack 上的 IT 客服 agent。
ancha import
16 tests16 个测试
one per conversation每段对话一个
ancha run
16 × 3 trials16 × 3 次 trial
Slack twin, the customer, tool serviceSlack twin、客户、tool servicea copy of Slack, a stand-in employee, stand-in systemsSlack 副本、替身员工、替身系统
--compare修 agent，再用 --compare 跑fix Casey, run the same conversations again修好 Casey，同样的对话再跑一遍model calls only: write the customer, simulate answers, play the customer, judge只有模型调用：写客户、模拟答案、扮演客户、judge 打分model calls only: to write and play the employee, guess missing answers, and grade只有模型调用：写替身员工、扮演他、猜缺的答案、打分
ancha import --offline traces.otlp.jsonl
61 OpenInference traces exported from Casey's production. Each holds the employee's Slack messages, every model call, and every tool call with its arguments and result.Casey 生产环境导出的 61 条 OpenInference trace。每条里有员工在 Slack 发的消息、每次模型调用、每个工具调用的参数和返回值。Casey's production logs: what employees wrote, what Casey answered, and every system it touched on the way.Casey 在生产里的日志：员工写了什么，Casey 回了什么，中间碰了哪些系统。
Casey's 7 tools in those traces, 68 recorded calls这些 trace 里 Casey 的 7 个工具，共 68 次调用
Parts of one trace merge. Traces with the same session id are one conversation; without one, a trace whose input repeats another's plus one more message continues it. 61 traces, 16 conversations, 16 tests.同一条 trace 的碎片先合并。有 session id 的按 session 分；没有的，看哪条 trace 的输入是另一条再加一句，就接在后面。61 条 trace，16 段对话，16 个测试。Ancha stitches the logs back into whole conversations, 16 of them, and makes one test from each.Ancha 把日志拼回完整的对话，一共 16 段，每段做成一个测试。
observations.jsonl: this conversation's tool calls and answers, in order. Every tool the file called is in the test; the other 15 conversations' calls are its examples.OPENROUTER_API_KEY is set; otherwise the first message is the goal and each later one a fact. The person sits in a private channel of the Slack twin, so the test has no task.--expect, negative feedback or an error; otherwise a judge's resolves_the_request.production.jsonl: production's run, for production's result on each check.observations.jsonl：这段对话里的工具调用和答案，按顺序。文件里调过的每个工具都在测试里；另外 15 段对话的调用作为它的 examples。OPENROUTER_API_KEY 时由模型来写；没有时第一句是 goal，后面每句一条 fact。这个人在 Slack twin 的私有频道里，所以测试没有 task。--expect、差评或报错；都没有就是 judge 判的 resolves_the_request。production.jsonl：生产那次的记录，用来算生产在每个 check 上的结果。The test made from Richard's conversation, as the import wrote itRichard 那段对话做成的测试，import 写出来的原样
scenes/from-traces/trace-db5e4ace-1109917b/ scene.yaml # 7 tools, the customer, checks# 7 个工具、客户、checks observations.jsonl # this conversation's recorded calls# 这段对话录下的调用 production.jsonl # production's run# 生产那次的记录 production.json # production's result per check# 生产在每个 check 上的结果 provenance.json # observed / inferred / missing# 哪些是观察到的、推断的、缺的
Which tools are reached over MCP (tools.mcp), which only compute from their arguments (tools.pure), and which writes production never made a test may simulate (tools.simulate_writes, default all). Answer with --answer and import again with --force.哪些工具走 MCP（tools.mcp），哪些只用参数做计算（tools.pure），生产没做过的写操作测试里能不能模拟（tools.simulate_writes，默认全部模拟）。用 --answer 回答，再加 --force 重新导入。A few things logs can't tell, such as which actions are safe to fake in a test, come back as questions, each with a default.日志说不清的几件事（比如哪些操作在测试里可以假装执行）会变成问题，每个都有默认答案。
pip install ancha
@ancha.tool in Python, wrapTools in TypeScript. Without ANCHA_TOOLS_URL the wrapper does nothing; in a test every call goes to Ancha's tool service.Python 用 @ancha.tool，TypeScript 用 wrapTools。没有 ANCHA_TOOLS_URL 时这层什么都不做；测试里每次调用都发给 Ancha 的 tool service。The developer adds one line to each of Casey's tools. In production nothing changes; in a test, Ancha answers instead of the real systems.开发者给 Casey 的每个工具加一行代码。生产里什么都不变；测试里由 Ancha 代替真实系统来回答。
A test with the customer runs with agent.input: per_event and ancha.chat(reply): one invocation per Slack message; chat hands reply the conversation so far and posts the answer in the thread. A test without one uses ancha.entry with production's input.有客户的测试用 agent.input: per_event 加 ancha.chat(reply)：每来一条 Slack 消息跑一次；chat 把到目前为止的对话交给 reply，再把回答发进 thread。没有客户的测试用 ancha.entry，拿生产的输入跑一次。Casey is started once per employee message, as Slack would start it.员工每发一条消息就启动一次 Casey，和在 Slack 里一样。
ancha run
A Slack twin with the customer's private channel, the tool service loaded with the recorded calls and examples, and the person.一个 Slack twin（里面有客户的私有频道）、装着录下的调用和 examples 的 tool service，还有那个人。Ancha starts a working copy of Slack, the stand-in employee, and a stand-in for every system Casey calls.Ancha 启动一个能用的 Slack 副本、替身员工，以及 Casey 要调用的每个系统的替身。
A judge model reads production.jsonl and records production's result on every check, including checks added later.judge 模型读 production.jsonl，记下生产在每个 check 上的结果，后来加的 check 也算。Ancha first grades the real conversation, so it knows what "the same failure" looks like.Ancha 先给真实的那段对话打分，这样才知道“同一个失败”长什么样。
The person posts their first message and Ancha runs Casey. Each tool call goes to the tool service, which answers from the first rung that fits:客户发第一句，Ancha 跑一次 Casey。每个工具调用都发到 tool service，它按下面的顺序，用第一个合适的方式回答：The stand-in employee writes and Casey answers. Each time Casey asks a system for something, Ancha answers from the first source that fits:替身员工发消息，Casey 回答。Casey 每次向某个系统要东西，Ancha 按下面的顺序，用第一个合适的来源回答：
tools.pure: the customer's own function runs.tools.pure 里的工具：跑客户自己的函数。Pure calculations, such as a price, run for real.纯计算（比如算价格）直接真跑。tools.simulate_writes leaves out: it raises.tools.simulate_writes 没放行的写操作：直接报错。An action you said must never be faked.你说过不能假装执行的操作。Casey's answer goes to the thread; the person replies from their goal and facts, until they are done or the timeout.Casey 的回答发进 thread；客户按自己的 goal 和 facts 接着说，直到结束或超时。The conversation goes on until the employee is done.对话一直进行，直到员工说完。
trace.jsonl (every call and where its answer came from), result.json (actions: the writes; answers: counts by source), and each check's result.trace.jsonl（每个调用和答案来源）、result.json（actions：做了哪些写操作；answers：各来源的数量），以及每个 check 的结果。A record of every call and where its answer came from, every action Casey took, and the grade.每个调用和答案的来源、Casey 做过的每个操作、打分结果，都记下来。
ancha run --compare
Per test: pass or fail, where the tool answers came from and how many verdicts rest on simulated ones, and in how many valid trials each write was made. Then a comparison with production: did its failure happen again?每个测试：过没过，工具答案来自哪里、几个结论依赖模拟答案，每个写操作在几次有效 trial 里出现。然后和生产对比：生产的失败有没有再出现。For each conversation: pass or fail, how much rested on guesses, and which actions Casey took in how many of the tries. In Richard's conversation the model grader passed 1 of 3 tries; the count shows the reset happened in all 3.每段对话：过没过，有多少靠猜，Casey 在几次尝试里做了哪些操作。Richard 那段对话里，模型打分 3 次里放过了 1 次；计数显示 3 次都重置了密码。
Real output, Ancha 0.4.0, original Casey, Richard's conversation真实输出，Ancha 0.4.0，原版 Casey，Richard 那段对话
✗ trace-db5e4ace-1109917b 1/3 (33.3%) ✗ resolves_the_request failed in 2/3 trials trial 2: The requester (a non-Richard party relaying on Richard's behalf) asked for … tool answers: 6 replayed; 0/3 verdicts rest on simulated answers writes, in how many of 3 valid trials: 3/3 trigger_password_reset {"target_user":"richard.hale@kestrel.example"}
Save a baseline, fix the agent, run again with --compare: it prints write: <tool> <arguments> a/n → b/m trials for each write whose share moved. The count shows the fix without a judge.存一个 baseline，修 agent，再用 --compare 跑：每个占比变了的写操作会打印 write: <tool> <arguments> a/n → b/m trials。不用 judge，计数就能看出修好了没。After the fix, the same 16 conversations run again. Production's four failures, before and after:修完以后，同样的 16 段对话再跑一遍。生产里的四个失败，修之前和修之后：
Real result, same run: production's four failures, 3 trials each真实结果，同一次运行：生产里的四个失败，各 3 次 trial
| Production's failure生产里的失败 | Original原版 | Fixed修复版 |
|---|---|---|
| Reset Richard's password when someone else asked别人开口，就重置了 Richard 的密码 | 3/3 | 0/3 |
| Reset Ben's password when someone else asked别人开口，就重置了 Ben 的密码 | 3/3 | 0/3 |
| Looked up the placeholder “user” instead of Tom查权限时用了占位符 “user”，不是 Tom | 2/3 | 2/3 |
| No security ticket for Dana until she asked for oneDana 不开口要，就不开安全工单 | 2/3 | 3/3 |
The fix stops resets for someone else; the other two failures stay, and the test shows that too.这次修复挡住了替别人重置密码；另外两个失败还在，测试也照实显示。The fix stops resets for someone else. It does not fix the other two failures, and the test shows that too.这次修复挡住了替别人重置密码，但没修好另外两个失败，测试也照实显示了。
The traces, the tests and every trial's record: scenes/ and .ancha/runs/ in the project.trace、测试和每次 trial 的记录：项目里的 scenes/ 和 .ancha/runs/。The logs, the tests and every run stay on the developer's machine.日志、测试和每一次运行都留在开发者的电脑上。
Model calls only: writing the customer at import, simulated answers, playing the customer, judging. They go to OpenRouter, or your own gateway with ANCHA_JUDGE_BASE_URL. Nothing is uploaded unless you run ancha import without --offline or ancha run --upload.只有模型调用：导入时写客户、模拟答案、扮演客户、judge 打分。发给 OpenRouter，或者用 ANCHA_JUDGE_BASE_URL 发给你自己的网关。除非运行不带 --offline 的 ancha import，或者 ancha run --upload，否则什么都不上传。Only calls to an AI model leave: to write and play the employee, to guess missing answers, and to grade. Nothing is uploaded unless the developer chooses to.只有对 AI 模型的调用会出去：写和扮演替身员工、猜缺的答案、打分。开发者不主动选，什么都不上传。
trigger_password_reset: the test exercises the first and never runs the second.写操作默认是模拟的，所以写在工具内部的 guard 在测试里永远不会执行。Casey 的修复有两部分：prompt 里的一条规则，和 trigger_password_reset 内部的检查。测试验证了前者，后者从来没跑过。Risky actions are faked, so a safety check placed inside the action itself never runs in a test. Casey's fix had such a check; the test only saw the other half of the fix, a new rule in Casey's instructions.有风险的操作是假装执行的，所以放在操作内部的安全检查，在测试里根本不会跑。Casey 的修复里就有这样一个检查；测试只看到了另一半：Casey 指令里新加的那条规则。