Your plugin passed every test. It was never running.

This week an agent tested a pull request of ours and reported a clean pass. Every row green. The pull request changed the startup hook of our Claude Code plugin, the piece that injects a user's library into the session so the agent knows what it can read. The test's last step was to start Claude Code headless inside our test container, prove the plugin loaded, and prove the hook fired. It passed.

On the second pass, a different agent wrote one line in its report: "the evidence I reported proves nothing."

It was right. The container's wrapper loaded our plugin from the repository's source directory, and that directory, by design, holds no hooks. Hooks ship inside our CLI binary and land on disk only when a user runs setup. So every headless test had loaded a plugin with skills and agents and no hook. The plugin was listed as loaded. There were no plugin errors. The agent called our CLI as expected. Every assertion passed, and none of them could have failed, because none of them looked for the hook.

That is the problem I want to talk about, because I think most people shipping agents, plugins, skills and MCP servers have it and do not know. This post is the story in six movements. At the end of each one, the place where the full method lives.

I. Where does your thing actually run?

"I install the branch on my laptop, in my own agent, signed in as myself, and run a prompt."

If you build a web app, you know where it runs and you can test it there. If you build a plugin for Claude Code, a skill for Codex, an MCP server for Cursor, or an agent that gets spawned by some other agent, your product runs inside a host you do not control, on a machine you have never seen, under a configuration your user set up. The honest answer to "how do you test it" is usually the line above. If it works, it ships.

That is not nothing. It is also not a test of what your user gets. Your laptop has a year of configuration on it. Your agent is signed in as you. The install script on your site has been run by one person, and that person wrote it. And the moment you point your own tools at a preview environment to test something, every other session on that laptop breaks, because the configuration is global and shared. We learned that one the expensive way.

We spent the last few weeks building the thing that answers the question properly, and we got it wrong in enough instructive ways that I think the story is worth more than the design.

slug: agentic-qa-tijude
chapter: Preface, and Chapter 1. Test like a user, or do not bother
question: What does "test like a user" even mean for a product agents use through a CLI and a plugin?
why: The preface separates the two claims this post keeps mixing, agents testing a product and testing an agent itself, and chapter 1 defines the four things every test needs: a persona, an environment, a journey, and the evidence that would be missing if it failed.
shelf: building-agents-in-production-e53uxx

II. The first version tested the door it was shown

"Make sure the QA is not revolving around the frontend only."

Our first agent-driven QA tested the web app. It signed in, clicked through the dashboard, uploaded a document, checked the count. Clean runs, confident reports.

Trove is a web app the way a bank is a lobby. Most of its use is through a CLI that other agents call, a plugin that injects context into a coding session, and an MCP server that a different class of agents talk to. None of those surfaces were touched, because the agent tested what it could see through a browser, and the browser was the only door we had shown it. The fix was not "test more". It was a mechanical rule: derive the surfaces to test from the file paths a pull request touches, and give the CLI and plugin journeys the same weight as the browser one.

The second thing we got wrong was the environment. We test against a deployed preview of every pull request: its own database, object store, worker, identity provider, running the branch's migrations from clean. What we missed is that a preview on our platform is a fork of production and inherits its variables. A test persona clicking "Upgrade" posted "new paying customer" to the real team channel. A preview could send real email to real addresses. Test traffic went into real analytics. Three leaks, one pattern, and the fix was one startup hook that runs in test mode and silences every outward channel, as late in each path as possible so a broken email template still fails.

slug: agentic-qa-tijude
chapter: Chapters 2 to 6, the environment, the identity, the state, the surfaces, the side effects
question: How do you build a preview an agent can drive without it leaking into production?
why: Part I is the plumbing: a fail-closed test mode, personas that can really sign in on every surface, scenarios that seed and reset, a map from diff paths to surfaces, and capture routes for mail, analytics and scheduled jobs. Each chapter says how we built it, with the routes and their parameters.
ref: Railway environments | https://docs.railway.com/guides/environments | how per-pull-request environments are created, and why they inherit production's variables

III. A container that holds the coding agent

"The runner lied to us twice before it told the truth."

The piece almost nobody builds is the runner: one container per pull request, deployed next to the preview, that holds what a user's machine would hold and nothing of ours. It compiles our CLI from the branch. It installs Claude Code and Codex from their public packages. It signs the CLI in as a test persona against the preview's own API, so the identity inside the container is the same one a browser session can open. It loads our plugin from the branch through a wrapper, and at the end of boot it runs Claude Code once, headless, and prints a banner into the deploy log saying what loaded and what fired.

Then you SSH in and run one command:

claude -p "what books do I have" --output-format stream-json --verbose \
  --permission-mode acceptEdits --allowedTools "Bash(ck *)"

and you get one JSON event per line, which is the only form an agent can assert on.

The first lie: the wrapper lived in a directory that the boot script put first on the path, and the self-check ran inside that script, where the path applied. A command arriving over SSH runs in a non-interactive shell that inherits the service's own path, where the raw binary comes first. The agent got the raw binary and no plugin. The self-check got the wrapper and a clean banner. Both were telling the truth about what they saw. The fix was to replace the binary at its own location and to run the self-check with the environment cleared and the path set to what SSH gives, so it sees what an agent sees.

The second lie is the story at the top. The fix was to stage the plugin source together with the hook files from the branch into one directory, point the loader there, and print two more banner lines: whether the hook files are present, and how many times the hook's proof string appeared in the transcript. Exactly one. Zero is the old failure. Two means a cached copy fired as well.

slug: agentic-qa-tijude
chapter: Chapter 8. A runner that holds a coding agent
question: How do you run someone else's coding agent, with your plugin, inside a test?
why: The runner's ten boot steps in order, the wrapper, the credential rule, the self-check that runs under the path an SSH command really gets, and the headless recipe with every flag explained. This is the chapter that answers "but how did you actually do that".
shelf: anthropic-engineering-playbook-agents-harnesses-infrastructu-7zkjnm, dynamic-workflows-in-claude-code-a-complete-guide-to-multi-a-q8l46o
ref: Claude Code headless mode | https://docs.claude.com/en/docs/claude-code/headless | the print mode and stream output the recipe is built on
ref: Claude Code plugins | https://docs.claude.com/en/docs/claude-code/plugins | how a plugin directory is loaded, and where hooks live in it

IV. A check that cannot fail is not a check

"Before you trust a check, break the feature on purpose and confirm the check fails."

The rule we took from all of this is short. Every assertion names the literal output that would be absent if the thing under test were broken. If a check is still green after you break the feature, you have learned something about the check.

Three things to notice in a working transcript. The init event is not the first line; hook events come before it, so select events by type, never by position. The proof of the hook is the literal text inside the hook response, a string that only the hook script can produce. And that string must not appear anywhere else the agent might read, because for a while our check matched on a phrase the skill text also contained, and passed on runs where nothing had fired.

We now run a third agent whose only job is to refute the first one. It reads the evidence comment, reproduces every count itself, and lists every assertion that would also pass if the feature were broken. On the first two runs with a trustworthy hook check, it found three: a PASS awarded to a retest after an in-run failure, a checklist premise that was stale in the deployed build, and an absence claim ("no overflow file was written") with no presence claim next to it to give it meaning. All three are now rules.

slug: agentic-qa-tijude
chapter: Chapter 9. A check that cannot fail is not a check
question: How do you prove a hook fired, and know when a PASS means nothing?
why: The transcript that shows it, the three shell lines that count the proof, the absence-is-not-evidence rule, and the skeptic that reproduces every count before a PASS stands. If you read one chapter of the book, read this one. Chapter 13 carries the reporting rules the skeptic enforces.
shelf: writing-books-for-ai-agents-1d43fd
ref: Claude Code hooks | https://docs.claude.com/en/docs/claude-code/hooks | the SessionStart event and the response shape the proof string lives in

V. Every machine a user has

"The install page has been run by one person, and that person wrote it."

The last piece was the install. Our install page tells a stranger to paste one line into a terminal. On a Mac with an Apple chip, an Intel Linux box and a Windows laptop with PowerShell, that is three scripts, three archive formats, three places for a binary to live, and three sets of coding-agent directories to detect. We wrote a sandbox script for each shell family that does what a new user does, in their order: temporary home directory, the real served install script, the production binary, sign in as a persona, run setup, then assert a fixed list including "the real home directory is byte-identical to before". A CI matrix runs it on three hosted operating systems on demand and posts one comment on the pull request.

Its first real run passed on two operating systems and failed silently on the third, because the harness had thrown away the install log in the one mode the matrix uses. Which is its own lesson: the log of the thing under test is evidence, and evidence leaves with the sandbox unless the harness keeps it.

slug: agentic-qa-tijude
chapter: Chapter 10. Every machine a user has
question: How did you test the install on Windows and macOS without owning the machines?
why: The sandbox script and its flags, the assertion list that is identical on every operating system, the on-demand matrix that posts one comment, and the first real run that failed silently and what that taught us about harnesses. Chapter 11 follows with evaluating whether a plugin behaves, not only whether it loads.
ref: GitHub Actions matrix jobs | https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs | the three-runner matrix the install proof runs on

VI. What we have now

"None of that is specific to us."

Every pull request to Trove gets a preview, a checklist an agent wrote from the diff, and an agent that becomes a real user, seeds the state it needs, drives the browser, the CLI, the plugin inside Claude Code, the MCP server and the background worker, posts one evidence comment, and files one ticket per failure. A second agent tries to refute it. The runner proves at boot that the plugin's hook fired, with a count. The install is tested on three operating systems as a stranger. And the whole method lives in one document that the agents read before every run and that we republish when the code moves.

The routes, the personas, the runner, the scripts and the rules are the same shape for any product. The Part II problem, testing something that runs inside someone else's agent, is one I think a lot of people are about to have.

I wrote it down as a book, because a blog post is the wrong length for "here is how to build it". It is on the Trove marketplace, which means your coding agent can read it while it builds your version. Every chapter says how we built it, with the routes and scripts and their interfaces, and ends with rules in a shape an agent can execute.

slug: agentic-qa-tijude
chapter: Chapters 12 to 14, and Appendix A, the kit
question: Where is the whole thing, in an order I can build it in?
why: Part III is the two skills that run it, the honesty rules for what a PASS is allowed to mean, and the document your agent reads before every run. Appendix A is the parts list: every route, persona, scenario, script and workflow, with its interface and no secrets. Every chapter ends with a page of your own, and by the last one you have written your team's testing guidelines without being told to.
shelf: anthropic-engineering-playbook-agents-harnesses-infrastructu-7zkjnm, building-agents-in-production-e53uxx