Agentic QA

Two things are true at once, and this book keeps them apart.

The first is general. An agent can test your product the way a user does: sign in as a real account, upload a real file, wait for the real background job, read the result with the real command-line tool, and tell you what broke. It can do this on every pull request, against a deployed environment, without a human in the loop. The catch is not the agent. The catch is that most systems are built so that a check cannot fail. The agent runs the checklist, every row turns green, and nobody can say what the green proves. Part I of this book is about building the system so that a check can fail, visibly, and so the agent can tell you when its own evidence is worthless.

The second is rarer. If what you ship is itself a plugin, a skill or an agent that runs inside someone else's coding agent, you have a problem the first method does not solve. Your product's behaviour happens inside a host you do not control, on a machine you have never seen, under a configuration your user set up. Almost nobody tests that today. They test on their own machine, with their own logged-in agent, and call it done. Part II describes an environment that holds the coding agent, the branch build of your plugin and a real user identity, per pull request, and produces evidence about what your plugin actually did in there. It extends to fresh installs on every operating system your users have, and to evaluating whether the plugin behaves, not only whether it loads.

Part III is about making the practice stick: the two skills that run it, what a PASS is allowed to mean, and the document that holds all of it.