The first time I heard someone claim their AI now wrote 80% of their tests, I asked the obvious question: 80% of what? After two decades building and leading test automation for consumer-scale platforms, my honest answer is that figure represents 80% of the typing, not 80% of the engineering. That remaining 20% is where the real work lives. Budgeting for a 2% leftover effort is a mistake—when the actual number is closer to 30%, that gap separates a pipeline that ships from one that silently accumulates a backlog of half-trusted features nobody can rely on.

This piece examines that gap. As part of an independent research project on LLM-augmented testing methodology, I built a six-stage agentic pipeline that takes a design from Figma and produces running tests in WebDriverIO, all connected end-to-end over the Model Context Protocol (MCP). It works. It has been useful. And the failures that surprised me were not the ones the hype cycle warns about.

How I wired a six-stage pipeline over one protocol

The pipeline runs six stages in sequence: design capture, requirements writer, ticket opener, code generator, test-case writer, and automation generator. Each stage is owned by a different agent, and every handoff crosses MCP. The end-to-end trace links a pull request back to a Jira ticket, a requirements section, and a Figma frame. Every artifact is stamped with the agent that produced it, the model it used, and the inputs it was given.

MCP is the boring middle that makes any of this work. The common analogy—that MCP is "USB-C for AI", one open protocol for any tool—is about 80% right. The part that matters is the 80%: I don't have to write a custom adapter for every system the agent talks to. One MCP server per tool, and every agent talks to all of them the same way. However, typed handoffs between agents are my own architecture, layered on top of MCP. Each agent writes a typed artifact the next agent reads. Each handoff is logged with provenance. When something goes wrong six stages in, I can replay the chain. Without that discipline, a multi-agent pipeline is a debugger's worst nightmare—you know the test plan is wrong, but you can't tell whether the mistake came from the Figma read, the requirements interpretation, or the ticket scaffolding. With it, I can point at exactly which stage went sideways and which inputs it was looking at when it did. The pattern lives in a public MIT-licensed reference implementation for anyone who wants to run it.

The sixteen-minute number is the marketing number. I ran the full chain end to end in about sixteen minutes on a synthetic net-new screen—Figma in, automation suite out. That repeated across my runs; it's not a demo trick. But sixteen minutes is the part of the story most fun to tell and least useful to learn from. It gets quoted in all-hands meetings. The hours that come after—when a human reviews each handoff—are where the work actually lives.

What actually broke in production-style runs

The failures that stalled my pipeline were rarely the ones I expected. I expected hallucinated APIs—and I got them: the agent confidently called endpoint names that sounded right but did not exist. I expected sparse-spec-in, sparse-spec-out—a Figma frame with no annotations produced a requirements doc with vague acceptance criteria, every time. I expected locator drift, the common UI-automation failure mode where a renamed component silently breaks an entire test suite. There is solid outside writing on non-determinism in tests covering this whole family of failure modes, and the agent inherited every one.

What I did not expect—and what kept the pipeline down longer than anything above—was the plumbing. The model backend timed out under load. It lost credentials silently and started returning empty strings, which the agent then read as confidence. A duplicate consumer on a shared long-poll API endpoint produced an HTTP 409 conflict that broke delivery without throwing anything visible. One unguarded exception inside one agent aborted a whole shared scheduler run and took the other agents in the registry down with it. The single worst incident cost me three hours to find: an environment variable had silently rotated overnight; every agent in the fleet was returning structurally valid but semantically empty requirements docs; the downstream stages were dutifully generating tests against nothing.

None of these are model bugs. They are infrastructure. The agent literature, which I reviewed when I started this work, mostly does not talk about them. The fix was not better prompts—it was adding circuit-breaker review checkpoints between stages and what I now call the four-guard discipline. These four small guards are non-negotiable on any unattended agentic pipeline:

• Bulkhead pattern: An unhandled exception inside one agent can no longer abort the shared run; the offending agent fails fast with a structured error and the others keep going.
• Pure-data fallback: A model timeout produces a deterministic output explicitly marked as degraded mode, rather than an empty string the next stage will misread as confidence.
• Single-owner lease: Every shared external endpoint has a single owner, preventing duplicate-consumer conflicts like the one that ate a Sunday afternoon.
• Synthetic canary: A one-line test every agent must pass before any real work begins, so a credentials rotation or silent backend failure trips an alert before downstream stages generate artifacts against garbage.

None of these guards is novel. They are textbook stability patterns applied at a new boundary: the seam between the LLM agent and the rest of the system—a seam most existing agent literature still treats as a solved problem.

The 20% you don't see, and when not to do this

Here is the part the demo videos leave out. Even when the pipeline works, the human time per stage does not go to zero. Human review time per ticket across the five pipeline stages breaks down like this: code review takes 60–180 minutes; automation review and the flaky-fix loop takes 30–90 minutes; ticket architecture and sequencing takes 30–60 minutes; test data and environment setup takes 15–30 minutes; and requirements review takes 20–30 minutes. Net result: the human still spends 20–30% of the original effort, almost all of it reviewing rather than creating. The pipeline saves 70–80%, not 98%. The trap is budgeting for the 2% you don't save.

When does this kind of pipeline make sense? In my experience, when the Figma is richly annotated and acceptance criteria are clear up front; when there is review capacity to absorb the work the pipeline shifts onto humans; when the stack is well represented in the training data; and when the feature is net-new rather than a deep edit of legacy code. When does it not? When the design lives on a whiteboard; when the integration touches old code with hidden contracts; when the path is regulated or safety-critical; when there is no senior reviewer who can hold the line; when the work is exploratory and writing the spec is the actual point of the exercise.

Teams I have seen succeed with agentic pipelines budget for the rework explicitly, staff the review queue, and treat the saved hours as capacity for harder problems rather than headcount they can release. Teams I have seen struggle did the opposite: declared victory at the demo and quietly accumulated a backlog of half-trusted features the next quarter had to clean up. The right unit of measurement is not how much the pipeline generates—it is how much of what it generates a human still has to touch before you would ship it. Call it the 80/20 rework rule: measure the rework, not the generation. The teams that get the rework number right are the ones whose AI investments compound. The teams that stop counting at the headline percentage are the ones that own the cleanup six months later.

Source:InfoWorld News