Run overnight multi-agent QA that files defect reports for morning review

A QA engineer's multi-agent testing system runs web and app tests overnight and files bugs; on one project it found 21 of the 30 defects later confirmed valid.

Done withCustom agent· Nov 6, 2026

What they did

The author, xulingfeng, built a multi-agent testing system at their company. Agents read product requirement documents and prototypes, write test cases that other agents review, then drive web browsers and mobile apps to run them. Beyond the written cases they are told to probe invalid inputs, boundaries, interrupted flows, expired sessions and permission edges inside a controlled test environment. Bugs go into the team's issue tracker as reports, pass a second review and human confirmation, then reach a developer. After a fix the agents rerun regression checks. The author queues runs before leaving work and triages the reports each morning. Agents also capture the API calls behind business flows and validate close to 300 APIs on a daily schedule. After each cycle the agents summarise lessons, and reviewed ones become reusable skills.

How it went

On one knowledge-base project, 47 defects were raised: 36 by the agents and 11 by people. Of the 30 confirmed valid, 21 came from the agents, about 70 percent. Early reports were dense walls of text and took several rounds of format changes before developers found them easy to act on. Some agent bugs were rejected because product and development had changed requirements in conversation without updating the PRD, so the agents were testing against a stale spec.

Worth knowing

About 15 of the agents' 36 reports were not valid, so the morning human triage stays essential. The author does not name the models or framework behind the agents. A single shared test environment caused clashes when several features were in development at once, and coverage of older Windows browsers was still missing.

Try it

With Custom agent· 6 parts to fill in

You are the QA agent for [product] in [test environment URL], logged in as [test account]. Read the requirements in [PRD location] and write test cases, then run them in the browser. After the written cases, explore edge cases: empty and overlong inputs, boundary values, repeated clicks, refreshes mid-flow, expired sessions and actions this account should not be allowed to do. Never touch [production URL]. For each problem, draft a defect report with steps to reproduce, expected versus actual behaviour and a screenshot, and save it to [folder] instead of the tracker. Stop when the run finishes and give me a summary list so I can review and confirm each report in the morning.

Square brackets are the parts to fill in. The prompt is ours; the story is theirs.

Where they said it

Related