Automated testing now finds more potential vulnerabilities faster and cheaper than human testers can. What it still cannot do reliably is judge whether a finding is truly exploitable and what it means for the business. The strongest programs split the work: machines discover, people validate, and the handoff decides the value.
Security leaders evaluating AI-based testing often frame the choice as automation against people. A more useful question is which parts each does best, and where they should meet.
What Is Automated Discovery Better At?
Coverage is where automation wins clearly. A human tester in a two-week engagement can examine only a limited number of hosts, endpoints, and parameters, while an automated agent can test thousands of assets continuously and return to them whenever the environment changes. Modern AI pentesting goes beyond the signature matching of older scanners, chaining steps together, adapting payloads to responses and attempting exploitation rather than only flagging a version number.
The results are now publicly measurable. In June 2025, XBOW reported that its autonomous agent had reached the top of HackerOne’s US leaderboard after submitting nearly 1,060 vulnerability reports across live bug bounty programs. Earlier, DARPA’s 2016 Cyber Grand Challenge showed that fully automated systems could find and patch flaws in real time, and DARPA’s AI Cyber Challenge carried that work into large language model systems. Machines have speed, breadth, and stamina.
What Does Validation Actually Involve?
Validation sits between “this looks vulnerable” and “this matters.” It answers questions that require context the tool does not have, and in a typical engagement it covers several distinct checks:
- Confirming that the finding is exploitable in this environment, not just theoretically present.
- Ruling out duplicates of issues already reported, accepted, or fixed.
- Judging business impact, such as whether an exposed record is test data or customer payment details.
- Chaining low-severity findings into a realistic attack path that a scanner would rate separately.
- Writing remediation guidance a development team can act on without a second round of questions.
Each of these depends on judgment about the specific organization. A finding with a high CVSS score on an isolated test server may matter less than a medium-rated flaw in an internet-facing login flow, and only someone who understands the system can make that call with confidence.
What Does a False Positive Actually Cost?
Teams rarely budget for false positives, and they are expensive. Each one consumes triage time from the security team, developer time to investigate, and the credibility the next report needs. When a team learns that most findings from a tool are noise, it starts ignoring the tool, and genuine issues end up sitting in a queue.
XBOW’s own figures show the scale of the filtering problem even for a leading automated system. Of its roughly 1,060 HackerOne submissions, 208 were marked duplicate and 209 informative, and the company noted that its security team reviewed every finding before submission to comply with HackerOne’s policy on automated tools. The automated validators did much of the work, and people still checked the output.
The stakes of missing a real issue are rising. Verizon’s 2025 Data Breach Investigations Report found that exploitation of vulnerabilities was the initial access method in 20% of breaches, up 34% from the previous year. A program that floods the team with unverified findings makes those real exposures harder to see.
What Does a Hybrid Engagement Look Like?
A hybrid model assigns each stage to whichever side does it best, and designs the handoff between them rather than improvising it. Most mature programs follow a sequence close to this:
- Automated agents continuously discover across the full attack surface, including new assets as they appear.
- Automated validators filter out findings that cannot be reproduced, so people see a smaller and cleaner set.
- Human testers review the remaining findings, confirm exploitability, and assess business impact.
- Testers spend the time saved on work automation handles poorly, such as business logic flaws, authorization bypasses, and multi-step attack chains.
- Confirmed findings go to engineering with clear reproduction steps, and retests verify each fix.
The same principle applies well beyond security. Complex systems usually fail at the points where components interact, a theme covered in this site’s look at enterprise software testing, and automated checks catch the predictable failures while experienced testers find the ones nobody anticipated.
How Do Providers Split the Work Between AI and People?
Providers in this market balance automation and human expertise differently, and the right fit depends on what a buyer needs most. Horizon3.ai’s NodeZero and Pentera focus on autonomous testing and attack validation that internal teams run on demand, which suits organizations wanting frequent, repeatable coverage without scheduling people.
XBOW focuses on autonomous offensive testing with built-in automated validators. Cobalt offers pentest-as-a-service, delivered mainly by human testers on a credit model. HackerOne and Bugcrowd run crowdsourced programs where independent researchers submit findings and platform teams triage them.
Synack combines AI-powered testing with its vetted Synack Red Team of researchers, which suits buyers who want automated scale and human validation from one provider, though it may be more than a small team with a narrow scope needs.
Each model has trade-offs. Fully autonomous tools offer speed and lower cost per test but leave business-impact validation largely to the customer. Human-led services provide depth but cost more and run less often. Crowdsourced programs bring diverse skills but can generate high volumes that need managing. Buyers should ask any provider that validates findings before they reach the customer how they prove exploitability and what share of reported issues they later reject.
