Two autonomous pentest platforms tested the same application, in the same study, with the same source code. The reviewers who validated every finding by hand could match only three of them to each other, and said they expected the real number was higher.
When Doyensec compared two autonomous pentest platforms in spring 2026, they gave both the same open-source application and the same source code, then validated every finding by hand. The study is published as Comparing AI Application Security Testing Platforms: Aikido vs. XBOW.
Aikido confirmed seventeen findings. XBOW confirmed twenty-four. The totals look comparable. What the two platforms actually found does not.
Aikido confirmed seventeen findings. XBOW confirmed twenty-four. Totals in the same range, from two products given the same code on the same week.
Then the reviewers checked how much the two sets overlapped, and were careful about how they described what they found. They called the analysis out of scope, ran it at the end, and gave it what they called a quick review. Three findings appeared to be identical.
Three is a floor on obvious duplicates, not a measurement of overlap, and the study says so directly:
Because of this low rate of obvious duplication, we suspected there were more that would just require digging into them more deeply to match root causes.
They name a case they declined to count: both platforms reported missing or misconfigured security headers, and they did not treat those as the same finding because the specific headers each named only partly overlapped. So the true overlap is higher than three, and nobody has published what it is.
One pentest is a sample of your attack surface, not a map of it.
Why you cannot compare these tools by vulnerability class
The obvious next question is which kinds of bug each platform found. The study publishes a chart of exactly that, and it is tempting to read the gaps in it as blind spots. It does not survive contact with the study's own text.
The chart shows Aikido with zero findings in the security-misconfiguration class. The same study, describing what Aikido reported on the same application, lists "missing/misconfigured security headers". The chart shows XBOW with zero findings in insecure design. The same study lists "a logic error" among what XBOW reported.
Both platforms reported those bugs. They were filed under different headings.
A zero in a vulnerability-class chart can mean the tool did not find that kind of bug. It can equally mean somebody filed it somewhere else.
This is worth knowing beyond this one study, because class counts are what vendor comparison tables are usually built from. Unless a single person classified every finding from every tool by one written rule, the columns are not measuring the same thing, and the zeros are the least reliable cells in the table.
It is also why we are not putting our own class breakdown beside theirs. We ran the same application and we could produce that column, but we would be classifying our own findings by our own judgement and setting them next to someone else's judgement about somebody else's tools. The number would look comparable and would not be.
What does compare: severity, after someone checked
Severity survives, because the reviewers re-scored every finding from both platforms by hand and published the result. Ours are ours, validated against the running application rather than by a third party, and we would rather say that plainly than borrow someone else's audit.
| Confirmed findings, after review | Aikido | XBOW | SQUR |
|---|---|---|---|
| Critical | 1 | 0 | 1 |
| High | 2 | 3 | 2 |
| Medium | 6 | 7 | 6 |
| Low and informational | 8 | 14 | 8 |
| False positives | 2 | 1 | 0 |
| Severity right as reported | 76% | 71% | not re-scored by a third party |
The three converge where it matters and diverge in the tail. Critical plus High comes to three for each. The gap between seventeen findings and twenty-four is almost entirely low and informational, and seven of XBOW's mediums were re-scored downward by the reviewers.
What that means for the number in the sales deck
If a vendor tells you they found twenty-four issues, the useful follow-up is not whether that is a lot. It is what they did not look at.
The same study gives two reasons a headline count flatters everyone:
- Duplicates inflate it. Seven of XBOW's twenty-five findings on this application shared a single root cause. One fix closes all seven, and one line in your risk register covers them.
- Severity inflates it. Seven of XBOW's twenty-four confirmed findings were re-scored downward by the reviewers: five from Medium to Low, and two to informational. Six of the twenty-four ended up informational.
Neither is a scandal. Both are what happens when a total is reported without a root-cause pass and without an executed exploit behind each severity. The fix is on the vendor's side, and you can ask for it.
The cost nobody puts in the comparison table
The study also recorded how long it took to get each platform running.
Aikido took under twenty minutes to set up and start both scans, self-serve. The Fider scan itself then ran for eight hours and forty minutes, and the report was delivered as soon as it finished.
XBOW needed a sales representative and a signed agreement before testing could begin, the run paused and crashed more than once, more than twenty-two support emails were exchanged, the engagement on this application took over a week, and the report arrived five days after testing finished.
Put that next to the coverage picture and the real problem appears. If a pentest takes eleven days to organise, you run it once a year. Once a year, against coverage that varies this much between competent tools, means your evidence is both a year old and partial. That is the gap an assessor finds in a DORA Article 24 file, and it is the gap an attacker has time to use.
Cadence beats a single count
Coverage variance is not something any single product fixes by being cleverer. It is a property of the problem: different tools take different paths through the same application, and so does every human pentester you have ever hired.
What you can control is how often you look, and how quickly a change gets tested.
- Time to first result decides whether security runs at the speed of your releases or at the speed of procurement.
- Re-test cadence decides whether your evidence describes the product you shipped this month or the one you shipped last year.
- Proof of exploit decides whether a finding survives review, and whether an assessor can reproduce it.
Where SQUR sits
We ran the same application, Fider v0.33.0, with the source code provided. It finished in under six hours and published seventeen findings, each validated against the running application, topped by an account takeover proven with a working exploit chain rather than described.
The part that matters more than the count: there was no sales call, no contract to sign before testing, and no wait for the report.
| Getting a pentest run | Aikido | XBOW | SQUR |
|---|---|---|---|
| Contract before testing | no | yes | no |
| Time to set up and start | under 20 min | over a week | 10 min |
| Report once testing ends | immediate | 5 days later | immediate |
| Turnaround on this application | about 9 hours | more than 12 days | under 6 hours |
| Repeatable monthly | yes | no | yes, included |
| Data residency | not EU-specific | not EU-specific | EU, GCP Brussels |
Aikido and XBOW figures come from the study's setup timeline and its published findings file.
A pentest that starts in ten minutes and returns in twenty-four hours is what makes running one every month realistic instead of theoretical. On the Professional plan that is exactly what happens: a full pentest every month, fixes re-tested, and the certificate refreshed so the evidence stays as current as the product. €1,995 per pentest, with all data in the EU (GCP Brussels).
See what is exposed before you book anything: the free external scan takes 60 seconds.