Severity is the field your team plans from. It decides what gets fixed this sprint and what waits for the next quarter. So it is worth knowing how stable that number is.
In spring 2026 the security firm Doyensec ran Aikido's autonomous pentest against Fider, an open-source feedback portal written in Go, and then did something most comparisons skip: they validated every finding by hand and re-scored the severity wherever they disagreed. The full study is published as Comparing AI Application Security Testing Platforms: Aikido vs. XBOW.
Nineteen findings were reported. Seventeen were real. And four of those seventeen came back with a different severity than the platform had assigned.
One moved up, and it was the one that mattered
Three findings reported as Medium turned out to be Low. Ordinary drift, and the kind any team learns to live with.
The fourth went the other way. A finding reported as High was re-scored to Critical: mass assignment in the sign-in binder, letting an unauthenticated attacker take over any account by supplying their own verification key. The platform found the bug. It under-called the impact.
That is the pattern worth taking away. On the second application in the same study, eleven of thirty-two severities moved, every one of them downward. The reviewers' explanation was consistent: the platform did not know how sensitive the data behind the finding actually was.
A severity that moves under review is a severity that was never measured. It was inferred from the shape of the bug.
Why severity drifts, in plain terms
- Impact is contextual. An exposed identifier is informational in one application and the first step of account takeover in another. Nothing in the request tells you which.
- Detection stops early. Confirming that a parameter is reflected is not the same as confirming what an attacker can do with it. The first is cheap. The second is the work.
- Scores get assigned, not derived. A number attached without an executed exploit is a judgement about a class of bug, not a measurement of this one.
Two false positives in the same report fit the same pattern. One claimed host-header poisoning in the sign-in email flow, and the reviewers could not reproduce it: there was no place in the source where the host header reached the emailed link at all. The claim was plausible. It was never executed.
The question to ask any vendor
Does every finding come with a working exploit I can re-run?
It sounds narrow. It is not. A severity backed by an executed exploit does not move under review, because the impact was demonstrated rather than estimated. It also gives your engineers something better than a score: a reproduction they can run before the fix and again after it.
For a DORA Article 24 or NIS2 evidence file, the same property is what makes a finding auditable. An assessor can re-run it instead of taking your word for the rating.
We ran the same application
We put Fider v0.33.0 through SQUR, the same pinned image, with the source code provided. It finished in under six hours.
Seventeen findings were published, each one validated against the running application. The top of the list is the same class of bug the reviewers had to promote to Critical, and SQUR carried it as Critical from the start because it built the exploit chain: it read the untagged struct field in the Go source, supplied its own verification key, redeemed it, and confirmed the account's address had changed. It also ran the control, a key that was never submitted, and got the rejection it expected.
| Fider, confirmed findings | Aikido | SQUR |
|---|---|---|
| Critical | 1 | 1 |
| High | 2 | 2 |
| Medium | 6 | 6 |
| False positives | 2 | 0 |
| Severities that moved under review | 4 of 17 | Every finding carries a proof |
| Time to result | 8h 40m | under 6h |
Aikido's severities in that table are the reviewers' adjusted numbers, taken from the study's published findings file. Ours are ours, validated against the running application rather than by a third party, and we would rather say that plainly than borrow someone else's audit.
What to take into your next vendor call
- Ask for a proof-of-exploit per finding, not a score with a description.
- Ask what happens to the severity when your data sensitivity is the context, not a generic one.
- Ask how a finding gets re-tested after the fix, and whether the evidence file updates with it.
SQUR runs a full pentest in 24 hours and proves every confirmed finding with a working exploit, so the severity your team plans from is measured rather than assigned. Reports are structured for ISO 27001, SOC 2, DORA Article 24, NIS2 and CRA, and a shareable certificate follows every clean run. €1,995 per pentest, with all data in the EU (GCP Brussels).
See what is exposed before you book anything: the free external scan takes 60 seconds.