How accurate are AI image detectors?

Ask a vendor and you'll hear a number like "99% accurate." Ask a researcher and you'll get a question back: on which images, at which threshold, against which generators, after how much compression? The honest answer is that an AI image detector doesn't have an accuracy — it has a performance curve, and where you sit on that curve depends on choices someone made and should be willing to disclose. We run a detection benchmark ourselves, so this article uses our own published numbers — including the unflattering ones — to show what those choices do.

Why "99% accurate" is usually a non-statement

A single accuracy percentage silently blends two very different errors:

• False positives — real photos wrongly called AI.
• False negatives — AI images that slip through.

You can trade one for the other freely by moving the decision threshold, which means a marketing number can be true and useless at the same time. Worse, "accuracy" depends on the mix of the test set. A detector tested on clean, full-resolution images from the same generators it was trained on will post a spectacular score that says almost nothing about the screenshot forwarded to you this morning.

The two errors also matter differently. If you check a thousand ordinary photos, of which a handful are AI, even a 5% false-positive rate produces far more false alarms than true catches. That's why the first question for any detector isn't "how many AI images does it catch?" but "how often does it cry wolf on real photos — and on what kind of real photos was that measured?"

What we measure, and what it shows

Our benchmark run (full_v4) scores 8,064 images across 8 conditions — 386,072 scored evaluations — because real images don't arrive clean. They arrive re-compressed by WhatsApp, screenshotted, downscaled, and re-shared. The hardest condition in the grid is a screenshot of a WhatsApp image, which is what actually lands in front of an analyst.

Three of our own numbers, side by side:

• Ranking quality (AUC) under the screenshot-of-WhatsApp condition: 0.933. Sounds excellent — but you can't act on a ranking; you need a threshold.
• AI images caught at a threshold capped at ≤5% false positives: roughly 55–64%, depending on condition. The honest operating number. Not 99%.
• AI images caught at the strict ≤1% cap (the evidentiary setting): roughly 36–45%. The price of alarms you can defend.

That distance between the impressive-sounding AUC and the operating detection rate is what we call the honesty gap. Every detector has it. Most quote the top number and stay quiet about the bottom two; we print the bottom two on the report itself.

For contrast: a well-known public detector run at its default 0.5 threshold flagged 27% of genuine photos as AI under a messenger re-encode in our benchmark — it reads compression artifacts as evidence of synthesis. Nothing is wrong with that model's science; everything is wrong with using it un-calibrated.

What moves the numbers

• Image quality. Compression, resizing, and screenshots destroy exactly the high-frequency statistical traces detectors read. A detector calibrated only on clean images degrades unpredictably on real-world ones — so the calibration has to be done per condition, and you should ask whether it was.
• Which generator made the image. Detectors are best on generators represented in their training and calibration data. Against a brand-new or unseen generator, detection rates drop — ours included. A detector that won't say this is not being straight with you.
• The threshold policy. The same model can be an aggressive tipster (more catches, more false alarms) or a conservative witness (fewer catches, alarms you can defend). Neither is "the accuracy" — they're operating points, and the choice should be yours, stated on the output. Ours issues verdicts at two: standard (≤5% false positives) and strict (≤1%).

Questions to ask any detector — including ours

1. What is the false-positive rate of a verdict, and on what real images was it measured? ("We don't publish that" is an answer too — a bad one.)
2. Was performance measured on compressed, resized, screenshotted images, or only clean ones?
3. Which generators were tested, and what happens on ones outside that list?
4. What does a negative result mean? (Correct answer: not much — a calibrated detector deliberately lets borderline images pass rather than inflate false alarms. "Not flagged" is never a certificate of authenticity.)
5. Is the calibration pinned to a named, versioned run, so the number on yesterday's report still means something after the model updates?

If you take one thing away: an accuracy claim without a false-positive rate, a test-condition description, and a generator list isn't a measurement — it's a slogan. The tell of a trustworthy detector isn't a high number; it's the willingness to show you the low ones.