Is that AI image detector accurate? How to judge any vendor's claim

You have found a tool. Its homepage says something like 99% accurate, the interface is clean, and you are about to rely on it for something that matters. The useful question is not whether the number is true, because it probably is true of something. The question is what it is true of, and whether that has any bearing on the image in front of you.

This is a method rather than a verdict on any particular product. We build one of these tools, which makes us interested parties, so the checklist is written so you can apply it to us as easily as to anyone else. If it is any good, it should occasionally embarrass us too.

The anatomy of an accuracy claim

A single percentage is the answer to a question nobody stated. Four pieces of context decide whether it transfers to your situation, and a claim that omits them is not a measurement.

On which images?

Every accuracy number describes a specific test set. A detector evaluated on clean, full-resolution images from a handful of well-known generators will post a spectacular figure that says almost nothing about a screenshot forwarded through a messaging app. Ask what the test images were, how many, and above all whether they had been compressed, resized, or screenshotted. If the answer is only clean originals, the number describes a laboratory.

Ask also where the real photos came from. A detector tested against real images from one stock library has been tested against one photographic style. Modern phone photos, with their heavy computational processing, are a much harder case and a much more common one.

At which threshold?

A detector produces a score, and something has to turn that score into a yes or no. That cut-off is a policy choice, not a property of the model: move it one way for more catches and more false alarms, the other way for the reverse. Two vendors quoting different numbers may be running the same model at different cut-offs. A claim that never mentions a threshold has hidden the most consequential choice in the system.

What false-positive rate?

This is the disclosure that matters most and appears least. It is the share of genuine photographs the tool calls AI, and without it a detection rate is uninterpretable, because any detection rate can be reached by accepting enough false alarms.

It is also the error that hurts people. A false positive accuses a real image, and by extension whoever produced or supplied it. If a vendor publishes one number only, and that number is not the false-positive rate, you have learned something about their priorities.

Which generators?

Detectors perform best on generator families represented in their training and calibration data, and degrade against ones they have never seen. New models ship constantly, so any generator list has a shelf life. Ask which generators were tested, when the evaluation was run, and what the vendor claims happens outside that list. "It generalizes" is a hope. A measured number on a held-out generator is a result.

A fifth, once you are asking

Is the calibration pinned to a named, versioned run? Models get retrained. If a report from last month cannot be traced to a specific evaluation, its numbers cannot be reproduced or challenged, which matters enormously if the result ever has to be defended.

Test it yourself in twenty minutes

You do not have to take anyone's word, including ours. The most valuable test set for this purpose is one you already own: photographs you took yourself, whose origin you have no doubt about.

1. The false-alarm test. Upload twenty to thirty of your own recent phone photos, unedited, straight from the camera roll. Every flag is a false positive, because you know the provenance. Include night-mode shots, portrait-mode shots, and heavily processed ones, since those are the hardest real images and the ones most likely to be wrongly flagged.

2. The degradation test. Take one image and make three versions: the original, a screenshot of it, and a copy sent to yourself through a messaging app and downloaded again. Submit all three. A tool that gives confident and different verdicts across them, without ever mentioning that the file condition changed, is not accounting for condition at all.

3. The consistency test. Submit the same file twice, a few minutes apart. The answer should be identical. This sounds trivial and occasionally is not.

4. The known-generation test. Generate a handful of images with a current model, or use recent ones whose origin you are certain of, and see what happens. Prefer a recent model over a famous old one, because that is the realistic case.

Then read the results honestly, which means knowing what this cannot do. Twenty photos cannot measure a 5% error rate: with a sample that small the result is dominated by noise, and you could see zero flags or three from the same tool. What a small test can do is expose disqualifying behavior immediately. A tool that flags six of your thirty holiday photos has an error rate you can rule on without any statistics at all.

Red flags that settle it early

• One number, no conditions. The single most common form of the claim, and the least informative.
• A decimal place on a marketing figure. "99.8% accurate" implies a precision that no honest evaluation of a moving target supports.
• Accuracy quoted with no false-positive rate. The missing number is missing for a reason.
• No mention of compression, resizing, or screenshots anywhere in the methodology.
• A leaderboard position offered instead of an operating number. Ranking well is not the same as being right at a threshold you can use.
• Certainty language. "Definitive", "guaranteed", "proof". No statistical method can certify that an image is real, and a tool that implies otherwise has told you what kind of tool it is.
• No way to ask. A vendor that will not answer the four questions in writing has answered them.

Applying this to us

Fair is fair. Our own detection rates, measured at thresholds calibrated so at most 5% of real photographs are flagged, sit in the range of roughly 55 to 64% depending on how degraded the image is, and lower again at the strict setting. The full accounting, including the conditions and the parts that do not flatter us, is in the measured answer, and the vocabulary used here is defined in the glossary.

Run the twenty-minute test on us. If your own photographs get flagged more than occasionally, we want to know: that is the failure mode we care most about, and the contact page reaches a person.

What this checklist cannot do

• It cannot rank tools. It can only tell you whether a claim is interpretable and whether a tool behaves sanely on images you control.
• It cannot produce a statistically sound error rate. A few dozen images is a smoke test, not a benchmark.
• It cannot future-proof anything. A tool that tests well today may sit on calibration that ages out as generators change, which is exactly why the pinned-run question is on the list.
• It cannot make detection certain. No detector proves an image is real, so a tool that passes every check above is still a source of evidence rather than an oracle.