The problem with AI code review is not that it misses bugs

A code scanner told me line 625 would raise a TypeError. It was specific. It quoted the line. The explanation was fluent enough that I opened the file expecting to find a bug.

Three lines above it sat except TypeError:, catching exactly the case it had described. The code was not broken. It had been written that way on purpose, by someone who had thought about the problem harder than the scanner had.

That took nine minutes to establish. The nine minutes are the whole problem.

Precision matters more than recall, and it is not close#

Every tool that reads your code without being asked makes a trade. Look harder and you find more real bugs and also more imaginary ones. Look less hard and you find fewer of both.

Most people write about that trade as though the two errors cost the same. They do not.

Miss a bug and you have one bug, which you already had. The tool changed nothing. Invent a bug and you have taken something worse than nothing: you have spent the reader's attention, and you have taught them that opening your panel is a coin flip. Do it twice in a row on a Tuesday and they stop opening it at all, and every real finding after that is worth zero because nobody is looking.

Note: This asymmetry only holds for tools that interrupt. If I go looking for an answer, I am already paying attention and a wrong answer costs me one search. A scanner that speaks up on its own is spending attention I did not offer it.

So the interesting engineering problem in this category is not "find more". It is "say less, and be right about what is left".

Asking the model to check its own work does not work#

The obvious first move is a second pass. Show the model its own finding and ask whether it holds up.

This fails, and it fails in a way that is worse than not trying. Same weights, same context, same reading of the code, so it reaches the same conclusion. The only thing that changed is that the finding now carries a confidence score derived from a model agreeing with itself. You have not verified anything. You have laundered a guess into a number.

What you need is a check that runs somewhere the model does not: against the actual file, mechanically, with an outcome the model has no vote in.

The ladder#

Every finding our scanner produces is put through five checks against your real source before it reaches your screen. A finding that fails one is dropped, not softened.

V0. The code it quoted actually exists#

A finding names an anchor, the fragment of code it is talking about. If that fragment is not in the file, the model wrote it from memory of code like yours rather than from the code in front of it, and everything downstream of it is fiction.

The matching is whitespace-blind on purpose:

def normalise(text: str) -> str:
    """Strip ALL whitespace. A finding that is right about the code and wrong
    about the indentation is still right about the code."""
    return "".join(text.split())

There is a floor. An anchor under 8 characters carries no evidence, because sync = 0 appears in a thousand files and matching it proves nothing. Short anchors do not fail the rung, they skip it, and the finding is marked as unverified rather than confirmed.

V1. The file parses#

If your file has a syntax error, the interesting fact about that file is the syntax error. A report about a possible null dereference in code that does not compile is describing a program that does not exist.

V2a. The check it says is missing is actually missing#

This is the most common shape of real finding: "there is no null check before this access". It is also mechanically decidable. Look for the check. If it is there, the finding is refuted and dropped.

V2b. The trigger is reachable#

A conditional bug needs a path that reaches the condition. If the finding says "when opts.retries is negative this loops forever" and nothing in the file can produce a negative value there, the finding is describing a program that could exist rather than the one you wrote.

V3. The fix is an edit that applies and still parses#

This is the newest rung and the one that changed the product most.

A fix arrives as an exact old string and new string. We locate the old string in your real file, apply the replacement to a copy in memory, and re-parse the result. If it does not parse, the edit is wrong, and no button appears.

def verify_fix(finding, lines, path, parses) -> None:
    # 1 shape      the edit has both halves and they differ
    # 2 substance  the anchor clears MIN_ANCHOR_CHARS
    # 3 no-op      new is not old with different whitespace
    # 4 located    the old text is in the file, on whole lines
    # 5 unambiguous exactly one match, or line_hint breaks the tie
    # 6 applied    replace, re-parse, and it still compiles

Rung 5 is there because of a specific failure. If the anchor appears three times in the file, we do not know which one the model meant, and picking is a coin flip that rewrites a line the user did not ask about. Ambiguity is not a fix.

Fix: The result is that the Fix button on a finding is a claim we have tested. It means: this edit was located in your file, applied, and the file still parsed afterwards. Not "the model is confident".

The rule that makes it usable#

Every guard is a refusal, and every refusal has a victim. Build a ladder like this carelessly and you get a scanner that is silent, has perfect precision, and is worthless.

So there is one rule above all five rungs: only positive evidence refutes.

A rung can say "I found the null check you claim is missing, and here it is on line 44." That is evidence, and the finding is dropped. A rung cannot say "I could not find the null check, so I assume you are right." Every ambiguity, every unparseable file, every anchor too short to test, resolves to keep and label, never to drop.

The failure mode this prevents is the expensive one. A scanner that occasionally shows you something weak costs you a shrug. A scanner that silently ate the one real bug in your file, because a check it could not run defaulted to guilty, has failed at its only job and left no trace that it did.

A bad fix never refutes its finding#

The same rule applies one level down, and it took us a while to see it.

When V3 rejects an edit, the temptation is to drop the whole finding. If we cannot even write a working patch, how confident can we be about the bug?

That is the wrong inference. The bug can be entirely real while our proposed patch is wrong, and those two facts are independent. Dropping a genuine defect because we wrote a bad fix for it hides the defect and protects nobody.

So a rejected fix strips the edit and keeps the finding. You see the problem, the line, and the reasoning, with no button. That is an honest state, and it happens often enough that having a name for it matters.

What this cannot catch#

Which brings us back to line 625.

That finding passed every rung. The anchor existed. The file parsed. The condition it described was real: that function genuinely can raise a TypeError on that input. Nothing about it was invented, which means there is nothing to disprove.

It was wrong because it misjudged intent. The except TypeError three lines up is not an oversight, it is the design. The author knew about that case and decided what should happen. The scanner read the code correctly and drew a conclusion a human maintainer would not.

Warning: A bigger model does not fix this. The failure is not a hallucination and it is not a reasoning error. It is a disagreement about what the code is for, and more parameters do not settle that.

What might fix it is another rung, and it is mechanical in the same way the others are:

the finding claims:  this raises TypeError
the enclosing try:   except TypeError
→ refuted. The code already handles the case you named.

That is checkable. We have not built it, and the reason is the next section.

Why there is no percentage in this post#

You have read a lot of posts in this category that end with a number. Ours does not, and the reason is worth writing down.

We ran the scanner across nine open source repositories nobody here has touched, and hand-reviewed every finding it produced, with a second independent opinion on each. After separating out the defects we had planted ourselves (which are true by construction and would score the same bug twice, both times in our favour), we were left with five findings about code we did not write. Three were right. Two were not.

Two out of five reads as 40%. Flip one and it reads 20%. The 95% confidence interval on two out of five runs from roughly 7% to 81%, which is a range containing both "excellent" and "unusable". Separating 10% from 40% with any confidence needs around thirty findings, and we have five.

So the honest thing to say is: the machinery is real, you can read what it does above, and the number that would tell you how well it works does not exist yet. We are collecting it from real usage rather than buying it from another corpus run, which also means it will be measured on the code people actually write instead of on requests and express.

Designing that extra rung against two examples would produce a rule about TypeError handlers, not a rule about misjudged intent. It waits for the data.

What it means in practice#

The ladder runs on every proactive scan, including the free tier. When a scan finishes, the panel tells you how many files it read and how many findings were dropped before you saw them.

That second number is the one I would want if I were evaluating this. Anyone can show you what their tool found. The interesting question is what it decided not to tell you, and whether it can explain why.

3 thing(s) worth a look
From 18 file(s), 2 of them verified against your source.
 
2 finding(s) were dropped: they did not hold up against your source.

DebugAI is free for 10 debugs a day with no card. The scan is included.