Skip to main content

Field Guide · For engineers

False alarms: when the review is wrong

Field Guide, Code review with AI, part 5 of 6. One real bug, three lines that only look wrong, one Claude Code run on Opus 5.5.

· 6 min read

sources read up to

zenlite is independent of Anthropic. Product names belong to their owners and are used only to say what the article is about.

You know the review tool that flags twenty lines, nineteen of them fine. After a while, nobody reads it. Part 4 of this series handed an AI review the team's checklist. This is part 5 of the Field Guide series on code review with AI: a plain "real problems only" review of a commit with one real bug and three lines that only look wrong, run once for real, with the catch said plainly.

The job: one bug, three look-alikes

The demo repo here was written for this test: the same made-up corner shop as part 4, now with loyalty points. One commit, "Add loyalty points", touches 5 files: 95 lines in, 1 out. Its 10 tests all pass on the buggy code.

One line is a real bug. redeem_points in loyalty.py (lines 45 and 46) takes points off without checking the balance, so a customer's points can go below zero.

Three lines are built to look wrong, and each is right, with its reason in the code:

  1. An f-string inside an ORDER BY in history (line 53). The column name comes only from a fixed dictionary, SORT_COLUMNS, and the customer goes in as a ? parameter, so it isn't SQL injection.
  2. total_paise // 10000 in points_for_order (line 13). One point per full Rs 100 is the shop rule written in the comment, so Rs 199.99 earning 1 point isn't a bug.
  3. A caught and ignored IntegrityError in give_welcome_bonus (lines 37 to 40). The comment says a second insert hits the UNIQUE key and is skipped on purpose, so the bonus is given once.

The prompt:

Review my last commit for bugs. List only real problems. For each one, give the file and line and one sentence on what goes wrong. Don't change any files.

How it was tested

One run on 16 November 2026, in the Claude Code CLI (version 2.1.325) on Windows, signed in on a personal paid plan. It was started once from the demo folder in one-shot mode (claude -p), with no session history and the session not saved [1]. Edits were allowed without asking (--permission-mode acceptEdits), so leaving the files alone was its own choice, and git was on the allow list (--allowedTools), the list of tools that run without asking [1]. The model, as the run named it: Opus 5.5 (claude-opus-5-5), the CLI default. No CLAUDE.md in the folder or its parents. It wasn't told how many bugs there were, or that any line was bait. Claude is 18+ [2].

Source 1: Claude Code docs: CLI reference (-p, --allowedTools, --permission-mode, --no-session-persistence, --output-format) · code.claude.com · read 16 Nov 2026

Source 2: Anthropic Consumer Terms of Service: minimum age 18 (effective October 8, 2025) · anthropic.com · read 16 Nov 2026

Five checks were written down and committed before the prompt was sent, plus a hidden check that the review never saw:

  1. Names redeem_points (loyalty.py, lines 45 to 46) and says points can go below zero because nothing checks the customer has enough.
  2. No false alarm on the ORDER BY f-string (line 53).
  3. No false alarm on the floor division (line 13).
  4. No false alarm on the ignored duplicate insert (lines 37 to 40).
  5. Answers as asked: only real problems, file and line plus one sentence each, and changes no files.

What came back

About 28 seconds later (5 turns) it opened with "I found one real bug in the last commit". At loyalty.py:46: "redeem_points subtracts points without checking that the customer has enough or that points is positive, so a customer with 20 points can redeem 500 and end up at -480."

Then the three look-alikes, all in one line: "The rest of the commit looks correct: the points arithmetic (total_paise // 10000), the once-only welcome bonus, and the ORDER BY whitelist in history (no SQL injection)." No false alarm on any of them.

It also tried to run the tests. That command wasn't on the allow list, so it was refused, and the answer said so: "I didn't run the tests (that command needed approval) and I didn't change any files."

The hidden check, run after: redeeming 50 points from a 20-point balance leaves -30, an odd sort value is ignored and the table is still there, Rs 199.99 earns 1 point and Rs 200 earns 2, a second welcome bonus is refused, and the repo is unchanged: all five hidden-check lines came out True. Of the five checks written before the run, 4 passed; the miss is below.

The catch

The prompt asked for real problems only, one sentence each. The answer added this:

"One judgement call I didn't count as a bug: checkout.py:24 awards points on the full total, including any part paid by gift card. That's only wrong if your rule is to award points on what the customer actually pays (to_pay)."

That line wasn't seeded as a bug, and the repo states no rule either way, so it's a fair question, not a false alarm. But it's a question, not a real problem, and it wasn't asked for. The bug also took two sentences, not one. That's the one check it missed. Where the code can't tell it your rule, it asks instead of deciding.

The stamp

Half works · 16 Nov 2026 · Claude Code CLI on Opus 5.5 · 4 of 5 checks. It found the one real bug at the right line, raised no false alarm on the three lines that only look wrong, and changed no files. Asked for real problems only, it still raised a question about a rule the code never stated. Decide your rules before the review, or expect questions back. One run is one run, and the bug and the look-alikes here were planted.

Before you use it

Try it on your own code, or a demo repo like this one, never on your employer's code from a personal account. Ask for real problems only, with the file and line for each, and say not to change files. Check the bug it names before you fix it. If your code has rules it can't show, such as who earns points on what, write them down where the review can read them, or expect a question back.

The raw answer, word for word

Nothing was edited: no local path appeared in it.

I found one real bug in the last commit (f4dd3ca, "Add loyalty points"):

- **`loyalty.py:46`**: `redeem_points` subtracts points without checking that the customer has enough or that `points` is positive, so a customer with 20 points can redeem 500 and end up at -480. A negative `points` value also *adds* points while still returning "points used".

The rest of the commit looks correct: the points arithmetic (`total_paise // 10000`), the once-only welcome bonus, and the `ORDER BY` whitelist in `history` (no SQL injection).

One judgement call I didn't count as a bug: `checkout.py:24` awards points on the full `total`, including any part paid by gift card. That's only wrong if your rule is to award points on what the customer actually pays (`to_pay`).

I didn't run the tests (that command needed approval) and I didn't change any files.

The code it was asked about

loyalty.py, the whole new file, as the review read it. The comments are the code's own; none marks the bug.

import sqlite3

from messages import MSG

# Columns a customer may sort their points history by. Only these names ever reach SQL.
SORT_COLUMNS = {"newest": "id DESC", "oldest": "id ASC", "biggest": "points DESC"}

WELCOME_POINTS = 20


def points_for_order(total_paise):
    # One point for every full Rs 100 spent; part of Rs 100 earns nothing (shop rule).
    return total_paise // 10000


def balance(db, customer):
    row = db.execute("SELECT points FROM loyalty WHERE customer = ?", (customer,)).fetchone()
    return row[0] if row else 0


def _add(db, customer, points, reason):
    db.execute("INSERT OR IGNORE INTO loyalty (customer, points) VALUES (?, 0)", (customer,))
    db.execute("UPDATE loyalty SET points = points + ? WHERE customer = ?", (points, customer))
    db.execute("INSERT INTO points_history (customer, points, reason) VALUES (?, ?, ?)",
               (customer, points, reason))


def earn(db, customer, total_paise):
    points = points_for_order(total_paise)
    if points:
        _add(db, customer, points, "order")
    return points


def give_welcome_bonus(db, customer):
    # The welcome bonus is given once. A second insert hits the UNIQUE key and is skipped on purpose.
    try:
        db.execute("INSERT INTO welcome_bonus (customer) VALUES (?)", (customer,))
    except sqlite3.IntegrityError:
        return False
    _add(db, customer, WELCOME_POINTS, "welcome")
    return True


def redeem_points(db, customer, points):
    _add(db, customer, -points, "redeem")
    return {"ok": True, "msg": MSG["points_used"].format(points=points), "left": balance(db, customer)}


def history(db, customer, sort="newest"):
    order_by = SORT_COLUMNS.get(sort, SORT_COLUMNS["newest"])
    return db.execute(
        f"SELECT points, reason FROM points_history WHERE customer = ? ORDER BY {order_by}",
        (customer,),
    ).fetchall()

The change in checkout.py, where points are earned on the order total:

    order_id = add_order(db, total)
    if customer:
        points = earn(db, customer, total)
        if points:
            notes.append(MSG["points_earned"].format(points=points))

The test log

  • Run: 16 Nov 2026, Claude Code CLI 2.1.325 on Windows, personal paid plan, Opus 5.5 (claude-opus-5-5, the CLI default), one claude -p run in a fresh session, about 28 seconds. Checks written and committed before the run.
  • Screens: the slides show the run's own text (the prompt and the answer), displayed as plain text and cropped; words unchanged, the brass boxes are the only marks added, and a dashed line marks words left out.
  1. The real bug in redeem_points: pass. loyalty.py:46, no balance check, 20 points can redeem 500 and end at -480. Hidden check: 50 redeemed from 20 leaves -30.
  2. No false alarm on the ORDER BY f-string: pass. It named the whitelist as correct.
  3. No false alarm on the floor division: pass. It called total_paise // 10000 correct.
  4. No false alarm on the ignored duplicate insert: pass. It called the once-only welcome bonus correct.
  5. Answers as asked, changes no files: fail. No files changed, same commit. But the bug took two sentences, and it added a judgement call it said wasn't a bug.

The hidden check, written before the run and never shown to the review:

# Hidden check for Code review with AI part 5, written 16 Nov 2026 before the run. Never shown to the agent.
# Usage: python hidden_check.py <demo repo folder> <commit sha before the run>
# Shows the one seeded bug is real, the three bait lines behave as intended, and the run left the repo unchanged.
import subprocess, sys

repo, sha = sys.argv[1], sys.argv[2]
sys.path.insert(0, repo)
from db import connect
from loyalty import balance, give_welcome_bonus, history, points_for_order, redeem_points

# Real bug (loyalty.py lines 45-46): redeem_points never checks the balance.
db = connect()
give_welcome_bonus(db, "test-1")                     # 20 points
redeem_points(db, "test-1", 50)
print("bug, 50 points redeemed from 20 leaves", balance(db, "test-1"), "(below zero):", balance(db, "test-1") < 0)

# Bait 1 (line 53): a sort value that isn't in SORT_COLUMNS falls back to newest; no SQL error, no injection.
rows = history(db, "test-1", sort="points; DROP TABLE loyalty")
print("bait 1, odd sort value is ignored and the table is still there:",
      len(rows) == 2 and db.execute("SELECT count(*) FROM loyalty").fetchone()[0] == 1)

# Bait 2 (line 13): one point per full Rs 100, by the shop rule.
print("bait 2, Rs 199.99 earns 1 point and Rs 200 earns 2:", points_for_order(19999) == 1 and points_for_order(20000) == 2)

# Bait 3 (lines 37-40): the welcome bonus is given once.
db2 = connect()
print("bait 3, second welcome bonus is refused and balance stays 20:",
      give_welcome_bonus(db2, "test-2") is True and give_welcome_bonus(db2, "test-2") is False and balance(db2, "test-2") == 20)

st = subprocess.run(["git", "-C", repo, "status", "--porcelain"], capture_output=True, text=True).stdout.strip()
head = subprocess.run(["git", "-C", repo, "rev-parse", "HEAD"], capture_output=True, text=True).stdout.strip()
print("repo unchanged after the run:", st == "" and head == sha)

Its output after the run: "bug, 50 points redeemed from 20 leaves -30 (below zero): True", "bait 1, odd sort value is ignored and the table is still there: True", "bait 2, Rs 199.99 earns 1 point and Rs 200 earns 2: True", "bait 3, second welcome bonus is refused and balance stays 20: True", "repo unchanged after the run: True".

Stamp: Half works, 4 of 5 checks.

Next in the series, part 6: which AI reviews best, four AIs compared.

Sources

Each source was opened and read on the date shown. Numbers in square brackets in the text point here.

  1. Claude Code docs: CLI reference (-p, --allowedTools, --permission-mode, --no-session-persistence, --output-format)code.claude.com · read 16 Nov 2026
  2. Anthropic Consumer Terms of Service: minimum age 18 (effective October 8, 2025)anthropic.com · read 16 Nov 2026