Field Guide · For engineers

A seeded bug: does the AI review find it?

Field Guide, Code review with AI, part 1 of 6. Plant one bug you know about, ask Claude Code to review it, and check what it finds. Tested, with the catch.

2 Nov 2026 · 4 min read · sources read up to 2 Nov 2026

Seen on Instagram: the post this article goes with.

Your tests are green and the diff looks fine. You ask an AI to review it before you merge. But you have never seen that reviewer miss anything, or catch anything, so its "looks good" means very little yet. This is part 1 of the Field Guide series on code review with AI: one job, tested in a real run, with the catch said plainly.

The job: test the reviewer first

Plant a bug you already know about, in a small diff, and see if the review finds it. The demo repo here was written for this test: a tiny order list with a commit that adds pages. The docstring says page 1 is the first 10 orders. One line gets the start wrong, so page 1 skips them. The 4 tests in the repo still pass, because the only paging test checks that page 1 has 10 orders, not which ones.

def paginate(orders, page, per_page=10):
    """Return one page of orders. Pages start at 1: page 1 is the first 10 orders."""
    if page < 1:
        raise ValueError("page starts at 1")
    start = page * per_page
    return orders[start:start + per_page]

Claude Code works inside your project folder: it reads the files, runs commands such as your tests, and edits code when you ask it to fix something ^[1]. So the prompt can ask for all three.

The wrong line is start = page * per_page: page 1 should start at order 0, but it starts at order 10.

Review my last commit for bugs. For each bug, give the file and line, say what goes wrong, and fix it. Then run the tests.

How it was tested

One run on 2 November 2026, in the Claude Code desktop app on Windows, on a personal paid plan. It was a fresh agent with no chat history, started by another Claude Code session in the same app and pointed at the demo repo. Besides the prompt, the agent was told only that it was Claude Code working in the demo folder, to touch nothing outside it, and not to commit. Claude is 18+ ^[2]. Four checks were written down before the prompt was sent, plus a hidden check for the fix that the review never saw:

1. Finds it: names the start line in paginate() without being told where to look.
2. Explains it: says what goes wrong for a user.
3. Fixes it: the fix passes the hidden check (page 1 is orders 1 to 10, page 3 is orders 21 to 25) and the old tests still pass.
4. No false alarms: nothing it calls a bug is correct code.

What came back

About 25 seconds later it named orders.py, line 18, quoted the line, and said the docstring promises pages from 1 while the maths counts from 0. So page 1 returns orders 10 to 19, the first 10 never show on any page, and the last page comes back empty. It changed the line to start = (page - 1) * per_page, ran the tests, and said page_count and the page < 1 check were right. The hidden check passed. All 4 checks passed.

The catch

The tests. All 4 passed on the buggy code, and the review said so: the paging test only counted the orders on page 1. It added a test that checks which orders are on each page, and that test fails on the old line. A green test run is not a review. Keep the test it writes, and read the fix before you accept it.

The stamp

Works · 2 Nov 2026 · Claude Code, paid plan · 4 of 4 checks, no miss. It found, explained and fixed the one seeded bug, with no false alarms. One planted bug in a 13-line diff is the easy case, and one run is one run. Part 2 tries a small diff with more than one problem.

Before you use it

Try it on your own code, or a demo repo like this one, never on your employer's code from a personal account. Run your own check on any fix it makes; a fix that passes the old tests can still be wrong.

The raw answer, word for word

The one edit is the local folder path, replaced with <demo folder>.

I found one bug in your last commit, eaebb95 "Add pagination to the order list", and fixed it. All 5 tests pass now. Nothing is committed.

**The bug: `orders.py`, line 18, in `paginate`**
start = page * per_page

The docstring says pages start at 1, but this line works out the start as if they started at 0. So page 1 returns orders 10–19 instead of 0–9. The first 10 orders never show up on any page, and the last page (page 3 of 25 orders) comes back empty.

Fix:
start = (page - 1) * per_page

**Why the tests didn't catch it:** `test_page_size` only checks that page 1 has 10 items. With the bug, page 1 still has 10 items, just the wrong ones. All 4 tests passed on the buggy code (I ran them before changing anything).

**Test added: `tests/test_orders.py`, new `test_page_contents`.** With 25 orders, it checks that page 1 is orders 0–9, page 3 is orders 20–24, and the page after the last one is empty. This test fails on the old line and passes with the fix.

The rest of the commit looks right: `page_count` (ceiling division, 0 pages for an empty list) and the `page < 1` check.

**Test run:** `python -m pytest` gives 5 passed. pytest also prints a warning from the pytest-asyncio plugin about an unset fixture loop scope. It comes from the plugin's settings, not your code.

Files changed (not committed):
- `<demo folder>/shop-orders/orders.py`
- `<demo folder>/shop-orders/tests/test_orders.py`

The code fences inside the answer (three backticks and python) are left out above so the block shows as one piece; the words are unchanged.

The diff the review saw

The last commit, exactly as the review saw it. No comment marks the wrong line.

diff --git a/orders.py b/orders.py
index 25fcaae..16524b1 100644
--- a/orders.py
+++ b/orders.py
@@ -4,3 +4,16 @@
 def order_total(items):
     """Sum of price * qty for each line."""
     return sum(i["price"] * i["qty"] for i in items)
+
+
+def page_count(orders, per_page=10):
+    """How many pages the order list needs."""
+    return (len(orders) + per_page - 1) // per_page
+
+
+def paginate(orders, page, per_page=10):
+    """Return one page of orders. Pages start at 1: page 1 is the first 10 orders."""
+    if page < 1:
+        raise ValueError("page starts at 1")
+    start = page * per_page
+    return orders[start:start + per_page]

The test log

• Run: 2 Nov 2026, Claude Code desktop app (Windows), personal paid plan, a fresh agent with no chat history started by another Claude Code session, about 25 seconds. Checks written before the run.
• Screens: the slides show the run's own text (the diff and the answer), displayed as plain text and cropped; words unchanged, the brass boxes are the only marks added.

1. Finds it, without a hint: pass. It named orders.py line 18, start = page * per_page.
2. Explains it: pass. Page 1 returns orders 10 to 19, the first 10 never show, the last page comes back empty.
3. Fixes it: pass. (page - 1) * per_page; the hidden test below passed, and so did the 4 old tests plus the one it added.
4. No false alarms: pass. It called page_count and the page < 1 check right.

The hidden test, written before the run and never shown to the review:

# Hidden check, written before the run: page 1 must be the first 10 orders, the last page must not be empty.
import sys; sys.path.insert(0, sys.argv[1])
from orders import paginate, page_count
o = list(range(1, 26))
assert paginate(o, 1) == list(range(1, 11)), paginate(o, 1)
assert paginate(o, 3) == [21, 22, 23, 24, 25], paginate(o, 3)
assert all(paginate(o, p) for p in range(1, page_count(o) + 1))
print("hidden check: PASS")

Stamp: Works, 4 of 4 checks, no miss.

Next in the series, part 2: a small diff with more than one problem.