The AI fixed the bug. Every test passed. The customer lost free shipping.

Here is a pet-shop checkout. It is small on purpose, but the coupling in it is the kind every real codebase has: two features quietly depend on a third.

TAX_RATE = 0.10        # 10% tax
LOYALTY_PER = 10.0     # 1 loyalty point per $10 of order total
FREE_SHIP_AT = 30.0    # free shipping (and silver tier) at/above this total


def checkout_total(items):
    subtotal = round(sum(i["price"] * i["qty"] for i in items), 2)
    tax = round(subtotal * TAX_RATE, 2)
    return {"subtotal": subtotal, "tax": tax, "total": round(subtotal + tax, 2)}


def loyalty_points(items):
    total = checkout_total(items)["total"]            # derived from total
    return {
        "points": int(total // LOYALTY_PER),
        "tier": "silver" if total >= FREE_SHIP_AT else "bronze",
    }


def shipping_fee(items):
    total = checkout_total(items)["total"]            # derived from total
    free = total >= FREE_SHIP_AT
    return {"fee": 0.0 if free else 3.5, "free_shipping": free}

A customer has three bags of dog food and a leash in the cart: $27.30 before tax, $30.03 after. Just over the $30 line, so they get free shipping and silver tier.

The fix

Now ask an AI agent to make the tax calculation "more precise". It changes one line:

-TAX_RATE = 0.10
+TAX_RATE = 0.095

The tests run:

$ pytest -q
...                                                    [100%]
3 passed in 0.01s

Green. The agent reports that the task is done.

But the total is now $29.89. The customer dropped below $30, so they lost free shipping and silver tier. Nothing failed, because no test said "this cart ships free". Nobody writes that test. You test the thing you are changing, not the two features that happen to read its output.

That is the pattern with AI-made changes: the agent fixes A, and B, C and D move with it. The suite stays green because nothing pins down B, C or D.

What it looks like with a gate

We ran the same change through nightward, a pytest plugin we built for exactly this. The output below is from the actual run, trimmed only for length:

$ nightward run .
3 passed in 0.02s

Boundary: breached (3 unapproved)
unchanged=0 changed=3 new=0 removed=0

group: billing
  - [CHANGED] checkout_total

group: fulfillment
  - [CHANGED] shipping_fee

group: loyalty
  - [CHANGED] loyalty_points

One line changed, three behaviors moved, in three different parts of the product. nightward review shows exactly what moved:

[CHANGED] checkout_total
-  "tax": 2.73,
-  "total": 30.03
+  "tax": 2.59,
+  "total": 29.89

[CHANGED] shipping_fee
-  "fee": 0.0,
-  "free_shipping": true
+  "fee": 3.5,
+  "free_shipping": false

[CHANGED] loyalty_points
-  "points": 3,
-  "tier": "silver"
+  "points": 2,
+  "tier": "bronze"

And the gate stays closed until someone decides:

$ nightward gate
boundary breached (3 unapproved)
$ echo $?
1

No expected values

Here is the whole test file. Notice what is missing:

from shop import checkout_total, loyalty_points, shipping_fee

CART = [
    {"sku": "dog-food", "price": 8.50, "qty": 3},
    {"sku": "leash", "price": 1.80, "qty": 1},
]


def test_checkout_total(behavior):
    behavior("checkout_total", checkout_total(CART), group="billing")


def test_loyalty_points(behavior):
    behavior("loyalty_points", loyalty_points(CART), group="loyalty")


def test_shipping_fee(behavior):
    behavior("shipping_fee", shipping_fee(CART), group="fulfillment")

There are no assert lines. The behavior fixture records what the code returns today. The first time you run it, a human reads those values (nightward review) and approves them once (nightward approve). They are saved as plain JSON next to your code and committed:

{
  "env": {
    "os": "Windows",
    "python": "3.12"
  },
  "group": "loyalty",
  "name": "loyalty_points",
  "payload": {
    "points": 3,
    "tier": "silver"
  },
  "source": "test_shop.py::test_loyalty_points"
}

From then on, every run compares against that approved baseline and lists everything that moved, grouped by feature.

If you have ever put off testing because writing the expected values felt like the real work, this is the part that changes. You don't decide what the right answer is up front. You look at what the code does now and say "yes, that".

What it does not do

This matters, so it gets its own section.

• It doesn't know what is correct. nightward cannot tell you that bronze tier is a bug. Maybe the new tax rate is the law and the customer really should pay shipping. It only makes sure the change can't pass silently. A human either approves it (intended) or fixes the code (regression).
• It only guards what you capture. If no test captures shipping, a shipping change goes unnoticed. Capture the outputs your users would notice.
• It is not a test generator. It doesn't write tests for you. It turns the tests you have into a gate.

Timestamps, UUIDs and similar volatile values are normalized before comparison, so a gate that fires on every run is treated as a bug. A noisy gate is a dead gate.

Built for AI agent loops

The split is deliberate: an agent can run the gate and read the verdict (nightward run, nightward gate, or the nightward_run / nightward_status MCP tools), but approving is a human-only CLI step whose result is a file diff that goes through code review. The agent gets a definition of "done" that it can check but not change.

Try it in five minutes

pip install nightward

Add a test that calls behavior(name, value, group=...) on something your users see, then:

nightward init
nightward run .
nightward review
nightward approve --all
git add .nightward && git commit -m "nightward baseline"

After that, nightward gate in CI exits non-zero whenever an unapproved change shows up.

• GitHub: https://github.com/LunenLabs/nightward (MIT, Python 3.10+)
• Overview: https://www.lunenlabs.com/nightward

How do you currently check what your AI agent changed besides the thing you asked it to change? I'd like to hear what works for you, and where nightward falls short.

Drafted with AI assistance; every output above is from a real run.