DEV Community

Dimitris Kyrkos
Dimitris Kyrkos

Posted on

Your AI-generated tests are grading their own homework

Ask an AI assistant to "write tests for this endpoint" and you will get tests. Lots of them. They compile, they pass, coverage goes up, and the PR looks great.

The problem is where those tests came from. If the model read the implementation and then wrote tests for it, the tests describe what the code does. They don't describe what the code is supposed to guarantee. Those are not the same thing, and the gap between them is exactly where security bugs live.

Tutorial tests vs. production tests

Tutorial tests answer "does this function return what I expect?"

Production tests, at least the ones that matter for security, answer a different question: "if someone breaks the thing that keeps us safe, will this suite notice?"

A test suite generated from the implementation is very good at the first question and surprisingly bad at the second. Here is how that plays out.

Failure mode 1: the test that mirrors the query

Illustrative example. Say you have a multi-tenant API and this handler:

def get_invoices(user, db):
    return db.query(Invoice).filter(Invoice.tenant_id == user.tenant_id).all()
Enter fullscreen mode Exit fullscreen mode

An assistant reads it and writes:

def test_get_invoices_returns_user_invoices(user, db):
    seed_invoices(db, tenant_id=user.tenant_id, count=3)
    assert len(get_invoices(user, db)) == 3
Enter fullscreen mode Exit fullscreen mode

Green. Now delete the .filter(...) line. The test is still green, because the fixture only ever seeded one tenant. The test was built from the code's happy path, so it inherited the code's blind spot: it never imagined a second tenant existing.

Failure mode 2: asserting what came back, not what shouldn't have

Implementation-derived tests tend to assert on presence. "The response contains the user's name and email." Fine. But the security property is about absence: "the response does not contain ssn, password_hash or internal_notes for a caller without the right role."

If a refactor adds a field to the serializer, a presence-based test keeps passing while you start leaking data.

Failure mode 3: the happy path, three times

Ask for "thorough tests" and you often get the same valid request with slightly different values. What's missing is the hostile input: a missing token, an expired token, a token for a different tenant, an ID that belongs to someone else, an empty list, the maximum page size plus one.

Start from invariants instead

Before generating anything, write down the properties that must hold no matter how the code is implemented. For example:

  • A user cannot read another tenant's records.
  • An unauthorized request cannot change protected state.
  • Sensitive fields are not returned to callers who lack permission.

These are written without looking at the code. That's the point. They come from the threat model and the product requirements, not from the function body, so they can't quietly copy its assumptions.

Now the AI is genuinely useful: give it the invariant and ask it to generate cases across normal, invalid and boundary inputs.

@pytest.mark.parametrize("caller", [
    "same_tenant_user",
    "other_tenant_user",
    "anonymous",
    "expired_token",
    "admin_of_other_tenant",
])
def test_invoices_never_cross_tenants(caller, two_tenant_db, make_client):
    client = make_client(caller)
    resp = client.get("/invoices")
    returned = {inv["tenant_id"] for inv in resp.json().get("items", [])}
    assert returned <= {client.tenant_id}
Enter fullscreen mode Exit fullscreen mode

The fixture deliberately seeds two tenants. The assertion is about what must never appear. Delete the filter and this test fails for same_tenant_user immediately.

Prove the suite can actually fail

This is the step most teams skip. A passing suite tells you nothing until you know it would fail for the bug you care about.

Mutation testing automates that check. Tools like mutmut (Python), Stryker (JS/TS, C#) and PIT (Java) make small changes to your code, such as flipping a comparison or removing a condition, and rerun the tests. If a mutant survives, no test noticed the change.

You don't need to run it on the whole codebase. Point it at your authorization checks, tenant filters and serializers. A surviving mutant in check_permission() is worth more attention than a hundred in your logging helpers.

The reframe

AI didn't create this problem. Tests derived from the implementation have always had it. AI just makes it cheap to produce a lot of them, which makes the gap easier to miss.

A green test suite is only useful if it would turn red for the failure you care about.

How are you deciding what your AI-generated tests should check? Do you write the properties first, or review what the model produced after the fact?

Top comments (7)

Collapse
 
alexcodebytes profile image
Oleksandr •

This is a really important distinction. I've seen AI-generated tests focus heavily on confirming that the implementation works as written, rather than challenging the assumptions behind it. I especially like the idea of writing invariants before showing the model the implementation. That changes the whole perspective from "does this code work?" to "can this code violate something it must never violate?" Mutation testing is a great addition too. A green test suite means very little if it can't catch the failures that actually matter.

Collapse
 
cyclopt_dimitrisk profile image
Dimitris Kyrkos •

Exactly. The key is that the model shouldn't be the one deciding what correctness looks like based on the implementation it's testing. I think that's where a lot of the false confidence comes from. You end up with a large suite that validates assumptions nobody explicitly agreed on. Starting with invariants gives you an independent reference point, and mutation testing helps verify that the tests actually enforce it.

Collapse
 
alexcodebytes profile image
Oleksandr •

Exactly. A test suite can have impressive coverage and still be completely wrong if the expected behavior was derived from the same assumptions as the implementation. Invariants give you an independent standard, while mutation testing checks whether the suite can actually detect violations rather than just execute the code.

I think the distinction between testing that code runs and testing that it behaves correctly is easy to overlook, especially when AI generates both sides. The real question is whether the tests can prove the implementation wrong.

Thread Thread
 
cyclopt_dimitrisk profile image
Dimitris Kyrkos •

That's a good way to put it. And I think the tricky part is that having AI generate both the implementation and the tests makes this even easier to miss. Everything can look consistent because both are based on the same assumptions. That's why I wouldn't judge the quality of a test suite by coverage or test count, but by the failures it can actually catch.

Thread Thread
 
alexcodebytes profile image
Oleksandr •

Exactly. AI can produce an implementation and a perfectly passing test suite that reinforce the same flawed assumption, making the whole thing look more reliable than it actually is. More tests don't necessarily mean more independent verification.

I think the real value of a test is proving that the system can fail when something is genuinely wrong, especially in cases the original implementation didn't anticipate. Tests based on independent requirements, adversarial inputs, or unexpected edge cases become much more valuable here.

The interesting question is how we preserve that independence when AI is involved in writing almost everything, including the tests meant to challenge its own code.

Collapse
 
maciek_chmura_building profile image
Maciek Chmura •

One gap in the parametrized test: returned <= {client.tenant_id} also passes when the endpoint returns nothing. A route that throws a 500, or a refactor that filters out every row, stays green for all five callers. I'd add two asserts next to it. The same-tenant user gets back exactly the invoices seeded for that tenant, and anonymous and expired_token get a 401 instead of an empty list. Then the suite fails in both directions, leaking too much and returning too little. For the serializer case, do you keep an explicit allowlist of response fields per role?

Collapse
 
cyclopt_dimitrisk profile image
Dimitris Kyrkos •

Good catch. You're right, the subset assertion alone doesn't protect against returning nothing. We need to test both sides: what must never be exposed and what must actually be returned. Otherwise we risk having tests that pass simply because the endpoint stopped working.

On the serializer side, yes, I'd favor an explicit allowlist per role. It's much safer than checking a few expected fields because newly added fields won't silently become part of the API response. That's exactly the kind of regression these tests should catch.