Forbidden Inference

A project of ours talks to two kinds of models. One is an embedding model that runs in-process — a small SentenceTransformer that turns job descriptions into vectors. The other is a hosted LLM, reached over plain HTTPS for an assistant panel. Neither is exotic; that is roughly the default shape of a web app now.

Getting the test suite in order turned up two problems that look unrelated. Some tests were not being discovered — they ran under the command a developer types and never ran in CI at all. And some tests, wherever they ran, were reaching live inference: the embedding ones by loading the real model, the chat ones by hitting the real provider.

They are the same problem. The rule against live inference was being enforced by a commented-out test — the one test that would have loaded the model was commented out of the suite, and that was the entire mechanism. A rule kept in somebody's head, same as a hand-maintained list of which tests exist. Neither can notice when it has become false.

What a live call in a test costs

Money, on a meter you don't control. A test that calls a hosted model bills on every run — and "every run" is not the test's decision. CI runs on every push, every force-push updating a pull request, every retry of a flaky job, every cell of a version matrix. A suite with live calls in it is a meter wired to the merge schedule. And the bill has no line items: the tokens show up on a monthly invoice with nothing to say which test spent them, so the cost is invisible right up until it isn't.

Nondeterminism, which quietly becomes no assertions. You cannot assert on a live model's output — the right answer today is a different string tomorrow. So live-call tests drift toward asserting almost nothing. The one we inherited had no assertions at all: a fetch, a commented-out pdb, and nothing else. It would have passed whatever the handler did. A green test that cannot fail is worse than no test, because it looks like coverage.

Flakiness, which costs more money. Providers rate-limit, time out, and have outages; a chat call with a 120-second timeout is a long time for a test to hang. A flaky suite teaches the team to hit re-run, and a re-run is another round of live calls. If CI and production share a key, a chatty suite can eat the quota production actually needs.

Credentials, in two directions. Live calls need real keys in CI, which is a blast radius of its own. But the failure mode shows up even earlier: the guards that say "chat is unavailable without credentials" only work if a placeholder reads as unconfigured. Ours compared against one sentinel value while .env.example shipped placeholders in a different dialect — so a freshly copied config reported every credential as present, the placeholder reached the provider, and the provider's 401 reads exactly like a revoked key. We spent real time chasing a key that had never been filled in.

Slowness, the local-model version of the same disease. Loading an embedding model downloads weights and burns CPU per call. A suite that does it is slow everywhere and fails wherever the download is blocked. Four-minute green runs, on every push.

And it tests the wrong thing. This is the deepest issue. A live call exercises the provider's behavior, not yours. What you own is everything around the call: the status codes, the JSON shapes, the null content a reasoning model returns when it runs out of budget mid-thought — the handler called .strip() on that and 500'd, a live call found the bug in production, and a stubbed test reproduces the exact response shape deterministically, forever. None of that needs the provider. All of it needs the seam.

Guard the category, not the instance

The first guard went where the last incident had been: the embedding loader. The tests package replaces it with a function that raises, naming the rule, so any test that reaches past its stub gets an error instead of a slow green run. Good guard. Incomplete guard.

The chat endpoint goes through an ordinary async HTTP client, and http_client.fetch doesn't look like inference — it looks like every other HTTP call in the app. Nobody thought of it as an unguarded exit until somebody went looking for exits, and there it was.

That is the general trap. Guards get placed where the last incident happened, and the next provider is always a different shape: a model loader here, a plain HTTPS call there, an SDK client somewhere else. A rule per provider is a treadmill — every new integration is a new unguarded exit until it bites. What needs guarding is the category: code that leaves the process to get a model's output, however it travels.

The setup that holds

One module owns inference. Route the model loader and the provider HTTP through one module, and two things become true: tests stub a named function instead of patching internals, and that module's function list is the complete list of exits. "Wherever the code feels like it" is not an enforceable boundary. One module is.

Poison the exits at import time. In the tests package's __init__.py, before anything else runs:

def _forbid_inference():
    import inference  # the one module that talks to providers

    def refuse(*args, **kwargs):
        raise AssertionError(
            "tests must not make live inference calls -- "
            "stub inference.embed / inference.chat in the test instead"
        )

    for name in ("embed", "embed_many", "chat"):
        setattr(inference, name, refuse)

_forbid_inference()

Import time, not inside the test runner, because the runner is only one entry point. python -m tornado.testing tests.chat — the command you type while iterating on a single test — never calls the runner. Importing the tests package is the one thing every entry point has in common, so that is where the rule lives. It also travels: the poison applies on every machine that runs the tests, not just CI. Blocking egress at the runner is a decent backstop, but it is blunt — the suite has legitimate network needs — and it does nothing for a developer's laptop, which is where the accidental live call happens first.

Keep the poison honest with a meta-test. The poison is an attribute assignment; a refactor can silently undo it. One test reaches for each guarded exit and asserts it raises:

def test_the_guard_is_still_in_place(self):
    try:
        inference._load_model()
    except AssertionError:
        pass  # the poison held
    else:
        raise AssertionError("live inference is reachable from a test")

Discover, don't register. The old suite kept a hand-maintained list of addTests() calls, and CI went through it — so a new test file ran fine locally and never ran in CI at all. A test nobody registered is a test nobody has. Discovery walks the directory, imports every module, and collects what it finds; both entry points share one discovery function, so they cannot disagree about what the suite contains. Three details decide whether it works:

  • Files named for what they cover (auth.py, chat.py) don't match unittest's default test*.py pattern — import the modules directly rather than fighting the pattern.
  • unittest collects imported TestCases too, and every test file subclasses the same base — so the base class's tests ran once per importing file. Keep only the tests a module defines.
  • An import error inside a test module hands back an empty suite, which reads as a clean run of nothing. Empty is an error, not a pass.

Make "not configured" mean it. The credential guards are part of the same wall: if a placeholder reads as a real key, the guard waves a live call through with a fake credential in it. Reject blanks, the old sentinels, and the placeholder shapes your example config actually ships:

def _is_configured(value) -> bool:
    text = str(value or "").strip().lower()
    if not text or text in ("xxx", "none", "changeme"):
        return False
    return not (text.startswith("your_") or "_your_" in text)

Rules that can fail

Every fix above is the same move. "Tests must not call live inference" started life as a comment and a commented-out test — enforcement by memory, which cannot notice when it has become false. It ended as an import-time poison that raises, a meta-test that checks the poison, a discovery function that finds new files on its own, and a credential check that rejects placeholders. Each one turns a rule somebody had to remember into a rule something enforces.

It is the same move as regenerating a derived artifact and failing on any diff — the vendoring note is that pattern applied to a copied file. Assert the invariant mechanically and let the suite police itself. The alternative is not a team that never forgets. It is a green CI badge on a suite that quietly bills you on every push.

Drafted with GLM, an AI assistant from Z.ai. The incidents were real; the generalizations are the machine's.