Lazy model, eager import

A deploy started failing on a small production box. The step that broke was the frontend build — a Vite build, run over SSH, which had worked fine for months and now got Killed at the transforming... stage with exit code 137. That is the kernel's OOM killer, not a build error.

The box had no memory left to give it. Sorting processes by resident size turned up the culprit immediately, and it was not the web server:

PID   USER       RES     %MEM  COMMAND
909   worker    478 MB   23.7  python   # the crawler
908   web       258 MB   12.8  python   # the web service
879   other      90 MB    4.5  uvicorn

The crawler. A process that fetches pages, parses them, and writes rows to SQLite — using a quarter of the machine, and more than the web service it supports.

The bug is one line, in a file nobody suspected

The application embeds text for semantic search, and the module that does it opened like this:

from sentence_transformers import SentenceTransformer

def _get_embed_model():
    """Get or create a cached model (thread-safe)."""
    if not hasattr(_get_embed_model, "_model"):
        with _get_embed_model._lock:
            if not hasattr(_get_embed_model, "_model"):
                _get_embed_model._model = SentenceTransformer(MODEL_NAME)
    return _get_embed_model._model

Read that function on its own and it is entirely correct. The model is cached, built on first use, and guarded with double-checked locking against two threads racing to build it. Somebody thought about this. Somebody wrote a docstring about it.

And it does not help at all, because the expensive thing already happened on line 1.

The entry point wires up every HTTP handler at module scope, the way essentially every Python web app does:

from api.embed import EmbedHandler       # <- drags in the import above
from api.chat import Chat
from api.health import Health
...

def main():
    if options.crawl:      ...   # never touches embeddings
    if options.mkindex:    ...   # never touches embeddings
    if options.serve:      ...

One file, several CLI modes, all the imports at the top. So --autocrawl, which never calls _get_embed_model() and has no use for a neural network, still paid for sentence_transformers — and through it transformers, and through that, PyTorch — before main() was even called.

The distinction that gets lost

Instantiating a model and importing the library that defines it are two different costs, and only one of them was deferred.

That sentence is the whole note. It gets missed because the expensive part of ML work feels like it should be the weights: the download, the several hundred megabytes on disk, the load into memory. Lazy-loading those is the well-known optimisation, it is the one people implement, and having implemented it they stop looking.

But import torch is not a cheap statement that merely makes a name available. It executes an enormous package tree, registers thousands of operators into dispatch tables, initialises the C++ runtime, and allocates before your code has asked for anything:

$ python -X importtime -c "import sentence_transformers"
...
import time:  4,515,572 us | sentence_transformers   (cumulative)
import time:  2,672,197 us |   transformers
import time:  1,089,155 us |     torch

Four and a half seconds, and hundreds of megabytes resident, to import a name you will never reference.

Measured directly on our entry point, before and after moving that one line inside the function:

Same code paths, same behaviour, 290 MB back. (Exact figures move around by platform and version; the ratio does not.)

Why almost every project of this shape has it

Three ordinary decisions combine into it, and each one is defensible alone:

1. Imports go at the top of the file. PEP 8 says so, linters enforce it, and for 99% of dependencies it is correct.
2. The entry point imports every handler at module scope, because it needs them all in the route table. Flask, FastAPI, Django and Tornado apps all look like this.
3. One process image, several jobs. A main() that branches on CLI flags — serve, crawl, migrate, index — is the standard way to avoid maintaining four separate entry points.

Put together, the transitive closure of every dependency your largest mode needs gets loaded by your smallest one. Usually that is fine, because usually the dependencies are small. It stops being fine the moment one of them is a machine learning framework.

The same trap sits under a long tail of other libraries — pandas, scipy, matplotlib, boto3, selenium, spacy, most cloud SDKs. Anything where the import itself is heavyweight and the usage is conditional.

Finding it in your own project

Ask what your cheapest mode is actually loading:

python -X importtime -c "import yourapp" 2>&1 | sort -t'|' -k2 -rn | head -20

The second column is cumulative microseconds per module. Anything surprising near the top is a candidate. Then check whether the code path that needs it is one your background workers ever take.

For memory specifically, the crude version is the reliable one — import your entry point, sleep, and read the process's resident size from outside. Comparing that number before and after a change is more convincing than reasoning about which module pulls in what.

Also worth knowing: sys.modules after startup tells you what actually got loaded, which is often surprising.

python -c "import yourapp; import sys; print(len(sys.modules))"

Doing the fix without making it worse

Move the import into the function that needs it, and say why — an import in an unusual place reads as a mistake unless the comment explains it:

def _get_embed_model():
    """...

    sentence_transformers (and the torch it pulls in) is imported here
    rather than at module level: importing it costs ~290 MB resident
    before a single weight loads, and the entry point imports this
    module unconditionally regardless of which CLI mode is running.
    """
    if not hasattr(_get_embed_model, "_model"):
        with _get_embed_model._lock:
            if not hasattr(_get_embed_model, "_model"):
                from sentence_transformers import SentenceTransformer
                _get_embed_model._model = SentenceTransformer(MODEL_NAME)
    return _get_embed_model._model

Three things worth being deliberate about:

Cost is paid once, not per call. sys.modules caches it. A function called in a loop does one dictionary lookup after the first time through.

Put it where the caching already is. Our import went inside the existing double-checked lock, so the expensive import happens exactly once, under a lock somebody already reasoned about. Dropping it at the top of a hot function body would work but re-litigates thread safety for no reason.

Import failures move from startup to first use. A missing dependency now surfaces when the feature is used rather than when the process boots. Usually an improvement — the crawler no longer refuses to start over a package it does not need — but it is a real behaviour change, and worth a health check on the mode that does need it if a late failure would be expensive.

The part that generalises

The interesting thing here is not that a background worker used too much memory. It is that the code contained a careful, correct, well-commented lazy-loading implementation, and that implementation is exactly what stopped anyone from noticing the problem sitting three lines above it. The presence of a solution to a similar-sounding problem reads as evidence the problem is handled.

It surfaced through a completely unrelated failure — a frontend build getting OOM-killed — on a box that had been quietly running two hundred megabytes closer to the edge than anyone believed for months. Nothing logged a warning. The crawler worked perfectly. It was just fat, and nothing in a normal day's operation reports that.

Which is the argument for occasionally sorting your processes by resident size and asking whether each number is one you can explain. Not in response to an incident — the numbers are always there, and they are one of the few things a running system will tell you honestly if you bother to ask.