Shipping the index, not the query

On the job boards we build and run, search does not happen on the server. You type, and a few milliseconds later ten thousand postings are ranked — by keyword and by meaning at once — without a query ever leaving your machine. The only thing the server does at search time is turn your phrase into a few hundred numbers.

That's inverted from how this usually goes. The reasoning behind it is mostly about where work gets duplicated, and it generalises well beyond job listings, so it's worth writing down.

The version that fits on one screen

Semantic search starts out almost insultingly simple. Embed every document, embed the query, sort by cosine similarity:

model = SentenceTransformer('all-MiniLM-L6-v2')
vectors = model.encode([doc.text for doc in documents])

def search(query):
    q = model.encode(query)
    return sorted(documents, key=lambda d: cosine(q, vectors[d.i]))

That's the whole idea, and the idea is correct. Everything below is the same algorithm surviving contact with ten thousand documents, a modest shared server, and a browser.

Find the work that's being done twice

Our first real version did the sensible-looking thing: the browser fetched the listings and built its own search index, stemming every description on page load.

That costs 9.4 seconds of blocked main thread and 145 MB of heap across 61 MB of text — and it produces a result identical for every visitor. Everyone who opened the site paid, in full, to compute the same table everyone else was computing.

Stated that way the fix names itself. Stem the corpus once, on the server, pack the result into a binary, and let the browser map typed arrays over the bytes and do no parsing at all.

The counterintuitive part is that the search was never the expensive part. A full hybrid query in the browser measures about 5 ms over the whole corpus. It was the setup that hurt. So:

• Server, on a schedule: stem the corpus, invert it, quantise the vectors, pack one binary.
• Browser, once per visit: download a few megabytes, point typed arrays at it. No parsing.
• Browser, per keystroke: walk postings, dot int8 vectors. ~5 ms.
• Server, per search: embed one phrase. ~40 ms of CPU.

Shipping the data beats shipping the query, when the data is the same for everybody and small enough to move.

Packing it

The artifact is one file: a header, then a run of aligned sections.

magic + format version
counts: ndocs, nterms, npostings, dims
ids          uint32  x ndocs
scale, low   float32 x dims
vectors      int8    x ndocs*dims
offsets      uint32  x (nterms + 1)
postings     uint32  x npostings
terms        utf-8, newline separated

Two choices in there are about memory rather than elegance, and both were forced by measurement.

Postings are a flat array plus an offsets table, not a map of sets. A Map<string, Set<number>> in the browser costs roughly 36 bytes per posting; at four million postings that's 145 MB of heap. A flat Uint32Array is 4 bytes each. The server side has the identical problem in a different language — a Python list of ints stores a pointer plus a boxed integer object, about 36 bytes to hold a number that fits in four. Switching those lists to array('I') was most of a 318 MB build peak on a box that also has an embedding model resident.

Postings store positions within the artifact, not database ids. That keeps them in uint32 range and lets them subscript the vector matrix directly. The artifact carries its own id list so a separately-fetched listing can be matched by id rather than by position — the listing and the artifact are fetched at different moments, and anything written between them would otherwise slide every score onto the wrong document.

The format carries a version number, and readers refuse anything they don't recognise. A layout you half-understand parses into plausible nonsense rather than failing, and plausible nonsense is the expensive kind.

int8, and the part that's easy to get wrong

Embeddings are 384 float32 per document. Stored as JSON that's about 8.4 kB each, which is not something to send ten thousand of. Measured over ten realistic queries against exact float32 ranking:

int8 is 22x smaller than the JSON and costs essentially nothing in quality. Binary quantisation is not usable here — it returns visibly wrong results.

Now the trap, which is the most transferable thing in this piece. Write the approximation as X_i ≈ a_i * X8_i + b_i. Then:

X · q  =  sum(a_i q_i) X8_i  +  sum(b_i q_i)
                                ^^^^^^^^^^^^ identical for every document

The offset term is the same for every document in the corpus, so it cannot affect the ordering. Quantise only the documents, fold the per-dimension scale into the query at search time, and the inner loop stays a plain int8 multiply with no dequantisation per element.

Quantise both sides and dot them together, and those offsets leak into the comparison. Recall@50 drops from 99.2% to 69%. Our first attempt did exactly that, and the reason it lasted a while is that 69% recall looks fine. The top few hits are still roughly right. Nothing throws, nothing logs, no test goes red. The results are just quietly worse than they should be, and nobody can see it, because nobody knows what they should have gotten instead.

Both stemmers have to be the same stemmer

Here's the other silent one, and it's the better story.

The corpus is stemmed in Python when the index is built. The query is stemmed in JavaScript when someone types. Those are two different implementations of the same algorithm, and a term stemmed differently on the two sides simply never matches. No exception. No warning. The result just isn't in the list.

The JavaScript stemmer we started with implements an older revision of the Snowball English algorithm. Measured against the real corpus it disagreed with the C implementation on 78 of 30,856 alphabetic tokens. That's a quarter of a percent — except the disagreements weren't randomly scattered. They clustered almost entirely on words ending in -ologist.

Which meant "dermatology" did not match "Dermatologist" — on a job board whose whole audience searches by medical specialty.

The fix wasn't a better stemmer, it was the same stemmer. The indexer uses the canonical C library through its Python binding; the browser uses a wasm build of a Rust port that wraps the same algorithm revision. Those two agree on all 85,994 tokens in the vocabulary. They agree by construction rather than by hope, which is the only version of that property worth having.

The browser stemmer loads asynchronously, and there is deliberately no fallback to unstemmed tokens while it loads. Falling back would not degrade gracefully — "engineers" doesn't match the stored term "engin", so the fallback returns nothing rather than something slightly worse. Better to wait for it. It starts loading immediately and the search box debounces, so in practice it's ready long before anyone stops typing.

Two passes, and why the user sees the first one

A search runs in two stages, and that's a feature rather than an implementation detail.

The keyword pass needs no network, so it lands immediately. It walks the postings for each query term rather than testing every document against every term — the latter is O(docs × terms) to produce an answer that's almost entirely zeroes. A title hit weighs ten times a body hit. Sub-millisecond over the whole corpus.

Then the phrase goes to the server, comes back as an embedding, and the semantic pass rescores against the int8 matrix locally. It's weighted so that meaning can move a document by about one title hit's worth — enough to matter, not enough to bury an exact match.

If the embedding call fails, the keyword ranking is already on screen and simply stays there. Degrading to keyword search costs one early return and is a far better outcome than an empty page.

That embedding endpoint can't sit behind a login, because search is what the site is for. So it's budgeted instead, with a token bucket sized to comfortably absorb a fast typist correcting themselves while capping what any single client can consume. An open CPU-bound endpoint with permissive CORS is otherwise an invitation.

Cache by content, not by clock

The artifact is a few megabytes gzipped and rebuilds on a schedule, but its contents only change when the corpus does. A time-based cache would re-download the whole thing several times an hour to discover nothing had changed.

So it's an ETag over the file contents, with must-revalidate and no max-age. The bytes are deterministic for a given corpus, so identical input genuinely produces an identical tag, and an unchanged corpus revalidates in a single 304. The reason it's must-revalidate rather than a short max-age is that a stale index silently omits everything added since it was built, and the visitor has no way to tell. One conditional request per page load is a cheap way to make that impossible.

One detail that cost us an afternoon: compute the tag on the uncompressed bytes and add a suffix for the encoding. Hashing whichever representation you happened to serve gives the same content two unrelated tags, so a client whose Accept-Encoding varies between requests re-downloads the entire artifact instead of revalidating.

The thread running through it

Three of the failures above — the leaked quantisation offset, the mismatched stemmer, scores landing on the wrong document — have the same shape. None of them raises. None fails a test that existed at the time. They just return slightly wrong answers.

Slightly wrong answers from a search box are nearly invisible, because the person searching has no idea what they should have gotten back. There's no red build, no stack trace, no complaint in the inbox. Just a quietly worse product.

The performance work was the easy half; it announced itself with a nine-second page load. The rest had to be gone looking for on purpose, with a recall measurement and a token-by-token diff, because none of it was ever going to complain on its own.