Your SPA is invisible, and Googlebot is the only crawler that doesn't notice

This site is a Svelte single-page app behind a Tornado backend: one bundle, one client-side router, zero server-side rendering. Fetch any route the way something without a browser would, and here's what comes back:

<body>
  <div id="app"></div>
</body>

That's it. No title, no description, no text — everything a visitor sees gets drawn in by JavaScript after the page loads. Which is fine, right up until something reads the page without running that JavaScript. A lot of things don't.

Who's actually looking

"Search and social" gets treated as one bucket called SEO, but the clients involved don't behave the same way:

  • Google runs an evergreen, current Chromium under the hood. It'll fetch the HTML, notice there's nothing there, and queue the page for a second pass where it actually executes the JS and indexes what renders. Slower, but it works.
  • Bing does this inconsistently. Sometimes yes, often no.
  • Link unfurlers — Slack, LinkedIn, X, iMessage, Discord — run nothing. They fetch the HTML, read whatever's in <head>, and render a card from that. No second pass, no JS.
  • AI crawlers — GPTBot, ClaudeBot, PerplexityBot and friends — mostly behave like the unfurlers: fetch, read, move on.

For a small site, that second group usually matters more than organic search — most traffic arrives as a link someone pasted into a chat, not a search result. A blank <head> means that link unfurls with no title, no summary, no image, just a bare gray box.

Worth being specific about why each one behaves the way it does, since it's not all the same failure. Bing actually has a render pipeline, just a tighter compute budget than Google's — shorter timeouts, a smaller share of pages actually get the JS pass, which is what "inconsistent" cashes out to. Unfurlers aren't degraded crawlers at all; they're not browsers to begin with, built to paint a preview card in under a second, so there's no JS engine behind them to fail at anything. Most AI crawlers doing bulk collection (GPTBot, ClaudeBot, CCBot) are the same story as unfurlers — plain fetchers, by design, not by limitation. The one wrinkle: a live, on-demand fetch triggered by an actual user query is a different code path than the bulk crawler for some of these, and more likely to render — "does this bot render JS" and "does this product render JS when someone pastes a link into it right now" aren't always the same answer.

Two ways to fix it, and which one to reach for

The complete fix is prerendering: generate real HTML for every route at build time, ship that, and let the SPA hydrate on top for client-side navigation. That's the right call if you need crawlers to actually read your page content. It's also a real project — your router almost certainly runs client-side only, so this means driving a headless browser at build time, not just calling your framework's own server-render function.

The smaller fix, and the one worth reaching for first: have the server substitute correct metadata into the page shell before it goes out, based on which route was requested. This doesn't give a crawler any body text. It does mean every route gets an accurate title, description, and social card — which is the entire problem for anything that only reads <head> and never renders the page.

The pattern: swap a marked block in the shell

Every request for a page still gets served the same index.html — the SPA doesn't change. Mark a block in it that's safe to replace per request:

<!-- web/index.html -->
<!-- seo:start -->
<title>Pearachute | Linux, Python & AI Systems</title>
<meta name="description" content="..." />
<!-- seo:end -->

Keep a table of what each route should say:

# seo.py
PAGES = {
    '/': HOME,
    '/kanban': {
        'title': 'pkanban | Pearachute',
        'description': 'A kanban board you drive from the terminal...',
        'path': '/kanban',
    },
    # ...
}

And a function that reads index.html, finds the two marker comments, and replaces everything between them with the tags for the requested route — falling back to the file unchanged if the markers are missing, so a build that ships without them degrades instead of 500ing:

def shell(route, path):
    with open(path, encoding='utf-8') as f:
        html = f.read()

    head, sep, rest = html.partition(START)
    if not sep:
        return html

    _, sep, tail = rest.partition(END)
    if not sep:
        return html

    return head + tags(route) + tail

Wire that into whatever handler is already serving the SPA fallback — here, a StaticFileHandler subclass that serves index.html for any path with no real file behind it:

class SpaStaticFileHandler(tornado.web.StaticFileHandler):
    async def get(self, path, include_body=True):
        if path in ('', 'index.html'):
            return self.write_shell('/')
        try:
            await super().get(path, include_body)
        except HTTPError as e:
            if e.status_code == 404 and not path.startswith(('assets/', 'data/')):
                return self.write_shell('/' + path)
            raise

    def write_shell(self, route):
        html = seo.shell(route, join(self.root, 'index.html'))
        self.set_header('Content-Type', 'text/html; charset=UTF-8')
        self.write(html)

Two things to get right when you do this:

Override compute_etag. StaticFileHandler's default etag comes from self.absolute_path, which only gets set when it actually streams a file off disk through the normal path. Once you're writing a dynamically assembled body instead, that attribute is never set, and Tornado will throw trying to read it on every single request. Fall back to hashing the response body instead — which is what you want anyway, now that the body varies by route:

def compute_etag(self):
    if getattr(self, 'absolute_path', None) is None:
        return tornado.web.RequestHandler.compute_etag(self)
    return super().compute_etag()

Send Cache-Control: no-cache on the shell. It references hashed asset filenames from your build; a stale cached copy of the shell can point at bundles a deploy has already replaced. The bundles themselves stay cacheable forever — it's only the shell that needs revalidating every time.

Keep the browser tab title in sync during client-side navigation

The substitution above only runs on the initial server request. Every route after that is the client router swapping components, and the server never sees it. If you want the tab title to stay correct while someone clicks around, that's a separate, purely client-side concern.

Do it with one reactive title at the app root, driven by the current path, rather than a <svelte:head><title> in each route component:

<!-- App.svelte -->
const TITLES = {
  '/kanban': 'pkanban | Pearachute',
  '/pitchperfect': 'Pitch Perfect | Pearachute',
};

$: title = TITLES[$router.path] || HOME_TITLE;

<svelte:head><title>{title}</title></svelte:head>

A <svelte:head> owned by an individual route component only removes its own tag when that component unmounts — it doesn't restore whatever title was there before. Navigate away and the tab is stuck on the last route's title. One title, computed reactively at the top, sidesteps the whole problem.

The canonical tag has to vary by route too

A <link rel="canonical"> is a direct instruction to a search engine: "this page is a duplicate, index the other URL instead." If you add Open Graph and canonical tags to your shell but leave them static, every non-home route ends up serving:

<link rel="canonical" href="https://example.com/" />

which tells Google, in as many words, not to bother indexing anything but the homepage. That's worse than having no canonical tag at all — it's an active instruction rather than a missing one, and it'll look completely fine if you only ever read the source rather than the actual served output per route. The canonical, like the title and description, has to come out of the same per-route table.

What this doesn't do

Worth being precise about the boundary. This gives every crawler and every unfurler an accurate title, description, and social card. It does not give any of them body text — the HTML they receive still has an empty <div id="app"> underneath that correct <head>. If a route is genuinely interactive — a D3 chart, a form, anything the client actually computes — getting its body text indexed means the full prerendering project: driving a headless browser at build time to capture what it renders. There's no shortcut around that one.

There's a cheaper way out for the specific case of a route that was never interactive to begin with. A blog or docs section backed by a folder of markdown files doesn't need a browser to find out what it looks like — the content is already fully known, in your backend language, at request time. Pull that section out of the SPA entirely and have the server render the real page directly, the way this site's own /notes does: real <h1>s and paragraphs in the response body, a genuine 404 for a slug that doesn't exist, and no <div id="app"> anywhere in the output. Same table-driven idea as the rest of this post — a listing page merges each item's own metadata into the same per-route lookup instead of a hand-typed table entry — just carried one step further, into the body instead of stopping at the <head>.