Skip to main content
    Norditco logo
    All engineering posts
    Published November 8, 2026Norditco Engineering

    Cloudflare Workers as a Crawl Surface for Origins We Cannot Edit

    • infrastructure
    • cloudflare
    • seo
    • discovery
    • architecture

    Most crawl policy lives in a handful of files at predictable paths. /robots.txt tells crawlers what they may fetch. /sitemap.xml tells them what exists. /ads.txt tells ad buyers who is allowed to sell inventory on the domain. These files are tiny, and when you own the origin they are trivial to ship. You commit them to the repository, deploy, and move on.

    Two of our products do not fit that pattern. Corsorr runs on Blogger behind a custom domain. Trip Planner is a single-page app on Vercel whose web source is not in the repository our automation works from. In both cases the origin cannot serve the files we need, and in neither case is rewriting the origin the right fix. This post describes how we put those files at the Cloudflare edge instead, what the Workers look like, where the design has sharp edges, and which parts are not yet live.

    It is a companion to Building the Norditco Discovery Mesh, which describes the hub-and-spoke structure these files exist to support. That post introduced the Worker approach in a few paragraphs. This one is the detailed version.

    The problem in concrete terms

    The two products fail in different ways, and the difference shapes the two Workers.

    Corsorr is a Blogger site served at corsorr.norditco.com. Blogger does publish a robots.txt, so the path does not 404. What it publishes is thin. A request today returns an allow-all stanza with empty Disallow and Allow values and a Sitemap line, and nothing else. There are no per-crawler groups, so there is no written policy for AI search or training crawlers. Blogger also generates a sitemap for us, which is fine, so the gap is narrow. It is the policy file only.

    Trip Planner is a client-rendered SPA on Vercel at tripplanner.norditco.com. A Vercel project without a matching static file or rewrite returns a platform NOT_FOUND for unknown paths. That is what /robots.txt, /sitemap.xml and /ads.txt return today. A crawler that gets a 404 for robots treats the site as unrestricted, which is survivable. But it also means we cannot publish a sitemap pointer, we cannot state a crawler policy, and the ad verification file that AdSense expects is missing.

    The honest options for each were:

    OptionCorsorrTrip Planner
    Edit the originPossible through Blogger's custom robots field, but limited to one fileNeeds access to the web repo and a redeploy through a pipeline we do not own
    Redirect the path to the hubWrong. Robots must be served at the host it governs, not redirectedSame
    Serve the file at the edgeWorksWorks

    Robots rules apply per host, and crawlers expect a 200 response for the file at that host. Redirecting to another hostname is the kind of move that works in some crawlers and fails in others. Serving the real body from the right host is the least surprising behavior.

    Both hostnames already sit behind Cloudflare. That makes the edge the one layer we control for both products.

    The request path

    A Cloudflare Worker bound to a route runs before the request is forwarded to the origin. It can answer directly, or it can call fetch(request) to pass the request through unchanged. Our Workers use both behaviors. A small allowlist of paths gets a response generated at the edge, and every other path continues to the origin.

    Architecture diagram. Expand for a larger view with zoom.

    The origin never learns that the Worker exists. It does not need a new route, a new file, or a new deploy. Vercel keeps serving the SPA shell and Blogger keeps serving posts. If we removed the Worker tomorrow, the origin would behave exactly as it did before we added it.

    That property is the main reason this approach is attractive for products we cannot fully edit. The blast radius of the change is a single route binding on a zone we own, and reversing it is deleting that binding.

    The Trip Planner Worker

    The Trip Planner Worker owns three paths and passes everything else through. The structure is deliberately flat, with no framework and no dependencies.

    const ADS_TXT = `google.com, pub-XXXXXXXXXXXXXXXX, DIRECT, f08c47fec0942fa0
    `;
    
    const ROBOTS_TXT = `# Trip Planner: open crawler policy (search + AI search/training).
    
    User-agent: Googlebot
    Allow: /
    
    # ...Bingbot, Twitterbot, facebookexternalhit, GPTBot, ChatGPT-User,
    # OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended
    
    User-agent: *
    Allow: /
    
    Sitemap: https://tripplanner.norditco.com/sitemap.xml
    `;
    
    const SITEMAP_XML = `<?xml version="1.0" encoding="UTF-8"?>
    <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
      <url><loc>https://tripplanner.norditco.com/</loc></url>
    </urlset>
    `;
    
    const TEXT = {
      "content-type": "text/plain; charset=utf-8",
      "cache-control": "public, max-age=300",
    };
    const XML = {
      "content-type": "application/xml; charset=utf-8",
      "cache-control": "public, max-age=300",
    };
    
    export default {
      async fetch(request) {
        const { pathname } = new URL(request.url);
        if (pathname === "/ads.txt") return new Response(ADS_TXT, { headers: TEXT });
        if (pathname === "/robots.txt") return new Response(ROBOTS_TXT, { headers: TEXT });
        if (pathname === "/sitemap.xml") return new Response(SITEMAP_XML, { headers: XML });
        return fetch(request);
      },
    };
    

    The publisher identifier above is a placeholder. The real value is the same one the rest of the company uses, and it appears in the ops repository next to the Worker.

    A few decisions in that file are worth explaining.

    Bodies are constants in the module. There is no KV namespace, no environment variable, and no fetch to a config service. The policy ships with the code, so the version running at the edge is always the version in the repository. A crawl file that can drift away from source control is the same drift problem the Discovery Mesh registry exists to prevent.

    Exact path matching. The Worker compares pathname with strict equality. /robots.txt/ and /Robots.txt fall through to the origin. That is on purpose. We would rather serve exactly the well-known paths crawlers request than invent a normalization layer whose edge cases we then have to test.

    Five-minute cache. The max-age=300 header is short because crawl policy is the one class of static file where staleness is a bug. If we change a robots rule, we want the new rule visible to crawlers within minutes, without a manual purge. The cost is more requests reaching the Worker, which for files this small is negligible.

    A one-URL sitemap. The sitemap stub lists only the home page, and that is a statement about the product, not a shortcut. Trip Planner is a client-rendered SPA with essentially one crawlable entry point today. Listing URLs that do not resolve to distinct server-rendered content would produce a sitemap that is longer and worse. When the app gains real, indexable routes, the stub should be replaced by a generated sitemap, and the Worker is the wrong place to maintain a long list by hand. We note that as a limit of this approach below.

    An explicit policy pack. The robots body names each major crawler, including OAI-SearchBot, PerplexityBot, ClaudeBot, GPTBot and Google-Extended, with an allow rule, and ends with the wildcard group and a Sitemap line. As with the hub, the explicit list is functionally redundant with the wildcard. We keep it because the file is also the written policy, and the Discovery Mesh post explains why that policy is open.

    The Corsorr Worker

    Corsorr's Worker is narrower, because the gap is narrower. It handles one path and is bound to one path.

    const ROBOTS_TXT = `# Corsorr: open crawler policy (search + AI search/training).
    
    User-agent: Googlebot
    Allow: /
    
    # ...Bingbot, GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot,
    # PerplexityBot, Google-Extended
    
    User-agent: *
    Allow: /
    
    Sitemap: https://corsorr.norditco.com/sitemap.xml
    `;
    
    export default {
      async fetch(request) {
        const { pathname } = new URL(request.url);
        if (pathname === "/robots.txt") {
          return new Response(ROBOTS_TXT, {
            headers: {
              "content-type": "text/plain; charset=utf-8",
              "cache-control": "public, max-age=300",
            },
          });
        }
        return fetch(request);
      },
    };
    

    The Sitemap line points at the sitemap Blogger already generates, so the Worker does not stub one. It only replaces the policy text that Blogger's default lacks.

    The route for this Worker is the single path corsorr.norditco.com/robots.txt, not a wildcard. That is the most conservative binding available. Requests to posts, feeds, images and the Blogger sitemap never enter the Worker at all, so there is no code path in which this Worker can break post rendering. For Trip Planner the binding has to be tripplanner.norditco.com/*, because three distinct paths are served and a wildcard route is simpler than three route entries. The cost of the wildcard is that every request, including HTML and assets, now passes through Worker code. The pass-through is a single fetch(request) call, but it is still a hop and it is still a dependency.

    There is also a simpler alternative for Corsorr, and we keep it documented. Blogger has a custom robots field under crawler settings, and pasting the same body there achieves the same result without any Worker. We prefer the Worker for one reason. It keeps the policy in version control alongside the other product policies, rather than in a settings screen that nobody reviews. If we ever stop wanting to run an edge dependency for a one-file gap, the field is the fallback.

    Where the Workers sit in the mesh

    The Discovery Mesh has a hub and eight spokes, and spokes differ in who owns the files. The useful question for each is whether the origin can serve its own crawl files. If it can, we ship the files through that product's own deploy path. If it cannot, a Worker covers the gap.

    Architecture diagram. Expand for a larger view with zoom.

    The shape of the diagram is the point. The edge-owned group is small, and it exists because of specific constraints, not because we prefer the edge. Where an origin can serve robots.txt through its own pipeline, it should, because that keeps the file next to the application it describes. The Workers are an exception path and we treat them as one. They stay in the ops repository, next to the registry notes that explain why each exists.

    That registry detail matters. The product registry has a notes field for operational facts that should not appear in public output. The Corsorr and Trip Planner entries are where the reason for each Worker is recorded, so a future engineer reading the registry learns that these two hosts depend on an edge route and does not discover it during an incident.

    What is installed, and what is not

    This section is the one we would most want to read if we were evaluating the design, so we will be direct.

    The Worker source for both products is written, reviewed and stored in the ops repository. Neither Worker is attached to its hostname yet. Attaching a route on the Cloudflare zone is a privileged change to shared infrastructure. We keep that step human-gated, and the machine our automation runs on has no zone credentials by design. So the final step, creating the Worker in the dashboard and adding the route, is a manual action that has not been completed.

    We checked the public URLs on the day this article was published, and the results match that status.

    URLResultWhat it means
    tripplanner.norditco.com/ads.txt404, Vercel NOT_FOUNDScript uploaded; route not attached
    tripplanner.norditco.com/robots.txt404, Vercel NOT_FOUNDScript uploaded; route not attached
    tripplanner.norditco.com/sitemap.xml404, Vercel NOT_FOUNDScript uploaded; route not attached
    corsorr.norditco.com/robots.txt200, Blogger default stanzaScript uploaded; route not attached
    corsorr.norditco.com/sitemap.xml200, Blogger-generatedOrigin-owned, as intended

    As of 8 November 2026 the Worker scripts for Trip Planner and Corsorr are in the Cloudflare account (uploaded from the ops tree via an allowlisted install path). Public responses still match the table above, which is expected: attaching the zone route remains a human-gated console step. Until that route exists, crawlers never enter the Worker.

    The installation itself is short.

    1. Confirm the Worker script for the product exists in the Cloudflare account (source of truth stays in the ops repository).
    2. Under the Worker's triggers, add the route. For Trip Planner this is the hostname with a wildcard path. For Corsorr it is the hostname with the exact /robots.txt path (or the broader path set if privacy and llms.txt are also edge-served).
    3. Confirm the hostname is proxied through Cloudflare. A route on a DNS-only record never executes.
    4. Purge any cached 404s for the well-known paths, then run the verification block below.

    The proxy requirement in step 4 is the usual reason a correct Worker appears to do nothing. Both of our hostnames are already proxied, but any new product subdomain needs the check.

    Verification

    We verify against the public URL, never the origin, because the edge is what crawlers see. After install, each product gets a short set of checks.

    # Trip Planner: expect 200 and the seller line
    curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' https://tripplanner.norditco.com/ads.txt
    curl -sS https://tripplanner.norditco.com/ads.txt
    
    # Policy bodies and sitemap pointer
    curl -sS https://tripplanner.norditco.com/robots.txt
    curl -sS https://corsorr.norditco.com/robots.txt
    
    # Pass-through still works: the app itself must still return its HTML
    curl -sS -o /dev/null -w '%{http_code}\n' https://tripplanner.norditco.com/
    curl -sS -o /dev/null -w '%{http_code}\n' https://corsorr.norditco.com/
    

    The last two lines matter more than they look. A Worker that serves the right robots.txt and breaks the home page is a regression, and the wildcard route on Trip Planner puts every request through the Worker. The home page check is the proof that pass-through behaves.

    These checks belong in the quarterly crawl audit described in the Discovery Mesh post, which fetches every public crawl file across the spokes and diffs it against the registry. A Worker silently unbound by a zone change would show up there as a robots file that suddenly stops matching.

    Trade-offs and failure modes

    An edge Worker is not free, and it would be misleading to describe it as such.

    It adds a dependency on the request path. For Corsorr, the exact-path route means almost no traffic enters the Worker, so the exposure is nearly zero. For Trip Planner the wildcard route means every request does. Cloudflare lets you choose a route's failure mode, so an exception or platform fault can either fail open and send the request to the origin, or fail closed and return an error. For a wildcard route in front of a product people use, fail open is the correct choice. We would accept a crawler seeing a 404 for robots during a Worker outage before we would accept users seeing an error page.

    The Worker is a second place a policy can live. If someone later edits the hub's robots file and forgets that product spokes have their own copies, the policy diverges. The mitigation is process rather than code. The policy packs live in one repository, they share a structure, and the quarterly audit compares them.

    Hand-written sitemaps do not scale. The one-URL stub is right for a SPA with one entry point. It is wrong for a product with hundreds of pages. At that point the correct move is for the origin to generate its sitemap, or for the Worker to fetch a generated one from a controlled source. We have deliberately not built that, because there is no use case for it yet and speculative infrastructure is its own kind of debt.

    ads.txt has its own rules. The file we serve for Trip Planner carries the same publisher identifier used across the company. Only norditco.com is registered as a site in AdSense, and subdomains inherit from it, so the Worker's job is to make the verification file reachable, not to create a separate AdSense property. We keep the content a single line on purpose. Anything more should come from the ad platform's own guidance, not from us improvising.

    Caching cuts both ways. A five-minute max-age means a change reaches crawlers quickly. It also means a mistake does. Because the bodies are constants reviewed in the repository, we accept that trade, and a bad deploy is fixed by redeploying a corrected module rather than waiting out a long TTL.

    Why this fits how we work

    The pattern is deliberately unambitious. Each Worker is under a hundred lines, has no dependencies, holds no secrets, and reads like a config file. That is the property we want for anything that sits in front of a production host and that a single engineer may need to reason about at an inconvenient hour.

    It also fits the way we keep roles apart. Content and policy are written and reviewed in the repository. The act of binding a Worker to a hostname is a privileged edge change, so it is performed by a person with zone access and verified afterward with public checks. Automation drafts and tests the module. A human flips the route. We are comfortable with that split for shared infrastructure, even though it is the reason the status table above still shows 404s.

    What comes next

    1. Attach the Cloudflare zone routes for both Workers (scripts are already uploaded), then run the verification block and record the result in the ops documentation.
    2. Add the Worker URLs to the crawl audit as explicit expectations, so a missing route fails a check instead of waiting for someone to notice.
    3. Add visible "By Norditco" links on Corsorr and Trip Planner pages, so the hub relationship appears in HTML as well as in schema.
    4. Revisit the Trip Planner sitemap once the SPA exposes more than one indexable route, and move to a generated file at that point.
    5. Decide whether Corsorr should drop its Worker in favor of Blogger's custom robots field. The edge copy is the better-governed option today; the settings-screen option is simpler to operate.

    The underlying idea is simple. Crawlers should find the same well-known files on every Norditco host, regardless of how that host is built or who can edit it. A Worker is the lightest way to guarantee that for the two spokes where the origin cannot, and keeping the surrounding facts honest, including the fact that the route is not yet attached, is what makes the rest of the mesh worth trusting.