Building the Norditco Discovery Mesh for Search Engines and AI Crawlers
- seo
- discovery
- ai
- architecture
Norditco ships eight consumer products on eight different hosting arrangements. Learn Persian with Sepideh runs as a content-heavy web app with an Android sibling. Corsorr is a Blogger site behind Cloudflare. Trip Planner is a single-page app on Vercel whose web source does not live in our ops tree. Midad is an Arabic writing tool that wakes on demand on a VPS. Audio Player Pro is a Chrome extension with a landing page. Farsi TTS is a Piper-based speech tool. Tourist Talk is a Play Store listing with a thin web landing page.
Each of these has its own deploy path, its own owner on a given week, and its own idea of what a robots.txt looks like. From the outside, a crawler sees a company called Norditco and a scatter of subdomains with no obvious relationship. A language model asked "who makes Learn Persian with Sepideh" has to guess, and guessing is where wrong answers come from.
The Discovery Mesh is our response. It is a small set of conventions, one source of truth, and a few pieces of infrastructure that make the relationships between our properties explicit to machines. This post describes the design, what we shipped, what we got wrong on the first pass, and how we judge whether any of it works.
What we are and are not trying to do
The starting position matters, because it removes a lot of bad ideas early. Google has been direct that optimizing for its generative features is still search optimization. llms.txt is not a ranking signal for Google Search. No other major provider has committed to treating it as one either. So the mesh is not an attempt to trick a model into mentioning us.
What it does try to guarantee is narrower. When any crawler, whether Googlebot, Bingbot, OAI-SearchBot or PerplexityBot, arrives at any Norditco property, it should be able to answer four questions from the page itself and from a short walk along explicit links. What is this thing called, who publishes it, where is the canonical URL, and what else does the same publisher make. Everything else (content quality, citations from other sites, response time) is ordinary web work that no schema file replaces.
We wrote down a short set of principles before building anything, and they have held up. Pages are written for the people who use the product, and machine-readable layers describe what is already visible. There is one canonical URL per concept, and we do not stand up parallel "AI landing pages". Product names are byte-identical across the web app, the store listing, social profiles and structured data. Structured data never claims anything that is not on the page, which rules out invented ratings and awards. And operational hosts such as the API, the automation server and the monitoring dashboard are kept out of every index on purpose.
Hub and spokes
The structural idea is an entity hub with product spokes.
norditco.com is the hub. It is the only place that describes Norditco as an organization, and it carries the list of everything the organization owns. Each product is a spoke with its own host, its own sitemap and its own schema, but the spoke points back at the hub as its publisher, and the hub points out to every spoke.
The two directions are deliberate. A crawler that lands on a spoke first, which is the common case since most traffic arrives through a product query, can climb to the hub through the publisher link and learn the full product family. A crawler that lands on the hub can walk down. Either path ends in the same graph.
Not every spoke carries equal weight in how we invest. Learn Persian with Sepideh is the largest content surface we run, with courses, articles, a FAQ and topic hubs under /learn, so it gets the deepest treatment: FAQPage markup that mirrors visible questions, an answer-first article template, and its own llms-full.txt. Smaller tools such as Audio Player Pro and Farsi TTS get the same skeleton at lower depth: a landing page with a unique title and description, a robots.txt, a sitemap, one SoftwareApplication node.
One registry, generated outputs
The first version of this existed as hand-edited fragments. A product description lived in the hub's JSON-LD, again in llms.txt, again in a Play Store listing, and again on the product's own site. They drifted within a week. Drift is the specific failure that hurts here, because the useful property of an entity graph is that every source agrees.
So the source of truth is a single YAML file in our ops repository, product-registry.yml. It holds the company block (name, legal name, URL, social profiles) and one entry per product with an id, display name, priority, type, web URL, store URLs and a one-sentence description. It also holds a notes field for operational facts that should not leak into public output, for example that Corsorr's robots file is Blogger's default and cannot be extended without a Worker.
The hub consumes the registry through a typed catalog in src/config/site.ts. That catalog drives the JSON-LD graph, so adding a product is one entry and the Organization node, the Offer list and the SoftwareApplication node all appear together. The shape of a generated product node is plain:
{
"@type": "SoftwareApplication",
"@id": "https://norditco.com/#product-learn-persian-with-sepideh",
"name": "Learn Persian with Sepideh",
"url": "https://learnpersianwithsepideh.norditco.com/",
"applicationCategory": "EducationalApplication",
"provider": { "@id": "https://norditco.com/#organization" },
"isPartOf": { "@id": "https://norditco.com/#organization" },
"sameAs": [
"https://play.google.com/store/apps/details?id=com.ahmed_mansour_ouda.LearnPersianWithSepidehV4",
"https://www.facebook.com/learnpersianwithsepideh"
]
}
Using @id references instead of nesting full objects is what makes this a graph. The Organization says it owns the product by id, the product says the Organization is its provider by id, and a consumer that merges the nodes gets a consistent picture regardless of which side it reads first.
Two details are worth flagging for anyone copying the pattern. First, sameAs is for identity, not for any link you like. We use it for the Play listing, the Chrome Web Store page and the product's Facebook Page, which are the same product on other platforms. Second, the hub JSON-LD exists twice: once as a TypeScript object for the React tree and once inline in index.html. Many crawlers do not execute JavaScript, or execute it late, and the inline copy means the graph is present in the first byte of the response. The cost is a duplicate that has to stay in sync, which we handle with a comment at the definition and a review checklist item. A build step that injects the object into the HTML would be cleaner, and it is on our list.
A robots policy that says what it means
We chose an open policy. Search crawlers and AI crawlers are allowed on consumer surfaces, including the ones used for model training.
That is a business decision, not a technical default, and it deserves a sentence of justification. Our products are free or low-cost tools for people who are looking for exactly what we make. A learner who asks an assistant how to start with Persian is a learner we want to reach, and being absent from the training data of those assistants has a cost that being present does not. A publisher whose revenue depends on exclusive text would reasonably decide the other way. The important part is that the decision is written down, because the alternative is a robots.txt that nobody can explain.
The hub file lists user agents explicitly rather than relying on a single wildcard allow:
User-agent: Googlebot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
# ...plus Bingbot, ChatGPT-User, Applebot-Extended, CCBot, and others
User-agent: *
Allow: /
Sitemap: https://norditco.com/sitemap.xml
An explicit list is functionally redundant with User-agent: *, and we know that. We keep it because the file doubles as documentation. Anyone who later wants to block training crawlers while keeping search crawlers can see which agents belong to which group and change the lines without researching the vocabulary again. The distinction is real and often missed. OAI-SearchBot and PerplexityBot fetch content to answer a live query and cite it. GPTBot, Google-Extended and CCBot collect material for training or grounding. They are separate user agents because the operators expect sites to treat them separately.
The sitemap is kept short and honest. The hub lists four URLs. We exclude anything thin or authenticated, and we do not emit lastmod values we cannot stand behind, because a sitemap that claims everything changed yesterday teaches crawlers to ignore the field.
The same policy has a negative half that matters as much as the positive one. API, automation, and monitoring hosts are separate with Disallow and noindex. They never appear in a sitemap, in llms.txt, or in schema. The mesh is a map of what we want found, and the absence of those hosts is part of the design.
llms.txt as a courtesy map
llms.txt sits at the root of the hub as a curated Markdown index: a one-paragraph description of the company, links to the site's main pages, a canonical list of products with a single sentence each, entity and legal details, and social profiles. A longer llms-full.txt carries more prose for agents that want to read rather than follow links.
We treat it as a courtesy and not as infrastructure. Some agent frameworks and developer tools read it. Google Search does not. That asymmetry shaped three choices.
It is generated from the same registry as the schema, so it can never claim a product the graph does not know about. It is curated rather than exhaustive. A file that lists every URL is a worse sitemap, and the point of llms.txt is that a reader with a small context budget gets the right ten links. And its cache lifetime is short. Early on, the hub's nginx served it with the same long cache as static assets, so a registry change took a day to reach the edge. We now serve /llms.txt, /llms-full.txt and /robots.txt with a short cache at the origin and purge the Cloudflare edge on deploy. Crawl policy files are the one class of static asset where staleness is a bug.
Origins we cannot edit
The registry and the hub were the easy half. The hard half was spokes where we do not control the origin's file serving.
Corsorr is a Blogger site on a custom domain, fronted by Cloudflare. Blogger serves a default robots.txt with a basic allow, and its custom robots field is limited. Trip Planner is a Vercel single-page app. Its web source is not in the repository our automation works from, and an unknown path on a Vercel SPA returns a platform 404, which is what /robots.txt and /ads.txt were returning. A crawler that gets a 404 for robots treats the site as unrestricted, which is acceptable, but it also means we cannot publish a sitemap pointer or any AI crawler lines, and our ad verification file was missing too.
Both cases have the same shape. The origin cannot serve a handful of well-known paths, and the zone is already on Cloudflare. A Worker routed to those paths solves it without touching the origin.
The Trip Planner Worker serves /robots.txt, /sitemap.xml and /ads.txt and forwards everything else untouched. The Corsorr Worker is narrower, bound to /robots.txt only, since that is the single gap. In both cases the Worker source lives in the ops repository next to the policy pack it serves, so the file a crawler sees is reviewed in a pull request like any other change.
We would rather not pretend this was fully closed. At the time of writing, the Workers are written, and attaching them to the Trip Planner and Corsorr hostnames is still a deliberate manual step on the Cloudflare zone. Edge configuration is shared infrastructure, so we keep privileged zone changes human-gated and verify afterward with a short set of public URL checks against /robots.txt, /ads.txt where applicable, and the product home page.
What went wrong on the first pass
Three problems cost us time, and each changed how we work.
Tourist Talk's application id drifted. The live Play package is com.norditco.touristtalk, but a variant with a numeric suffix circulated in our own notes and was easy to confuse with it. A schema node with a store URL that resolves to nothing is worse than no node. The registry now carries a note that the id in the app configuration must match the store, and the hub catalog uses the live one.
Several products were marked as operatingSystem: "Web" because the schema generator takes one value for the whole catalog. That is accurate for most of them and wrong for an Android listing and a Chrome extension. The fix is to make the field a per-product property in the registry, and it is the next change to the generator.
Cache and edge purges bit us on Midad. The origin's robots.txt and sitemap were corrected in the app repository, but Cloudflare kept serving the old file until we purged. Any change to a crawl policy file now ends with a purge and a curl against the public URL, not the origin.
What we measure
We separate signals we can instrument from signals we can only sample.
Instrumented signals come from Google Search Console and Bing Webmaster Tools, set up per property. For Learn Persian with Sepideh we track indexed counts for /articles/*, /courses/* and /faq, and impressions on brand queries and on the broader language-learning queries we want to appear for. For every spoke we track whether the public robots.txt returns 200 with the expected content, and whether the hub's JSON-LD parses and validates in the Rich Results test after each deploy. Traffic proxies come from server logs and Play Console install data. Analytics events fire only after consent, which was in place before we added any of this.
Sampled signals are about generative answers, where no dashboard exists. On a monthly cadence we ask a fixed set of questions across ChatGPT, Perplexity and Claude, such as how to learn Persian online and what Learn Persian with Sepideh is, and record whether Norditco is cited, whether the product name and publisher are stated correctly, and whether the URL is the canonical one. The log is qualitative and noisy, and we read it for direction. Two consecutive months of wrong publisher attribution would send us back to the schema, not to the content.
What comes next
Some of the named gaps closed in November 2026: Learn Danish with Freja now has FAQPage markup, topic hubs, a Freja-only Study with AI prompt, and tenant-aware llms.txt; a registry-driven crawl-surface audit script landed with a first report. Tourist Talk and Know Before You Go have static landing and privacy drafts ready for DNS. The remaining near-term work is still concrete.
- Finish the Cloudflare route installation for the Trip Planner and Corsorr Workers (Worker sources now include
/privacyand/llms.txt), and add reciprocal "By Norditco" footers on those two products so their pages link up to the hub in visible HTML as well as in schema. - Move
operatingSystemand a store-specificdownloadUrlinto the registry so the Android listing and the Chrome extension stop being described as web apps. - Replace the duplicated JSON-LD with a build step that renders the registry into
index.html, removing the manual sync. - Enable DNS and nginx for the Tourist Talk (and Know Before You Go) static hosts so the drafted landing and privacy pages are public, not only files in ops git.
- Purge Cloudflare for spokes where origin crawl files are ahead of the edge (TTS and Midad
/llms.txtand/privacyhave been the recurring examples). - Re-run the quarterly crawl audit after those edge installs, and keep the citation spot-check log filled with real assistant samples—not only web-index proxies.
The audit remains the piece we expect to pay back most. The mesh is only as good as its agreement with itself, and agreement decays quietly unless something checks it.
