nginx Stays the Front Door - Traefik as a Canary, Caches We Actually Use, and Certbot TLS
- infrastructure
- nginx
- traefik
- tls
- caching
- edge
Choosing an edge proxy is easy to treat as a brand decision. nginx is the familiar workhorse. Traefik is the label-driven option that fits Kubernetes stories. Caddy promises automatic HTTPS with less typing. On a roughly 2 GiB VPS that already terminates every Norditco hostname, the question is not which logo looks modern. It is which arrangement keeps certificates boring, keeps RAM available for product containers, and still lets us learn the next edge without risking Learn Persian with Sepideh.
This post is the decision record and the day-to-day practice behind it. We kept host nginx as the public front door. We ran a single-host Traefik bake-off behind that door for Midad. We manage TLS with Certbot on nginx for every public name. We use nginx less as a giant disk cache and more as a place to set deliberate Cache-Control for crawl files, ads verification, and static landings, while Cloudflare sits in front as a second cache that we have to purge on purpose.
If you want the broader footprint story (hot vs cold apps, on-demand wake), see edge routing and reliability. If you want why production is still Compose, see Compose versus k3s. For crawl files on hosts we cannot edit at origin, see Cloudflare Workers as a crawl surface.
What we were deciding
The server roadmap treated the edge as an open Phase 1b choice, not a migration already sold. The options on paper were:
| Option | Appeal | Cost on this box |
|---|---|---|
| Keep host nginx | Already working, low RAM, known server blocks | Manual site config and Certbot hygiene |
| Replace with Caddy | Automatic HTTPS, simpler files | Rewrite every host; learning curve during cutover |
| Replace with Traefik | Labels and Ingress habits; fits a future k3s story | Easy to fight nginx if both terminate TLS; more always-on RAM |
| Cluster ingress only | One edge inside k3s | Needs capacity we have not justified yet |
The rule we wrote before touching production was simple: lab one cold hostname behind the candidate. nginx keeps everything else. Compare RAM, TLS renewal, config size, and comfort after about a week. Only then migrate remaining hosts, one checkpoint at a time. Until that bake-off finished, the default recommendation was keep nginx as the public edge, and if a cluster ever arrives, proxy from nginx into it instead of stealing ports 80 and 443 on day one.
The bake-off we actually ran
Midad was the canary. It is cold, rarely visited, and low blast radius if the experiment failed. The accepted path after the experiment is:
Visitor → Cloudflare → nginx (TLS) → Traefik on loopback → on-demand wake proxy → Midad container.
Every other public host stays nginx → upstream with no Traefik in the middle. Traefik listens only on localhost, so nothing on the internet can hit it directly. nginx still owns certificates for midad.norditco.com. Traefik only routes by Host header to the same wake proxy the other cold apps use.
That arrangement is accepted, not a migration in progress. It gives us a live Traefik path to study and a one-command undo that puts Midad back on plain nginx. The cost is a small always-on process, on the order of a few tens of MiB, serving a single cold app. If memory stays tight, folding Midad back to nginx-only is still the simplest reclaim of RAM on the host. A new edge earns its RAM only if it clearly earns it.
What we learned about nginx versus Traefik
nginx won the public edge. It was already the map of every hostname, it used very little memory, and operators already knew how to test and reload it. Replacing it wholesale would have been a multi-host cutover for fashion.
Traefik won as a learning canary, not as a TLS owner. Dynamic routing and file-based providers are useful. Fighting nginx for certificate renewal on the same ports is not. Keeping Traefik off the public listeners avoided duplicate ACME, avoided double HTTPS configuration, and kept rollback trivial.
Caddy was not the bake-off we ran. Automatic HTTPS is attractive, but the migration cost of every server block on a live multi-product host was higher than the value of trying a second full replacement while Traefik already answered the “what does a modern router feel like” question.
Lab Traefik is separate from prod Traefik. On a laptop k3d cluster we use Traefik as the Ingress controller for playground apps and NetworkPolicy drills. That work informs capacity decisions. It does not move production hosts off nginx.
TLS: Certbot on nginx, one renewal story
Public TLS for Norditco hostnames terminates in nginx. Certificates come from the existing Certbot flow. New static product hosts follow the same pattern: DNS points at the VPS, an nginx site is enabled, TLS is issued or attached through that flow, then nginx -t and a reload.
Important consequences of the bake-off design:
- Traefik does not obtain or renew public certificates for Midad. nginx presents the cert; Traefik sees HTTP on loopback.
- Adding a hostname does not invent a second ACME client on the box. One renewal story stays easier to audit than two.
- Rollback does not touch certificates. Disabling the Traefik canary only rewires the upstream. Visitors keep the same HTTPS endpoint.
We treat certificates as ops hygiene, not as a reason to switch proxies. The pain we care about is expiry, wrong server_name, and serving the wrong chain after a partial edit. The proxy that already owns 443 is the right place to keep that pain contained.
How we use “nginx cache” in practice
People say “nginx cache” and mean three different things. We use the phrase carefully.
1. Origin Cache-Control headers (what we lean on)
For Discovery Mesh and ads verification, the important files are tiny and change on purpose: /robots.txt, /sitemap.xml, /ads.txt, /llms.txt, sometimes /llms-full.txt. On hosts we control at origin, nginx serves them with exact location = blocks and short Cache-Control values (often on the order of five minutes, sometimes up to an hour depending on the host). Some installs also set a CDN-oriented header so Cloudflare can honor a short edge TTL without inventing a long one from a missing directive.
Exact locations matter. A SPA try_files fallback that sends /llms.txt to index.html is a silent SEO bug. The fix is not a smarter CDN rule first. It is an exact nginx location that returns text/plain from a real file.
2. Cloudflare as the cache that surprises you
Cloudflare sits in front of the VPS. Origin can be correct while the edge still returns a cached 404 or an old HTML shell. That shows up after hub deploys, after robots installs, and after privacy pages land. Short origin max-age reduces how long a mistake lives. It does not remove the need to purge specific URLs when a change must be visible immediately. Our automation can purge an allowlisted set of crawl files when credentials are present on the host. HTML for the company hub often needs a broader purge after deploy because long edge TTLs keep an old Last-Modified alive.
So “we use nginx cache” in the marketing sense is incomplete. We use nginx to declare cache policy, and we treat Cloudflare as a second store that must be invalidated when policy and content change together.
3. What we are not doing
We are not running a large proxy_cache disk farm on the VPS to accelerate dynamic Learn Persian HTML. Memory and disk are scarce. The hot app stays warm as a container. Cold apps wake on demand. Static hub assets and crawl stubs are the right place for explicit TTLs. Turning the origin into a second CDN would spend disk and complexity for traffic volumes that do not ask for it.
Pattern for static product landings
New public landings (privacy pages, store listing URLs, small marketing sites) follow a shared nginx shape:
rootunder a per-host docroot- exact locations for ads, robots, sitemap, and llms with short public cache headers
/privacymapped to a real HTML file- SPA
try_filesonly for the app shell paths that need it
That pattern keeps crawl files out of the SPA fallback and keeps cache headers consistent enough that operators can reason about TTL without reading a different style per product.
nginx as the security boundary, not only the router
Edge choice is also a hardening choice. After localhost-only Docker binds, application ports are not reachable from the internet. Public entry remains 80/443 through nginx. Traefik’s entrypoint stays on loopback for the canary. Skipping the front door by guessing a published container port is not part of the design.
That is why “move everything to Traefik” and “expose Traefik on 443” were never free upgrades. They would change the threat model and the certificate model at the same time. The canary keeps the learning value without widening the listening surface.
Day-to-day operations that follow from the design
A few habits fell out of the edge decision and are now normal work:
- Map first. We keep an edge map of hostname → proxy path → upstream class (hot, cold, static). New hosts get a row before they get traffic.
- Canary before fashion. One cold host behind a candidate proxy. Measure. Undo should be one reviewed automation step, not a manual forensics exercise.
- One TLS owner. Certbot and nginx own public certificates. Anything behind them speaks HTTP on localhost.
- Cache headers are product surface. robots, ads, llms, and privacy are part of Discovery Mesh and store readiness. Exact locations and short TTLs are reviewed like product code.
- Purge is part of deploy. Origin green is not the same as visitor green when Cloudflare has a HIT. Especially for HTML and for files that previously 404’d at the edge.
- Hits are a proxy, not truth. Morning nginx hit digests help capacity and experiment readouts. They include bots. They do not replace analytics when we need humans.
What we would change later
The decision is stable, not eternal.
- If Traefik’s RAM stops paying for itself, Midad goes back to nginx-only.
- If a k3s pilot ever serves a real hostname, the preferred first step is still nginx → cluster, not Traefik stealing the public listeners on day one.
- If certificate automation becomes the main pain, we would evaluate Caddy or ACME inside a single owner carefully, not by running two ACME clients in parallel.
- If traffic patterns demand origin disk caching, we would add it for specific static paths with budgets and eviction, not as a default for every proxy_pass.
Closing
nginx versus Traefik was not a religious argument for us. It was a bake-off with a cold canary, a hard RAM budget, and a requirement that TLS renewal stay boring. nginx remains the public front door and the Certbot termination point. Traefik remains a loopback router we can study and undo. Cache work is mostly honest Cache-Control on the files crawlers and ad networks actually fetch, plus disciplined Cloudflare purges when the edge remembers yesterday.
That combination is less glamorous than a full edge rewrite. It is also the one that kept Learn Persian, the hub, and the cold apps behind one door while we still learned the next door in private.
