The cast

Before anything else, it helps to picture the stack. Almost every problem in this story came from one of these layers being confused about another.

   Browser
      │
      │  HTTPS
      ▼
   ArvanCloud edge          185.143.233.238 / 185.143.234.238
   (CDN — terminates TLS,   two edge IPs, round-robin
    caches, then re-asks
    your server)
      │
      │  HTTP or HTTPS?  ← this question caused three separate bugs
      ▼
   Your origin server       185.204.171.154
   nginx 1.24 on Ubuntu
      │
      │  HTTP to localhost
      ▼
   Docker containers        :3001 Next.js  ·  :4001 backend
                            :3000 blog     ·  :4000 GraphQL
                            :5432 Postgres

Two things about this shape matter for everything that follows.

The CDN is a second web server you don't control. It has its own opinions about protocols, its own cache, and its own error pages. When it can't reach you, it invents a 502 — one that never touches your nginx and never appears in your logs.

"Origin protocol" is a real setting with real consequences. The edge speaks HTTPS to the browser regardless. But the leg from edge to origin can be HTTP or HTTPS, and Arvan stores that choice per DNS record, in a field called upstream_https. Most of this story is the consequence of not knowing that.

Act I — The 502 that left no trace

It started with Next.js JavaScript chunks failing:

GET https://jainsta.ir/_next/static/chunks/0yb2wil16uz6h.js
502 Bad Gateway
Remote Address: 185.143.234.238:443

The nginx config looked reasonable. So we checked the error log.

It was empty.

🔑 Concept: whoever writes the error owns the failure

nginx physically cannot return a 502 silently. Any upstream failure — connect() failed, upstream prematurely closed connection, no live upstreams — gets written to the error log at error level. Always.

An empty error log next to a 502 is not "no information." It is a very strong statement: nginx did not generate this response.

That single fact reframed everything. If nginx didn't produce the 502, something in front of it did. And 185.143.234.238 — the address the browser reported talking to — wasn't the server at all. It was an ArvanCloud edge.

Lesson: before theorising, ask which machine actually produced the bytes you're looking at. Remote Address in the browser's network tab tells you, and it is frequently not your server.

Act II — A private IP appears

Then a stranger showed up in the network tab:

http://10.10.34.35/

10.x.x.x is RFC1918 — private address space. It cannot be routed on the public internet. If a browser is talking to one, the response didn't come from the internet; something on the local path answered on the site's behalf.

10.10.34.34, .35, and .36 are the addresses Iran's national filtering system returns for blocked domains. The browser was being pointed at the block-notice page.

The full mechanism, visible in the network tab once we looked closely:

GET /_next/static/chunks/06_19_ao4yfnx.js
   → 302 Found,  Location: http://10.10.34.35/
      → browser refuses to follow
         → 0.0 kB "(failed) net::ERR_..."

🔑 Concept: mixed content blocking

A page loaded over https:// may not load subresources over http://. Browsers block it outright — otherwise an attacker could downgrade individual scripts on an otherwise secure page.

So the redirect to http://10.10.34.35/ didn't even reach the block page. It died at the browser, as a zero-byte failure. The app broke not because the block page loaded, but because the scripts silently didn't.

There was also a second tell we should have caught faster: a 302 on a _next/static chunk is impossible. Next.js serves those files or 404s them. It does not redirect them. Any redirect on a static asset URL means something between you and the server rewrote the response.

Act III — curl says 200, Chrome says 302

This was the most confusing stretch, and the most instructive.

From the server, and later from the same Mac running the same browser, on the same network, at the same moment:

curl -sI --resolve jainsta.ir:443:185.143.234.238 \
  https://jainsta.ir/_next/static/chunks/0yb2wil16uz6h.js
# HTTP/2 200 · content-length: 5587 · x-cache: MISS

1. Both edges. x-cache: MISS meaning the edge fetched it live from the origin and the origin answered correctly. The file existed, the app was healthy, the CDN was healthy.

Meanwhile, Chrome got a 302 to the block page for the same URL.

🔑 Concept: filtering can key on how you connect, not what you ask for

If the trigger were the URL, the domain, or the IP, curl and Chrome would fail identically. They didn't.

The most likely explanation is TLS fingerprinting. The ClientHello — the first message of a TLS handshake — carries cipher suites, extensions, and their ordering in a pattern that is highly distinctive per client. Chrome's fingerprint looks nothing like curl's. Deep packet inspection can classify on this without decrypting anything.

Practical consequence: curl succeeding from your laptop is not evidence that a site works. It only proves the path works for curl.

A related trap cost us time here too:

ubuntu@2-vcpu---4-gb-ram:~$ dig +short jainsta.ir

We ran the "is DNS being intercepted?" test three times — from the server. But the interception was happening on the client's network. The server's DNS was never in question.

Lesson: when a problem is client-side, every diagnostic must run on the client. The prompt in your terminal is worth reading before you trust the output.

Act IV — The bug that was ours all along

Buried in the original config was this:

proxy_set_header Connection "upgrade";

alongside:

upstream jainsta_app {
    server 127.0.0.1:3001;
    keepalive 32;
}

These two lines are in direct conflict.

🔑 Concept: hop-by-hop headers and keepalive

Connection is a hop-by-hop header. It describes the single TCP connection it travels on, not the end-to-end request. Two things follow:

1. It's sent on every request, not just websockets. Hardcoding "upgrade" tells your Node process "we're switching protocols" on a plain GET for a JS file. Node sees an upgrade request with nothing to upgrade to and typically closes the connection. nginx sees the close and returns 502.

2. It defeats keepalive. Connection reuse requires sending Connection: "" toward the upstream. Connection: upgrade tears the socket down after each request, so the pool of 32 never pools anything — and a reused-but-poisoned connection fails intermittently, which is exactly the "some chunks fail, most don't" pattern.

The correct form, if you need websockets, uses a map in the http {} block:

map $http_upgrade $connection_upgrade {
    default upgrade;   # client sent Upgrade: websocket → pass it through
    ''      '';        # client sent nothing → empty, so keepalive works
}

proxy_set_header Upgrade    $http_upgrade;
proxy_set_header Connection $connection_upgrade;

If you don't use websockets — which turned out to be the case here — it's simpler:

proxy_set_header Connection "";

This wasn't the cause of the block-page problem. It was a genuine, independent bug that would have produced sporadic 502s forever.

Act V — Going direct, and the three bugs that followed

The decisive experiment was to bypass the CDN entirely: get a real Let's Encrypt certificate on the origin, point DNS straight at it, and see what the browser does.

10.10.34.35 vanished.

That was the answer to the original question. But going direct broke three other things in quick succession, and each one teaches something.

Bug 1 — ERR_HTTP2_PROTOCOL_ERROR

Turning off the CDN toggle for techstory.dev produced an immediate, cryptic browser error. The cause was in the config all along:

# 000-default-catchall
server {
    listen 443 ssl http2 default_server;
    server_name _;
    return 444;    # drop the connection, send nothing
}

Every site's config was listen 80 only, because Arvan had always terminated TLS. So when the browser connected directly to port 443, no server_name matched, nginx fell through to the default server, and 444 killed the connection mid-handshake.

🔑 Concept: SNI matching and default_server

On a shared IP, nginx picks a server block by comparing the TLS SNI (and later the Host header) against server_name. If nothing matches, it uses the block marked default_server — or, failing that, the first-defined block on that socket.

return 444 is an nginx-specific code meaning "close without responding." It's a good catchall (it stops unmatched traffic from leaking whichever backend happens to be defined first) but it produces baffling errors when you accidentally route real traffic into it.

This exact mechanism came back later for graphql.techstory.dev — record flipped to HTTPS origin, but no 443 block in its config, so the edge hit the catchall, got nothing, and reported 502.

Bug 2 — ERR_TOO_MANY_REDIRECTS

Certbot's --nginx plugin helpfully adds an HTTP→HTTPS redirect:

server {
    listen 80;
    return 301 https://$host$request_uri;
}

Sensible in isolation. Catastrophic when the CDN's origin protocol is HTTP:

Browser → edge (HTTPS)
        → origin :80
        → 301 to https://jainsta.ir
        → edge follows... to itself
        → origin :80
        → 301 ...            ∞

🔑 Concept: never redirect to HTTPS on a plaintext origin

When something in front of you terminates TLS and re-requests over HTTP, that thing has already served HTTPS to the user. If you then insist on redirecting to HTTPS, you're telling it to redo work it already did — and it will, forever.

Either the front end must speak HTTPS to you, or you must not redirect. There's no third option.

This is also why X-Forwarded-Proto gets hardcoded to https in a CDN setup: $scheme would say http, and the app would generate http:// links and its own redirects, building the same loop one layer up.

Bug 3 — 404 Not Found on every tenant

https://roozaashop.jainsta.ir/  →  404 Not Found · NGINX

Certbot had rewritten the port-80 block like this:

server {
    if ($host = www.jainsta.ir) { return 301 ...; }
    if ($host = jainsta.ir)     { return 301 ...; }
    listen 80;
    server_name jainsta.ir *.jainsta.ir;
    return 404;                 # ← everything else falls here
}

roozaashop.jainsta.ir matched the wildcard, matched neither if, and hit the 404. Certbot had no idea this was a multi-tenant app — it only knew about the two names on the certificate.

But why did the apex work while subdomains didn't? The Arvan API gave the answer instantly:

@         a   https      ← works
*         a   default    ← 404s  (default = HTTP)
meeseeks  a   default    ← 404s
www       a   https      ← works

Records set to https reached the working 443 block. Records set to default reached port 80 and hit certbot's return 404.

🔑 Concept: read the API, not the panel

A per-record setting buried in a web UI is invisible during debugging. One curl against Arvan's API produced a four-line table that explained a problem we'd been circling for an hour.

When a CDN or DNS provider has an API, learn the one endpoint that dumps your records. It converts "why does this hostname behave differently" from a mystery into a lookup.

Act VI — Certificates for a multi-tenant app

The real fix required a certificate that covers arbitrary subdomains.

🔑 Concept: HTTP-01 vs DNS-01

HTTP-01 — Let's Encrypt fetches http://yourdomain/.well-known/acme-challenge/<token>. Simple, automatic, and it cannot issue wildcards, because proving control of one hostname says nothing about all possible subdomains.

DNS-01 — you publish a TXT record at _acme-challenge.yourdomain. Proving control of the zone proves control of every name in it, so wildcards are possible. It needs API access to your DNS provider to be automatic.

There was a second, sneakier reason DNS-01 became mandatory here: once the origin spoke HTTPS only, port 80 no longer received edge traffic, so HTTP-01 validation could no longer reach the server at all. Switching the origin to HTTPS quietly breaks every certbot renewal that relies on HTTP-01.

Arvan runs the DNS (they require NS delegation), and acme.sh ships a plugin for their API:

export Arvan_Token="apiKey <token>"
~/.acme.sh/acme.sh --issue --dns dns_arvan \
  -d jainsta.ir -d '*.jainsta.ir' \
  --server letsencrypt

The first attempt failed with invalid domain — an unhelpful message that turned out to mean the token was wrong, not the domain. Testing the API directly with curl is what isolated it. When a wrapper gives you a vague error, call the underlying API by hand.

Installing the cert used --install-cert rather than pointing nginx at ~/.acme.sh/:

~/.acme.sh/acme.sh --install-cert -d jainsta.ir --ecc \
  --key-file       /etc/nginx/ssl/jainsta.ir/privkey.pem \
  --fullchain-file /etc/nginx/ssl/jainsta.ir/fullchain.pem \
  --reloadcmd      "sudo systemctl reload nginx"

🔑 Concept: stable paths and reload hooks

acme.sh regenerates files in its internal directory on renewal and the paths aren't guaranteed. --install-cert copies them to a location you choose and — crucially — records a reloadcmd that runs automatically on every future renewal.

Without the reload hook, your certificate renews perfectly and nginx keeps serving the expired one until someone notices.

Act VII — Multi-tenant routing rules

The final configuration hinges on two nginx behaviours worth knowing precisely.

Host $host is what makes multi-tenancy work.

proxy_set_header Host $host;

$host is the name the client requested. The app receives roozaashop.jainsta.ir and resolves the tenant from it. The alternative, $proxy_host, would send 127.0.0.1 and collapse every tenant into one — a silent, spectacular bug.

Exact server_name matches beat wildcards.

server_name jainsta.ir *.jainsta.ir;   # block A — all tenants
server_name meeseeks.jainsta.ir;       # block B — the API

nginx checks exact names first, then leading wildcards. meeseeks.jainsta.ir reliably lands in block B even though block A would also match. That's how you carve one subdomain out of a wildcard, and it's why meeseeks kept working while roozaashop broke.

And a warning that isn't a warning:

[warn] protocol options redefined for 0.0.0.0:443

In nginx 1.24, http2 and ipv6only are properties of the listening socket, not the server block. The first block to bind 0.0.0.0:443 decides for all of them; the rest get this warning. HTTP/2 was already on for every site via the catchall.

nginx 1.25.1+ replaced this with a per-server http2 on; directive — which is why that syntax errored out on 1.24 with unknown directive "http2".

The thing that mattered most, and had nothing to do with any of it

Midway through, docker ps showed:

0.0.0.0:5432->5432/tcp   postgres          up 6 days
0.0.0.0:4000->4000/tcp   fleet-street-app
0.0.0.0:3000->3000/tcp   the-engineering-mind-web

0.0.0.0 means every interface, including the public one. A live Postgres, and a GraphQL API bypassing nginx, TLS, and the CORS rules configured for it.

The access log had already noticed:

31.56.58.124  "GET /.git/config"      404   Mozilla/5.0 (Debian...)
139.59.42.255 "GET /"                 200   Mozilla/5.0 zgrab/0.x

zgrab is an internet-wide scanner. /.git/config is a standard probe for leaked repositories. These aren't targeted attacks; they're the background radiation of the public internet, and they mean your IP is on lists.

🔑 Concept: Docker publishes past your firewall

docker run -p 5432:5432 binds 0.0.0.0 by default and writes its own iptables rules — which are frequently evaluated before ufw's. So ufw enable may report success while the port stays wide open.

Bind explicitly instead:

ports:
  - "127.0.0.1:5432:5432"

And always verify from outside the machine:

nc -zv <public-ip> 5432    # must refuse or time out

Better still: if only other containers need the database, put them on a shared Docker network and remove the ports: block entirely. A port that isn't published can't be scanned.

What I'd take away

Read the absence of evidence. The empty error log was the single most informative artifact of the night. Silence from a component that is required to speak means it wasn't involved.

Check timestamps. A curl at 22:48 against a last log entry at 22:42 proved the request never arrived — no theory needed.

Verify from the affected machine. Three DNS tests ran on the server for a problem that lived on the client. The shell prompt was telling us the whole time.

Impossible responses are the loudest clue. A 302 on a static asset. A private IP in a public network tab. When something can't happen, stop debugging the thing you assumed and start asking who could have produced it.

Layers disagree silently. Nearly every bug here was two layers holding different beliefs: the CDN thought HTTP, the origin thought HTTPS; certbot thought two hostnames, the app thought unlimited; nginx thought per-server, the socket thought per-socket. None of these throw errors. They just behave strangely.

Fix the boring thing. The filtering was interesting and largely outside our control. The open Postgres was dull and entirely within it. Interesting problems are seductive; that's exactly why they crowd out the important ones.

Final state