The security fix that did not hold

On 4 August the crawler started checking the status of outbound links, and it does it by default. The feature is short to describe: for every link leaving your site, one HEAD request to find out whether the target is still alive, never reading the body, never following redirects.

It also changes what the tool is. Until that day it only requested your own pages. From that day it requests URLs somebody else wrote.

What the review found

The only filter before requesting a URL from a third party was the scheme. If it started with http or https, it went out. Nothing else stood between the two.

So a site under audit could put this on any of its pages:

<a href="http://169.254.169.254/latest/meta-data/">.</a>

And the crawler requested it from the machine of the person running the audit. It does not help that the response body is discarded: the crawl file keeps the status, the response time, and an error message precise enough to tell a refused connection from a timeout.

That is a map. It says which addresses have something listening and which do not, and how fast they answer. Spread enough links across a private range and a hostile site draws the office network of a consultancy during a routine audit, with nothing anywhere making a noise.

Where the hole lives is what makes it worse. The .sqlite crawl file is precisely the artefact this product is built to send to a client.

The first screen

The fix shipped in 0.5.0. Before probing a URL, look at its host: reject every address that is not globally routable, and reject the names localhost, *.local and *.internal. Later came .lan, home.arpa, .corp, fritz.box and the short names that resolve inside a cluster or a cloud instance.

The screen is not unconditional, and that was a decision. It switches off when the audited site is itself local, because auditing an astro dev on localhost, or a client’s staging box on the office LAN, means whoever launched the crawl is already inside that network. The exception to the exception is the cloud metadata range, which is screened either way. A crawl of localhost from a CI runner is a real thing people do, and that address answers with the instance’s credentials.

It went out with a comment in the code telling the truth: the screen is lexical, and a name that resolves to a private address goes through. That sentence was written before the release. We released anyway.

The second review

Public wildcard DNS services exist. They were built so a developer can test subdomains without editing /etc/hosts, they have been running for years, and using one requires registering nothing and owning nothing:

• localtest.me and lvh.me resolve to 127.0.0.1.
• nip.io and sslip.io resolve to whatever address you spell inside the name itself. 192.168.1.10.nip.io resolves to 192.168.1.10, and 10.0.0.5.sslip.io to 10.0.0.5.

None of those four names is a private address. None ends in .local or is called localhost. The screen passes all of them, because it reads text and the text is spotless.

This was not reasoned about, it was executed: a service listening on loopback, the screen on, the probe from our own binary. HTTP 200.

Then came 169.254.169.254.nip.io. That name also walks past the cloud metadata exception, the one screen the code called non-negotiable, added that same morning.

Why the second attempt was worse than the first

The design error was not shipping a partial screen while knowing it was partial. It was what happened once the first gaps showed up: we hardened it. .lan, home.arpa, .corp, a handful more short names. Every one of those additions is correct on its own and not one of them moves toward a solution, because they all answer the same question — what is this host called — and the answer does not live inside that question.

Hardening something that half works looks an awful lot like fixing it. It produces work, it produces a diff, it leaves the tests green, and the gap sits exactly where it was. Four days earlier the same thing had happened with the orphan-page rule, where the first correction touched the place the damage showed rather than the place that caused it. It repeats because the fix for the symptom is always the one closer to hand.

The fix that worked

A name does not say where it points. The resolver says that, and only after it has resolved. So the decision moved there: resolution now goes through our own resolver, which filters the addresses before any of them reaches a socket. The text of the host no longer decides anything.

With one rule that looks excessive and is not. If a single address behind a name is out of bounds, the whole name is rejected. Keeping the public ones and dropping the private ones sounds tidier and in practice hands the choice to DNS, which is half of a rebinding attack: the name returns two addresses, you keep the good one, and the next lookup returns only the other.

The lexical screen stays as the first line, and not out of caution. A literal IP typed into an href never reaches a resolver at all, so there is a whole class of case only the screen sees.

Five more findings came through the same door in that review, and one is worth telling because it explains why a perimeter has to live in exactly one place. The follow_external key in the config file was consulted only on the probe path, so a crawl with that scope reached the addresses the probe was forbidden to touch, and did it with a full GET whose body was parsed and stored. Executing the case wrote the title and the <h1> of an internal Kibana dashboard into the crawl file.

How to check yours

If you maintain anything that fetches URLs on someone else’s behalf — a link checker, a social card preview, a webhook with a configurable URL, a feed importer — this takes a minute:

curl -s -o /dev/null -w '%{http_code}\n' \
  'https://your-service.example/check?url=http://127.0.0.1.nip.io:8080/'

And the one that actually hurts, if your service runs on a cloud instance:

http://169.254.169.254.nip.io/latest/meta-data/

If your filter looks at the hostname, both go through. Anything other than a rejection coming back means you have the design we had a week ago.

What we took from it fits in one line, and it is written into the code so nobody undoes it in a year: a lexical screen cannot defend against a name.