The invariant was written three lines above

Version 0.9.0 went out to close two gaps noted down on 4 August, neither of them mysterious. In both cases the crawler already knew something and there was no way to see it from outside. Closing the second one turned up a scheduler bug that was on no list at all.

A table that had been filling up for nobody

The resources table has been in the schema since the first migration. The crawler started writing to it in 0.8.0, with a migration that gave it a unique index per URL so that resuming a crawl updates rows instead of duplicating them. All correct, all tested.

And nothing read it. No sheet in the spreadsheet, no CSV, nothing in the report. The case that justified having the table was specific: a 900 KB bundle.js served on every page of a site’s template. That number was sitting inside the crawl file and no command could get it out.

The workbook now has fourteen sheets instead of thirteen, and one of them is Resources. The export also writes a resources.csv. It is sorted by size, largest first, with one detail worth mentioning because it is easy to get wrong without noticing: in SQLite, null values sort last in a descending order, so a resource whose server sends no Content-Length falls to the bottom rather than to the top, which is where it would have landed under most other orderings and where it says nothing useful.

What the sheet does not tell you, and it should be said plainly: it does not know on how many pages each resource appears. It is one row per resource URL, not per page-and-resource pair. That relationship is only stored for images, and it was a data model decision that still stands. The sheet answers “what is the heaviest thing I am serving”, not “where am I serving it”.

The affiliate link that rots with nobody watching

The second gap costs more. Picture a /go/product on your site redirecting to a shop that is not yours. When that shop pulls the product, your page pays for the 404, and until 0.9.0 the crawl showed you the 301 on /go/product and stopped there. The destination existed as a row in no mode, so the redirect_to column was left unresolved and no rule could reach the far end to say it was dead.

The striking part is that the machinery was already built. Since 0.8.0 external links get recorded and status-checked with a HEAD request, which is what makes it possible to report an outbound broken link at all. A redirect destination never went down that path, because the block that handles it is a different one, a few lines further down the same loop.

It now gets the same treatment: it counts against the same cap on registered externals, joins the same probe queue, goes through the same network perimeter, and keeps the same courtesy of one request in flight per foreign host. It is still status only, with nothing parsed from someone else’s site. And the Redirects sheet carries a to_status column with the destination’s code, so you can see it without writing SQL.

The test failed for a different reason

I wrote the test before calling anything done: a site with two redirects to the same foreign shop, one to a product still on sale and one to a product that had been pulled. It failed, but not the way I expected.

assertion failed: externals.pending() == 0

That is a debug_assert in the scheduler, and it says something reasonable: with nothing in flight, nothing can be left to dispatch, because with a minimum limit of one request per host anything still queued had its chance in the last pool refill. Four lines of instrumentation showed the state at that moment:

DIAG pendientes=1 in_flight_by_host={}

Nobody in flight, and one probe waiting its turn. The queue had work and the scheduler could not see it.

Three lines above

The probe queue keeps two things: a map of hosts with what it knows about each, and a round of hosts with pending work that rotates so a slow server cannot hog the turn. The comment above that round states, in as many words, that a host is in it if and only if its queue exists and is not empty.

The code putting hosts into the round asked something else:

let nuevo = !self.by_host.contains_key(&host);

That is, “is this the first time I see this host?”. And a host’s entry in that map outlives the emptying of its queue, because that is where its probe budget and its failure streak live. The second time a host showed up, its URL went into the queue and the host did not return to the round, so nobody dispatched it.

The fix is to look at the queue instead of the map:

let fuera_de_la_ronda = state.queue.is_empty();

Why it had not shown up before comes down to how things get discovered. With links, all the external URLs on a page are queued at once, while the same document is being processed, and that host’s queue rarely empties in between. With redirects each destination arrives in a separate result, at separate moments, and two redirects to the same shop are exactly the case that breaks.

The panic is not the bad part. In a release build debug_assert does not exist, so the loop simply ended and that probe was counted at the end as an “external not checked”, with no reason attached. A bug that shows up as a slightly different number in a summary is far harder to find than one that blows up.

What ran before publishing

All three changes were verified by reverting them one at a time and watching their tests fail, which is the only way to know whether a test protects anything. The host round one breaks the scheduler’s invariant again. The redirect destination one returns None where a 200 belongs. The resources one cannot find the sheet.

With that done: 1,031 tests green, clippy quiet, and the performance regression compiled with optimisations, giving 111,809 items per second, 2,727 pages per second and 30.1 MB peak memory on the usual fixture. The change touches the main crawl loop and does not show up in the numbers, which was the fair worry.

One thing is left half done, and I would rather say it here than let you find out. The resources sheet shows the number and no rule judges it yet: nothing warns you that the bundle.js is too heavy. Weight rules touch the catalogue, the catalogue is what this site consumes, and that asks for a version of its own.