The broken link that fell between two conditions

We shipped 0.9.0 yesterday, and I wrote up the scheduler bug that turned up while writing it. One of the things it brought was that the destination of an external redirect finally gets a row and a status check. 0.9.1 is out today because that work was not visible where it mattered.

Where people look for broken links

In a view called v_broken_links, and in the spreadsheet sheet built from it. Its original definition was short and looked complete:

SELECT ... FROM links l
JOIN urls uf ON uf.id = l.from_url_id
JOIN urls ut ON ut.id = l.to_url_id
WHERE ut.status_code >= 400;

Links whose destination answers 400 or above. Now take yesterday’s case: a page of yours links /go/product, that URL answers 301 towards a shop that is not yours, and the shop returns 404.

The link points at /go/product, whose status is 301. It does not pass the filter. The 404 has been in the database since 0.9.0, with its row and its checked status, but no link points at it, because nobody links the final destination: you get there by redirect. The row fell between the two conditions and showed up nowhere.

It is awkward to admit because the data was right there. We had gone looking for it on purpose, with its HEAD request and its courtesy of one at a time per foreign host, and then the most obvious question anyone asks the tool could not find it.

Following the chain, and how far

The new view follows redirects to where they end up. If the end is broken, the link shows up, with two extra columns so nothing is lost along the way: via is the URL that was actually linked, the one you have to rewrite in your HTML, and hops how many jumps it took. On a direct row both are empty, and that difference matters: “this is broken” and “this leads to something broken” are not the same finding.

The chain is walked in SQL, with a recursive query inside the view itself. We could have done it in Rust, which is where the chain and loop rules already do it, but then the answer would only exist inside the tool. A crawl file is an ordinary SQLite database, and part of the promise is that you can open it with sqlite3, type SELECT * FROM v_broken_links and get the answer. Both forms coexist because they solve different questions.

There is a ten-hop cap, and it is not decoration. A redirect can come back on itself, A → B → A, and without a cap the recursion produces rows until somebody kills the process. With it, the query answers. There is a test that builds a real loop out of two URLs pointing at each other and checks two things: that the view replies instead of hanging, and that the loop is still reported by HTTP-REDIRECT-LOOP, which is the rule whose job it is.

Why this is 0.9.1 and not 0.10.0

This project’s versioning rule is written in the changelog, and it says a new rule, or a fix that changes what a rule reports, is a minor bump, because someone may have a CI gate that depends on that rule firing or not firing.

Neither happens here. No rule reads v_broken_links: only the export does. No rule ID changes meaning. What changes is that links which were broken all along are now listed, so if you count rows in broken_links.csv that number can go up. It is spelled out in the changelog, because a number that rises without warning is the kind of thing that makes people distrust a tool.

The slip, which also counts

I added the migration, wrote the view, ran the tests and every one of them fell over with this:

rastrear: WriterGone

The writer dies and does not say why. The cause was dull: the engine keeps a constant with the schema version it knows how to write, and I left it at 8 with migration 9 in place. The file declared a newer version than the code, and the code refused it, rightly.

The interesting part is how many things caught it. First the failure itself. Then, once the constant was bumped, two guards that read the migrations directory at run time and compare it with the list the test helpers apply, one in each crate, failing with exactly what is missing. And finally the compatibility test, which opens a file from every earlier version and checks it migrates without losing anything. Four places that have to move together, and all four say so.

What it cost, measured

On a real crawl of 5,865 URLs, 145,191 links and 486 redirects, counting the view goes from 8 to 21 milliseconds. The recursion only walks URLs that are redirects, and on a normal site those are a handful against the total.

And one number that does not flatter the headline but is the one we have: on that crawl the new view finds not a single extra row, because none of its 486 redirects ends in a 4xx. The case it fixes is real and happens to anyone using affiliate links or their own short URLs; on that particular site, that day, there was none. We publish the number anyway.

Everything else stands where it was: 1,033 tests green, the linter quiet, and the performance regression compiled with optimisations at 106,992 items per second and 30.4 MB peak memory.