88 pages that weren't broken

The field log for phase 1 has been empty since August 3. It’s the last thing left before the phase can close, and it can only be filled by using the tool on real sites. So today I gave it a session of its own: recrawl colorvivo.com onto a copy of the file from August 3, which gives a real two-month diff, then do the same with especialmundial.com, and write down whatever got in the way.

It got in the way within the first ten minutes. In the meantime there were two releases. Here’s both of them, in the order they happened.

0.16.0: the links nobody was reporting

HTTP-404-EXTERNAL reports outbound links that are gone, and it only trusts two codes for that: 404 and 410, plus a domain that no longer resolves. The 401s, 403s and 429s stay out on purpose. The check is a HEAD request with a bot user-agent, often from a datacenter IP, and that’s just what Cloudflare and similar services challenge. In August we measured it: medium.com, wsj.com and ft.com answer the check with 401 or 403, and all three open fine in a browser.

Keeping them out of the broken-link rule was right. Letting them disappear completely was not. A link behind a wall doesn’t prove it works either, and until now nothing in the report set it apart from a link that had been checked and was fine.

HTTP-EXTERNAL-UNVERIFIABLE closes that gap. It’s low, it has its own ID, and it says what happened: the other site refused the check, so open the link to confirm. Folding it into the broken-link ID would have changed what that ID means, and a historical diff depends on rule IDs never changing meaning. On today’s colorvivo crawl it catches 8 links answering 403, among them gartner.com and ostraining.com. The 7 that answered 404 still go to their own rule. The catalog now has 64 rules.

There are five new tests. Four run against an in-memory database and one runs end to end against the local test server, where one external route answers 403 and another 404, and each code has to land in its own rule and nowhere else. I broke four things on purpose to watch them fail: a query that never matches, the filter that only counts <a> links, the one that only counts external URLs, and a list of codes widened to include 407 and 451. Each one turned its test red.

The second fix in 0.16.0 came from somewhere else. While an agent was preparing copy changes for this site, it noticed that the width table the title and description rules use to estimate pixels was wrong about dashes. In Arial the em dash — is a full em, and the table gave it 556 thousandths. The en dash – measures 556, and the table gave it 333, the same as a hyphen. A Title — Site title came out about 9 pixels short. That’s the very separator this site uses for its “ — CrawlForge“ suffix, so the titles of all our own pages were measured too generously. A comment in the same file claimed the core kept a second copy of that table. It doesn’t: the core calls this one. The comment now says so.

0.17.0: a rate limit isn’t a broken page

On to the field session. Ten minutes after I started the colorvivo crawl, the file had 801 pages done, and 92 of them had a 429 status. I asked the same question the report would ask, which internal links point at a 4xx, and got 88 pages. Critical severity, “broken internal link”. A plain curl from the same Mac got a 429 too. None of those pages was broken: in the August crawl, the ones it had answered 200 or redirected, every one.

Cloudflare was limiting the crawler. In August the same site, crawled at the same concurrency, answered 200 on all 2,857 pages and didn’t send a single 429. Something changed on Cloudflare’s side between then and now. That’s fine; it can happen to any site. What isn’t fine is that we took the 429 as the page’s status.

The engine already had two defences. It retries a 429 three times, waiting 1, 2 and 4 seconds, and after three overload signals in a row it halves its concurrency on that host. That isn’t always enough. When a page was still getting a 429 after the third retry, its status was written to the file, and from there every rule that treats an internal 4xx as broken believed it. I counted more than ten queries that do: broken links, broken resources, canonicals and hreflang pointing at errors, the XLSX views. Patching them one by one would have meant leaving one out.

The fix comes earlier, in the engine. A page that still answers 429 after its retries isn’t processed at all. Its row stays pending, the way it was written when it was discovered, so no rule can mistake it for a broken page. The crawl is marked incomplete with a new reason, rate_limited, and the rules that need the whole site stay quiet, since those pages’ links were never read. The summary says how many pages were left and what to run later:

crawlforge resume crawl-colorvivo-com.sqlite

A finished crawl couldn’t be resumed until now, and that still holds except in this one case. resume accepts a finished file if it was rate-limited, fetches just the pending pages, and if none is left, the file stops calling itself incomplete. A 429 that clears up during the retries gets processed as it always did.

To test it, the local server needed something new: a route that fails the first N times and then answers. The test makes that page answer 429 four times (the original request and the three retries), checks that it’s left pending with no broken-link finding, then resumes, and checks the fifth request comes back with a 200 and the file is no longer marked incomplete. With the engine change removed, the test fails on the reason. Without the exception in resume, it fails trying to resume. The summary and report messages have their own tests, and each one fails without its branch.

What I got wrong today

To make the tests go red I break the code and then put it back. On one of those, I put the file back with git checkout, and that also reverted my own change from the same session, the list of codes for the new rule, which hadn’t been committed yet. The next build failed and I noticed straight away. Since then I put things back from a copy of the file, never from git.

The other one went out to the public. The previous entry, the one about the portfolio panel, said no rule hangs a finding from an external URL. HTTP-404-EXTERNAL does. I’d checked against August files, from before external links were probed at all. I corrected the entry and left a note at the end saying what changed. The code was right; the sentence wasn’t.

What’s still open

The engine doesn’t yet follow Retry-After for pages on the site itself, only for external probes. With that, it could wait as long as the server asks for instead of three fixed backoffs, and some of these pages wouldn’t need a resume. It’s the next piece, and it touches the crawl loop, so it needs its own measurement.

The 0.17.0 change does touch engine.rs, so I measured it against 0.16.0, alternating the two for twelve rounds. The Mac was very busy, with the load average reaching 27, and the numbers jump around a lot. The medians are 50,845 and 52,846 elements per second, which is a tie. The change is one integer comparison per page, so it would have been odd to see anything else.

The colorvivo crawl is still running with the old binary. At the time of writing it has 288 pages stuck on 429, 273 of them linked from an <a>, which is every one the old version would have reported as a broken internal link. Of those that were also in the August crawl, 213 answered 200 then and 62 redirected. It’s the best record of the bug I could have asked for, and the field log has entries again.