The field log from the first week of August has a line that had been waiting two months. Crawling
diariodelamancha.com, 9,446 rows of external URLs had an empty exclusion_reason: nothing said why
they had been left unchecked. URLs skipped by a pattern or by robots.txt did say so. It looked like
the smallest item on the list, a column left blank, and it went into 0.13.0 the same day as the
saved diff reader.
It turned out to be hiding something bigger.
What an empty column means
Every URL the crawler finds gets a row in the crawl file, including the ones it won’t download.
An external link is recorded so the report knows where the site points, and by default it gets one
HEAD request to check its status, which is what makes broken outbound links reportable. When that
check doesn’t happen, the row says why: nofollow for a link marked that way, pattern for one your
--exclude caught, local_network for an address inside your own network, which the crawler
refuses to touch.
Two cases were left without a reason. One is --no-external-check, when you turn the checks off
yourself. The other is crawlforge audit ./dist, the audit of a built folder, which has no HTTP
client at all and can’t check anything outside the folder.
An empty reason isn’t neutral. The engine already uses that exact shape, external, skipped, no status, no error and no reason, to mean “a probe that was cut short”, and that’s what a resumed crawl looks for to retry. The report uses it too. That was where the bug was.
Our own site’s report
To see what the change would affect, I looked at every place that reads that column. One is the note the report adds about external links, and I ran it on the audit of this website’s build, the same one we run before every deploy:
314 external links were not checked: the max_external cap was reached.
Raise it with --max-external.Each part of that is false. No cap was reached, since a folder audit doesn’t make requests, so
there’s nothing to cap. And --max-external is a crawl and list flag: audit doesn’t accept
it. Anyone who followed the advice would get an error from the same tool that gave it.
The note had been written for a real case, a crawl that finishes with links still unchecked because the default limit of 10,000 checks ran out. It recognised that case by the empty reason. The folder audit produced the same empty reason for a completely different cause, and the note couldn’t tell them apart. Neither could anyone querying the file.
We had audited this site with the tool several times. The line was always there, and in a report with a few dozen findings it read like routine.
Two reasons, and a way to read the old files
Each case now has its own value. external_check_off is written when you turn the checks off, and
no_network when the audit has no network to use. The two branches that record external URLs, an
ordinary link and the external destination of a redirect, both write it. Each one has its own test,
because a fix that only covered links would have left redirect destinations as they were, and the
two branches are separate pieces of code.
The report now says this for the folder audit:
314 external links were not checked: auditing a folder makes no network requests.
To check them, crawl the published site with `crawlforge crawl <URL>`.That still left the files that already exist. Every folder audit written before 0.13.0 has those
rows with the column empty, and a CI job can keep an old file around. The crawl file stores which
mode produced it, so the report also recognises filesystem mode. An old audit gets the right
message too, without needing to be run again.
The other side needed a check as well, because the mode mustn’t hide the case that really is the
cap. An HTTP crawl with unchecked external links and no reason still points at max_external, and
there’s a test that keeps it that way.
inspect, the card for a single URL, used to show these rows as “excluded from the crawl
(no_network)”. Nothing had been excluded: the URL was recorded and only its status was missing. It
now says “external URL — status not checked”, with the reason in brackets.
The test that stayed green
As always, each new test had to be seen failing with its fix removed. Six of the seven failed.
The one that didn’t covered a line I’d added to stop old folder audits from being counted twice, once as “no network” and once as “cap”. With the line removed, the test still passed. I looked at why: the note checks the no-network case first, and when it applies, the other count is never read. The line protected against something that couldn’t happen. I deleted it instead of writing a test to justify it.
The number that always comes up
This release does touch the engine, so the pre-publish benchmark mattered more than last time. I measured on the same machine, under the same load, alternating 0.12.0 and the new build for six rounds, three starting with each:
0.12.0 66,201 59,696 66,712 62,936 64,184 54,151
0.13.0 65,179 39,869 63,366 64,315 64,864 61,970The medians are 63,560 and 63,840 elements per second. Each side has one odd round, and the load average was between 13 and 20 the whole time. The change sets one field on rows that were already being written, so it was never likely to cost anything, but we’ve had a three-line change cost 10x before. The suite has 1,075 tests passing.
The field log is one line shorter now. What it really needs is new lines from someone using the tool every day, and there haven’t been any since August.