The devlog
Measurements, defects and worked examples, every two weeks. Numbers come with their method attached and defects are told in full — it is what we ask of the tool, so it is the least we can do.
7 min read
Installing CrawlForge without Rust
0.22.0 installs with Homebrew, a one-line installer or cargo, from binaries GitHub builds for free. Five things broke on the way there.
8 min read
A skip link to nowhere
0.21.0 checks that every #fragment in an internal link exists, backs off when a server pushes back, and passes a dependency audit.
8 min read
Each site with its own rules
0.20.0 lets each site turn rules off or change their severity without a new crawl, adds five free rules and prints its output in ten languages.
6 min read
Where Google stops reading
0.19.0 flags a canonical, robots or hreflang that ends up outside the head. To know whether it was right, I checked it against a full standards parser.
7 min read
The same stadium, twice
0.18.0 adds DUP-CONTENT-NEAR, which flags pages whose text is nearly identical even when their HTML isn't. The planned algorithm didn't hold up, and neither did the usual threshold.
7 min read
88 pages that weren't broken
0.16.0 adds a rule for external links behind a wall and fixes how dashes are measured. 0.17.0 stops reading a rate limit as a broken page.
5 min read
Fifty domains that counted as one site
0.15.0 makes the portfolio panel count sites, not files. A URL list with fifty domains used to pass as a single site, cap included.
5 min read
Two sites, one name in the title
0.14.0 names every site a URL list audited. The report of a two-site list used to carry only the first line's site, as if the other weren't there.
6 min read
314 links and the wrong advice
0.13.0 records why an external URL went unchecked. Fixing that showed our own site's report blaming a cap and suggesting a flag that doesn't exist.
7 min read
The diff file nobody could open
0.12.0 reopens the file that diff --out saves, lists every row without the cut-off, and fixes a summary that hid 1,171 sitemap changes.
7 min read
A list with two sites and 121 stray requests
0.11.0 audits every site in a URL list. Before, the first line decided, and a twelve-URL list sent 121 probes to our own pages.
9 min read
Four rules, a port, and a 24% that nearly slipped through
0.10.0 brings the rules that were missing to judge what the crawler already knew, and changes what counts as your own site. The interesting part is that performance dropped 24%, the regression test let it pass, and the first explanation I came up with was wrong.
8 min read
The test passed with the bug in place
0.9.2 fixes something small: a crawl waiting on other people's servers no longer looks dead. What is worth telling is that I wrote the test, watched it go green, reverted the fix to make sure it would fail, and it stayed green. The test was covering the part that was not broken.
8 min read
The broken link that fell between two conditions
0.9.0 started checking where external redirects end up. The view that answers "which of my links are broken" still could not find them, because a 301 is not broken and nothing links the 404 at the end. Recording the data and showing it are two different jobs, and this is the second one.
9 min read
The invariant was written three lines above
0.9.0 was meant to be two surface fixes: show a table nothing could read, and record the destination of external redirects. The second one uncovered a scheduler bug that had been there since probes existed, and that made no noise in release builds. It just dropped checks.
10 min read
A link to someone else's site wasn't a link to someone else's site
We audited our own website and got twelve warnings about internal links marked nofollow. None of those links were internal. Underneath was a fault that turned any link to another domain's home page into a link to ours.
10 min read
The button you couldn't read and the focus that hid
We ran our own website through a design audit and an accessibility one. Two measurable failures came out: the primary button sat at 3.09:1 contrast in dark theme, and eight of the twenty-four focusable elements disappeared under the header when moving through the page by keyboard.
9 min read
A rule that does not appear is not a rule that passes
The portfolio panel says on how many sites each rule fires. For that number to stay honest it has to track three states rather than two: fires, does not fire, and could not be evaluated.
9 min read
Your audit tool is not seeing your images
A news site with 18,000 pages of photos reported zero heavy images. The table was empty. The cause is a lazy-loading plugin that half of WordPress runs.
10 min read
The security fix that did not hold
We closed the hole that let an audited site aim the crawler at its operator's internal network. A second review showed the fix was worthless: four public DNS services walk straight through it, and the attacker has to own nothing.
11 min read
CrawlForge against Screaming Frog: a comparison that will not convince you
We gave both tools the same 300 URLs and compared field by field. They extract exactly the same thing. From there the comparison gets interesting, and not always in our favour.
8 min read
Why we wrote another SEO crawler
We look after more than a hundred sites. The tool we were using answers «how is this site doing» very well, and does not answer «what changed since last week», which is the question we have every Monday.
9 min read
How to check that your crawler is not lying to you
We gave two tools the same 300 URLs and compared field by field. Out of 1,800 comparisons one difference came back, and it was ours: months old, invisible to 810 tests.