The devlog
Measurements, defects and worked examples, every two weeks. Numbers come with their method attached and defects are told in full — it is what we ask of the tool, so it is the least we can do.
9 min read
Four rules, a port, and a 24% that nearly slipped through
0.10.0 brings the rules that were missing to judge what the crawler already knew, and changes what counts as your own site. The interesting part is that performance dropped 24%, the regression test let it pass, and the first explanation I came up with was wrong.
8 min read
The broken link that fell between two conditions
0.9.0 started checking where external redirects end up. The view that answers "which of my links are broken" still could not find them, because a 301 is not broken and nothing links the 404 at the end. Recording the data and showing it are two different jobs, and this is the second one.
8 min read
The test passed with the bug in place
0.9.2 fixes something small: a crawl waiting on other people's servers no longer looks dead. What is worth telling is that I wrote the test, watched it go green, reverted the fix to make sure it would fail, and it stayed green. The test was covering the part that was not broken.
9 min read
The invariant was written three lines above
0.9.0 was meant to be two surface fixes: show a table nothing could read, and record the destination of external redirects. The second one uncovered a scheduler bug that had been there since probes existed, and that made no noise in release builds. It just dropped checks.
10 min read
A link to someone else's site wasn't a link to someone else's site
We audited our own website and got twelve warnings about internal links marked nofollow. None of those links were internal. Underneath was a fault that turned any link to another domain's home page into a link to ours.
10 min read
The button you couldn't read and the focus that hid
We ran our own website through a design audit and an accessibility one. Two measurable failures came out: the primary button sat at 3.09:1 contrast in dark theme, and eight of the twenty-four focusable elements disappeared under the header when moving through the page by keyboard.
9 min read
A rule that does not appear is not a rule that passes
The portfolio panel says on how many sites each rule fires. For that number to stay honest it has to track three states rather than two: fires, does not fire, and could not be evaluated.
9 min read
Your audit tool is not seeing your images
A news site with 18,000 pages of photos reported zero heavy images. The table was empty. The cause is a lazy-loading plugin that half of WordPress runs.
10 min read
The security fix that did not hold
We closed the hole that let an audited site aim the crawler at its operator's internal network. A second review showed the fix was worthless: four public DNS services walk straight through it, and the attacker has to own nothing.
11 min read
CrawlForge against Screaming Frog: a comparison that will not convince you
We gave both tools the same 300 URLs and compared field by field. They extract exactly the same thing. From there the comparison gets interesting, and not always in our favour.
8 min read
Why we wrote another SEO crawler
We look after more than a hundred sites. The tool we were using answers «how is this site doing» very well, and does not answer «what changed since last week», which is the question we have every Monday.
9 min read
How to check that your crawler is not lying to you
We gave two tools the same 300 URLs and compared field by field. Out of 1,800 comparisons one difference came back, and it was ours: months old, invisible to 810 tests.