At Color Vivo we look after more than a hundred sites between client work and our own projects. Mostly WordPress, some Astro, the odd PrestaShop. Every month a batch of them needs reviewing, and until this summer we did that with Screaming Frog, which is a fine tool that we have been paying for without complaint for years.
The tool was not the problem. The question was.
A crawl answers the wrong question
The first time you audit a site, a crawl tells you everything: how many URLs there are, which ones 404, which titles are duplicated. You hand over the report, the client fixes some of it, and two weeks later you crawl again.
And there are a thousand rows in front of you again. Roughly the same thousand rows, with some differences you would have to find by hand. Are last month’s 404s fixed? Did anything new appear after the deploy? Did the template change break something?
You can export both crawls to a spreadsheet and compare them with formulas. We have done it. It takes half a morning per site and you get it wrong.
With a hundred sites, half a morning per site does not exist.
What we ended up doing
For a couple of years our routine was this: crawl only when someone complained. A client would call because traffic had dropped, we would look, and we would find something that had been broken for three weeks. We were always late, and not out of laziness — reviewing a hundred sites properly every month does not fit in anyone’s calendar.
What was missing was something else. Not a more thorough report, but one that starts with what changed and leaves the rest in the background.
The diff
The first version of CrawlForge was that and nothing else. Crawl the same site twice and say what moved:
$ crawlforge diff before.sqlite after.sqlite
── Got better ───────────────────────────────
+ 11 HTTP-404-INTERNAL resolved
404 → 301 /googlechrome
404 → 301 /internetexplorer
── Got worse ────────────────────────────────
− 1 HTTP-REDIRECT-CHAIN new
/googlechrome → article → mobile versionThat output is real, from an audit of colorvivo.com on 3 August. We had fixed eleven broken links with redirects, and one of those redirects pointed at an article that was itself already redirected. Two hops instead of one.
Without the diff we would never have seen it. The total finding count went from 2,834 to 2,856, so reading the number alone the conclusion would have been that the site got worse.
Why not a browser extension or a service
We considered it. A cloud service that crawls every week and emails you is what the market asks for, and we would probably sell more of it.
There is a problem: crawls are large. One of the sites we look after, a news outlet with fifteen years of archive, has 487,621 URLs. The full crawl file is 5.3 GB. Multiply that by a hundred sites and four weeks of history and you get infrastructure numbers that do not fit a thirty-euro-a-month tool.
On your own machine, 5.3 GB is just a file. You open it, query it, delete it when you are done.
Rust and SQLite
The two technical decisions that shape everything else were made on day one.
The engine is Rust because the work is parsing HTML fast and writing millions of rows without eating memory. On that 487,621-URL crawl memory stayed flat at 202 MB, and not because of any cleverness: memory follows the pending queue, not the total crawled. When the queue drains, it drops.
Every crawl is a SQLite file, and that turned out to matter more than we expected. It means you can ask it things we never thought of:
SELECT u.url, u.response_time_ms
FROM urls u
WHERE u.content_type LIKE 'text/html%'
ORDER BY u.response_time_ms DESC
LIMIT 20;No interface covers every case. An open file does.
What Screaming Frog does better
Worth saying, because we have used it for years and it is still on the laptop.
Its interface is better than ours, partly because ours does not exist yet: today CrawlForge is a command line. Its catalogue of checks is wider, built over fifteen years. It renders JavaScript, which we do not. It integrates with Search Console and Analytics. And it has a large community, with a tutorial for any question you might have.
If you audit a site now and then and want a mature tool with a graphical interface, buy it without hesitating. We are not trying to replace it in that case, because in that case we have nothing better to offer.
Where we think we win is in ours: many sites, repeated reviews, and needing to know what moved.
What exists today
The command line crawls over HTTP, audits a built folder before you publish, and compares two crawls. It has 59 audit rules, each with its own test case, because a rule that gets it wrong makes you stop trusting the whole report.
The code is published under Apache 2.0: the crawler, the rules, the command line and the WordPress and Astro adapters. The desktop applications, if they ever exist, will be paid. A native interface is a lot of work and we are not promising dates.
Meanwhile we write here about what we measure and what breaks. This week, for instance, we found
out we had been returning <h1> elements wrong for months whenever they contained a <br>. We
found it by comparing our extraction against another crawler on the same 300 URLs, and it deserves
its own article.