A hundred sites, not one
One board for the whole portfolio: what got worse, where, and since when. Built for people running dozens of projects, not for auditing one and closing the window.
Command line · Source is open
What changed on your sites since last week
Technical auditing for people who run many sites, not one. Compare crawls, catch what a deploy broke, and work through hundreds of thousands of URLs without bringing the machine to its knees.
Every two weeks: measurements, defects, worked examples. Nothing else.
Install the command line
# macOS and Linux, with Homebrew
brew install dcarrero/tap/crawlforge
# macOS and Linux, without Homebrew
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/dcarrero/crawlforge/releases/latest/download/crawlforge-installer.sh | sh
# Windows, in PowerShell
powershell -ExecutionPolicy Bypass -c "irm https://github.com/dcarrero/crawlforge/releases/latest/download/crawlforge-installer.ps1 | iex"
# With Rust 1.88 or newer
cargo install crawlforgePrebuilt binaries for five systems, a note about macOS, and building from source: every way to install →
$ crawlforge diff example.com27 Jul → 3 Aug · 3,348 URLs
Got better
+11HTTP-404-INTERNAL resolved
404 → 301 /googlechrome
404 → 301 /internetexplorer
Got worse
−1HTTP-REDIRECT-CHAIN new
/googlechrome → article → mobile
A crawl is a photograph. Your job is the film.
You audit a site, hand over the report, changes ship. Two weeks later you crawl again and there are another thousand rows in front of you, with no way to tell which ones are new.
Now multiply that by every site you look after.
Three things you do by hand today
Compare two crawls
What was resolved and what appeared after the last deploy. In your CI, it fails the build when something serious gets in.
No Java, no browser
Written in Rust. A table of two hundred thousand rows scrolls like your system's, because it is your system's.
Open source, and specific about which parts
A tool that asks for your staging credentials should be readable. And a rule that decides whether a client's page is indexable should be arguable. So the engine is open — and the app is what the subscription pays for.
Apache 2.0 · out now
Free and open, today
- The crawler and its 72 audit rules
- The command line: crawl, audit, diff, export
- WordPress and Astro adapters
- Every crawl is a plain SQLite file. Query it yourself.
Being considered
Where this could go
- Native apps for macOS, Windows and Linux
- A portfolio board across all your sites
- Scheduled crawls and change alerts
- Sync between machines and team members
No dates promised. A native interface is a lot of work, and the command line comes first.
Back the project
The open part has no price and no plans to get one. If it saves you an afternoon a month, a recurring contribution keeps the rules catalogue growing.
Become a backer →You choose the amount. Cancel whenever.
Sponsor as a company
Agencies and hosts that run this across a portfolio can sponsor the work at a level that matches what it replaces, with your name on the repository.
Talk to us →Invoiced, with VAT number.
○ macOS · thinking about it○ Windows · thinking about it● Command line · all three, today
The numbers, with their conditions
If you are going to audit with this, check how it was measured first. The method sits next to every figure.
487,621
URLs in a single crawl, with memory flat
A news site with fifteen years of archive. Memory follows the pending queue, not the size of the site.
0 / 1,800
extraction differences against an established crawler
The same 300 URLs to both tools. Status, title, meta description, H1, canonical and indexability.
72
audit rules, each with its own test case
The rules are the product: when one is wrong, you stop trusting the whole report.
Built in the open
Read all posts →3 Nov 2026Product
Installing CrawlForge without Rust
0.22.0 installs with Homebrew, a one-line installer or cargo, from binaries GitHub builds for free. Five things broke on the way there.
3 Nov 2026ProductA skip link to nowhere
0.21.0 checks that every #fragment in an internal link exists, backs off when a server pushes back, and passes a dependency audit.
3 Nov 2026ProductEach site with its own rules
0.20.0 lets each site turn rules off or change their severity without a new crawl, adds five free rules and prints its output in ten languages.
The code is out. Every two weeks for the rest.
The work stays visible: what gets measured, what breaks, and how a real audit is run — with the commands and the numbers, on sites that exist. No download required to read any of it.