Command line · Everything on this page exists in the current build
What it does today, and what that saves you
Grouped by what you need done, not by how the code is organised. Where a claim carries a number, the number links to how it was measured.
Know what a deploy broke
Crawling again onto the same file sets the previous crawl aside automatically, and the diff between the two tells you what was resolved and what appeared. A crawl is a snapshot; the diff is what makes it a routine.
In CI it exits non-zero when something serious gets in: --fail-on takes a severity, or the ID of one rule you cannot afford to regress on.
Both crawls are read with the same rule configuration, the one you pass or the later crawl's, so a rule disabled between them never shows up as resolved.
crawlforge crawl https://client.com/
# … the deploy happens …
crawlforge crawl https://client.com/ # the previous one becomes .prev.sqlite
crawlforge diff crawl-client-com.prev.sqlite \
crawl-client-com.sqlite --fail-on highThe whole portfolio in one panel
Point it at a folder of crawl files: what changed since each site's previous crawl, which rules fail across the portfolio, and one line per site, worst first. A rule firing on most sites is rarely content — it is usually a shared template or plugin, one fix that serves them all.
The counting is honest by design. A truncated crawl never evaluated the rules that need the complete link graph, so the panel separates fires, does not fire, and could not be evaluated — and it warns when the files were crawled with different rule catalogs. Each site is read with its own rules, and the panel names the sites that have a configuration.
crawlforge portfolio ./crawls
── Failing across the portfolio ─────────────
medium CANON-CROSS-DOMAIN 3 of 5 sites
critical HTTP-NO-HTTPS 2 of 5 sites
medium INDEX-DEEP-PAGE 1 of 5 sites (2 inconclusive)Audit before it ships
Three modes: a live site over HTTP, a built folder with no network at all — an Astro dist/ in CI — and an exact list of URLs. An interrupted crawl resumes where it left off, with the configuration stored in the file itself.
Recurring setups live in a YAML per site, and flags win over the file. Include and exclude patterns cut the crawl, and whatever is excluded is recorded as excluded — it does not vanish without a trace. A staging behind basic auth is crawled with a credential that never travels to another host and is never written to the crawl file.
The same YAML can carry a rules: block that turns rules off or changes their severity for that site. It is applied when reading: the crawl file keeps every raw finding, so changing your mind needs no new crawl. crawlforge config schema prints the file's JSON Schema, and an editor with a YAML language server completes keys and rule IDs as you type.
# In CI, over the built folder and with no network
crawlforge audit ./dist --base https://client.com/ --out new.sqlite
crawlforge diff reference.sqlite new.sqlite --fail-on high || exit 1Findings you can trust
72 audit rules, each with its own HTML fixture and test — a test fails if a rule ships without one. And when a crawl is truncated, the rules that need the complete link graph stay silent instead of guessing; the report says which ones could not be evaluated.
The same defect repeated by a template counts as one issue with its page count next to it, and a rule affecting 40% or more of the site states its share — a list of pages to fix one by one is not the same problem as one systemic cause.
Outbound links are checked by default: status only, one request in flight per host, and only a 404 or 410 counts as gone. An anti-bot wall answering 403 to a probe while the page opens fine in a browser is not a broken link, and is not reported as one.
It also goes easy on your server. Three 429s, 503s, dropped connections or timeouts in a row halve its concurrency on that host, and it climbs back one step at a time. A Retry-After is obeyed, up to two minutes, by pausing the whole host; a page still answering 429 is left pending for crawlforge resume instead of reported as broken, and the crawl summary says when the server pushed back.
medium META-TITLE-TOO-LONG 173,654 (80% of the site)
high ASSET-IMG-EMPTY-ALT-LINK 13 template issues (567 pages)
e.g. https://example.com/a · https://example.com/bA file you can take anywhere
A crawl is one SQLite file. Copy it, send it, open it months later without crawling again — report, export and diff work on the file, not on the network.
Read it in the terminal, export it to CSV or XLSX — fourteen sheets, verified to open clean in Excel — or produce a Markdown or HTML report to paste into a ticket. inspect answers an audit's most frequent question about any URL: who links here, with what anchor text, from which part of the page.
Any question the report does not answer, SQL does: the format is plain SQLite, with ready-made views for the usual questions.
The output reads in ten languages: English, Spanish, German, French, Italian, Dutch, Turkish, Brazilian Portuguese, Japanese and Polish, chosen with --lang or CRAWLFORGE_LANG. CSV and XLSX exports stay in English on purpose, because scripts read them.
crawlforge export crawl.sqlite --format xlsx --out audit.xlsx
crawlforge inspect crawl.sqlite '/that-page/' # who links here?
sqlite3 crawl.sqlite \
"SELECT url, status_code FROM urls WHERE status_code >= 400;"Install it in one line
No Rust needed. Homebrew, a one-line installer for a shell or for PowerShell, or cargo if you already have Rust.
# macOS and Linux, with Homebrew
brew install dcarrero/tap/crawlforge
# macOS and Linux, without Homebrew
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/dcarrero/crawlforge/releases/latest/download/crawlforge-installer.sh | sh
# Windows, in PowerShell
powershell -ExecutionPolicy Bypass -c "irm https://github.com/dcarrero/crawlforge/releases/latest/download/crawlforge-installer.ps1 | iex"
# With Rust 1.88 or newer
cargo install crawlforgeOr download the binary for your system from the latest release: macOS (Apple Silicon and Intel), Linux (x86_64 and ARM64) and Windows (x86_64), each with its checksum.
The macOS binaries are not notarized yet. Homebrew installs them without a warning, but a binary downloaded by hand needs xattr -d com.apple.quarantine crawlforge before Gatekeeper lets it run.
Or build it from source, with Rust 1.88 or newer:
git clone https://github.com/dcarrero/crawlforge
cd crawlforge
cargo build --release -p crawlforge
cp target/release/crawlforge ~/.cargo/bin/The numbers, with their conditions
If you are going to audit with this, check how it was measured first. The method sits next to every figure.
URLs in a single crawl, with memory flat
A news site with fifteen years of archive. Memory follows the pending queue, not the size of the site.
the method, in the devlog →extraction differences against an established crawler
The same 300 URLs to both tools. Status, title, meta description, H1, canonical and indexability.
the method, in the devlog →audit rules, each with its own test case
The rules are the product: when one is wrong, you stop trusting the whole report.
browse all 72 rules →What it does not do today
Said upfront, the same way the manual says it:
- It does not render JavaScript. A site whose content is assembled in the browser will look empty. It is planned.
- There is no graphical interface. The tool is the command line; native apps are an idea under consideration, with no dates promised.
- It does not schedule crawls. The panel reads the files your cron already produces.
- It does not follow external sites. Outbound links are checked for status; nothing of another domain is parsed, stored or crawled.
The full list lives in the manual and moves with each release.
The code is out. Every two weeks for the rest.
The work stays visible: what gets measured, what breaks, and how a real audit is run — with the commands and the numbers, on sites that exist. No download required to read any of it.