Command line · Everything on this page exists in the current build
What it does today, and what that saves you
Grouped by what you need done, not by how the code is organised. Where a claim carries a number, the number links to how it was measured.
Know what a deploy broke
Crawling again onto the same file sets the previous crawl aside automatically, and the diff between the two tells you what was resolved and what appeared. A crawl is a snapshot; the diff is what makes it a routine.
In CI it exits non-zero when something serious gets in: --fail-on takes a severity, or the ID of one rule you cannot afford to regress on.
crawlforge crawl https://client.com/
# … the deploy happens …
crawlforge crawl https://client.com/ # the previous one becomes .prev.sqlite
crawlforge diff crawl-client-com.prev.sqlite \
crawl-client-com.sqlite --fail-on highThe whole portfolio in one panel
Point it at a folder of crawl files: what changed since each site's previous crawl, which rules fail across the portfolio, and one line per site, worst first. A rule firing on most sites is rarely content — it is usually a shared template or plugin, one fix that serves them all.
The counting is honest by design. A truncated crawl never evaluated the rules that need the complete link graph, so the panel separates fires, does not fire, and could not be evaluated — and it warns when the files were crawled with different rule catalogs.
crawlforge portfolio ./crawls
── Failing across the portfolio ─────────────
medium CANON-CROSS-DOMAIN 3 of 5 sites
critical HTTP-NO-HTTPS 2 of 5 sites
medium INDEX-DEEP-PAGE 1 of 5 sites (2 inconclusive)Audit before it ships
Three modes: a live site over HTTP, a built folder with no network at all — an Astro dist/ in CI — and an exact list of URLs. An interrupted crawl resumes where it left off, with the configuration stored in the file itself.
Recurring setups live in a YAML per site, and flags win over the file. Include and exclude patterns cut the crawl, and whatever is excluded is recorded as excluded — it does not vanish without a trace. A staging behind basic auth is crawled with a credential that never travels to another host and is never written to the crawl file.
# In CI, over the built folder and with no network
crawlforge audit ./dist --base https://client.com/ --out new.sqlite
crawlforge diff reference.sqlite new.sqlite --fail-on high || exit 1Findings you can trust
63 audit rules, each with its own HTML fixture and test — a test fails if a rule ships without one. And when a crawl is truncated, the rules that need the complete link graph stay silent instead of guessing; the report says which ones could not be evaluated.
The same defect repeated by a template counts as one issue with its page count next to it, and a rule affecting 40% or more of the site states its share — a list of pages to fix one by one is not the same problem as one systemic cause.
Outbound links are checked by default: status only, one request in flight per host, and only a 404 or 410 counts as gone. An anti-bot wall answering 403 to a probe while the page opens fine in a browser is not a broken link, and is not reported as one.
medium META-TITLE-TOO-LONG 173,654 (80% of the site)
high ASSET-IMG-EMPTY-ALT-LINK 13 template issues (567 pages)
e.g. https://example.com/a · https://example.com/bA file you can take anywhere
A crawl is one SQLite file. Copy it, send it, open it months later without crawling again — report, export and diff work on the file, not on the network.
Read it in the terminal, export it to CSV or XLSX — thirteen sheets, verified to open clean in Excel — or produce a Markdown or HTML report to paste into a ticket. inspect answers an audit's most frequent question about any URL: who links here, with what anchor text, from which part of the page.
Any question the report does not answer, SQL does: the format is plain SQLite, with ready-made views for the usual questions.
crawlforge export crawl.sqlite --format xlsx --out audit.xlsx
crawlforge inspect crawl.sqlite '/that-page/' # who links here?
sqlite3 crawl.sqlite \
"SELECT url, status_code FROM urls WHERE status_code >= 400;"The numbers, with their conditions
If you are going to audit with this, check how it was measured first. The method sits next to every figure.
URLs in a single crawl, with memory flat
A news site with fifteen years of archive. Memory follows the pending queue, not the size of the site.
the method, in the devlog →extraction differences against an established crawler
The same 300 URLs to both tools. Status, title, meta description, H1, canonical and indexability.
the method, in the devlog →audit rules, each with its own test case
The rules are the product: when one is wrong, you stop trusting the whole report.
browse all 63 rules →What it does not do today
Said upfront, the same way the manual says it:
- It does not render JavaScript. A site whose content is assembled in the browser will look empty. It is planned.
- There is no graphical interface. The tool is the command line; native apps are an idea under consideration, with no dates promised.
- It does not schedule crawls. The panel reads the files your cron already produces.
- It does not follow external sites. Outbound links are checked for status; nothing of another domain is parsed, stored or crawled.
The full list lives in the manual and moves with each release.
The code is out. Every two weeks for the rest.
The work stays visible: what gets measured, what breaks, and how a real audit is run — with the commands and the numbers, on sites that exist. No download required to read any of it.