0.21.0 comes out the same day as 0.20.0, the release that let each site keep its own rules. It brings one new free rule, so the catalogue goes from 71 to 72, a crawler that slows down sooner when a server is struggling, a JSON Schema for the configuration file and an audit of every dependency before publishing.
The rule: LINK-FRAGMENT-BROKEN
An internal link like <a href="/page/#section"> promises a place inside the page. If the target
page has no element with id="section" and no <a name="section">, the browser opens the page at
the top and the promise is lost. LINK-FRAGMENT-BROKEN reports those links.
Matching follows the HTML standard. The fragment is tried as written, then percent-decoded, and the
comparison is exact and case-sensitive. Links within the same page count too. Some fragments are
never judged: a bare #, #top in any case, hash routes like #/… and #!… (those belong to
JavaScript apps) and text directives, which are cut first, so #x:~:text=… is judged as #x.
The target has to be internal and answer 200, the source page has to be 2xx, only <a> elements
are read and anything under /cdn-cgi/ is left out.
There is one finding per target page and fragment, grouped like the other template issues, so a
defect repeated by a theme shows up as one line with its page count. The severity is low because an
id added by JavaScript is not in the HTML the server sends, and the crawler only sees that HTML.
What it found on real sites
I ran it on four sites, 400 URLs each. On colorvivo.com it gave 396 findings, all for #content.
Every article starts with the usual accessible “skip to content” link,
<a class="skip-link" href="#content">, but the Elementor template for single posts has no
element with id="content". I checked one article by hand to be sure, and the home page does have
it. The report shows it as a single template issue across 396 pages, which matches what it is: a
one-line fix in the template. It’s also an accessibility bug, because a keyboard or screen reader
user who presses the skip link goes nowhere.
The other three gave nothing, and for good reasons. diariodelamancha.com has 379 skip links to
#content and the id exists. especialmundial.com has 398 to #contenido, also present. carrero.es
has 303 fragment links, 21 of them in tables of contents, and all of them land. I also audited
Wikipedia articles from disk: 1 finding out of 2,335 fragment links, #Early_Modern_Spain, a
navbox link to a section that article doesn’t have. That one is a true positive.
The real crawls also caught a bug before release. WordPress writes & and ' inside
href attributes, and the first version cut the URL at # before decoding entities, so it
produced fragments like 038;… that no page could ever have. Entities are now decoded first, and
there’s a test for it.
What it costs to store
Migration 012 adds two columns. links.fragment keeps the fragment, only for internal links.
pages.fragment_ids keeps the ids a page declares as a sorted list of 32-bit hashes, 4 bytes
each, looked up by binary search. A hash collision can only hide a broken fragment and can never
invent one, which is the safe direction for an audit.
Before /cdn-cgi/ was filtered, Cloudflare’s email-protection fragments, up to 1 KB each, were 6%
of a real crawl file. With the filter, the columns cost about 1.5% of the file on 400 URLs of
colorvivo, 1.3% on diariodelamancha and 0.1% on especialmundial. Three Wikipedia pages grew 5.4%.
What it costs to parse
This took more attempts than the rule itself. A lol_html [id] selector cost 9% with an empty
handler and 16% reading the attribute, on a 2 MB Wikipedia article with 9,187 ids, because it makes
the parser read the attributes of every element. A regular expression with captures cost 22%.
The version that shipped runs after the rewriter. It looks for the literal id= with the SIMD
literal search of the regex crate the engine already used, and checks the byte before each match
to reject data-id and ?id=. It costs 7 to 11% on long Wikipedia articles, 8% on one of 1.1 MB,
3% on a 324 KB WordPress article and between 0 and 3% on a 170 KB home page, roughly half a
millisecond for a 2 MB page. The release regression benchmark shows no measurable change, at
63,557 elements per second on the release build.
It has two known gaps, and both are documented: ID="x" in capitals and id = "x" with spaces are
not seen.
Older crawl files open and migrate, and the rule stays silent on them because the new columns are empty. As happened with 0.18.0’s migration, a 0.20.0 binary refuses a file written by 0.21.0.
A gentler crawler
The crawler already had an adaptive throttle per host. Three overload signals in a row halve the concurrency on that host, and it climbs back one step after 20 good responses. Until now only 429 and 503 counted as overload. Now dropped connections count too (resets, refusals, a body cut halfway), and so do timeouts, because a saturated server often doesn’t get as far as answering. DNS and TLS failures don’t count, since they would fail the same with the server idle.
Retry-After is now obeyed on the site’s own pages. A 429 or 503 that carries it, in seconds or as
an HTTP date in any of the three formats RFC 9110 allows, is retried after at least that long,
instead of the fixed backoff of 1, 2 and 4 seconds. The whole host waits in the meantime. Pausing
just that URL would let the other slots keep hitting the same server. The wait is capped at two
minutes: above that, the page isn’t retried, and a 429 stays pending for crawlforge resume, as it
has since 0.17.0. robots.txt and sitemaps honour the wait too.
The crawl summary now says when this happened. Against a local server answering 503 on 12 of 30 pages, it printed:
The server pushed back (429, 503, dropped connections or timeouts), so the crawl
slowed down: its concurrency was halved 3 times. Next time, try a lower --concurrency.On a healthy site it changes nothing measurable. The release benchmark stayed within noise against 0.20.0 over 8 alternating rounds, and a real 500-URL crawl of especialmundial.com took the same time with both versions, between 3.7 and 5.3 seconds.
There is a limit I’d rather say plainly. A host that stops answering altogether still gets four
attempts per URL. The bound for that case is --max-duration.
A schema for the configuration
crawlforge config schema prints a JSON Schema (draft 2020-12) of the YAML you pass with
--config, and it’s also published as docs/crawl-config.schema.json. The example file starts
with this line, so an editor with a YAML language server completes keys and rule IDs and underlines
typos as you write:
# yaml-language-server: $schema=https://raw.githubusercontent.com/dcarrero/crawlforge/main/docs/crawl-config.schema.jsonThe schema is generated from the code: rule IDs come from the catalogue and defaults from the engine, with no new dependency. Tests fail if it drifts from the configuration struct, which I proved by adding a dummy field. It also helps an assistant that writes the file for you, since it gets the real keys instead of guessing them.
The dependency audit
Before publishing, I ran the lock file through rust-audit-box, a tool of mine that runs cargo-audit, cargo-deny, cargo-geiger and clippy in a sandboxed container, with the build phase cut off from the network. It found 17 advisories.
The one that mattered was quick-xml 0.37.5, which parses sitemaps, and a sitemap is untrusted input.
It had two denial-of-service advisories: unbounded allocation for namespaces and a quadratic check
for duplicate attributes. Our code never reached either path (no NsReader, no attribute reads), so
moving to 0.42 was preventive. The rewrite had to keep two behaviours the new reader would have
broken. A bare & or a byte that isn’t UTF-8, like an ISO-8859-1 sitemap listing /café, costs
only that <loc>, not the whole sitemap. A 50,000-URL sitemap now parses in 26 ms, against 18 ms
before. The TLS stack (rustls 0.23.46 and webpki) and tokio 1.54.1 were updated too.
The zip advisories, rated critical and high, were about extracting archives, and the engine only
writes XLSX. Upgrading rust_xlsxwriter from 0.79 to 0.102, which brings zip 9, still paid off: the
XLSX export of a 700 MB crawl went from a peak of 1.23 GB of memory to 20 MB, in the same time
(about 18 seconds) and with identical cells. I checked that with openpyxl: 14 sheets, with the
Links sheet at 1,048,576 rows.
The final audit found no known vulnerabilities or RustSec warnings, and cargo-deny gave zero errors. A
deny.toml now writes the policy down: crates from crates.io only, nine allowed licences. The
workspace declares Apache-2.0 in Cargo.toml, where it said UNLICENSED although the repository
was already Apache 2.0. The minimum Rust version to build from source goes from 1.85 to 1.88.
Smaller things
The reasons a URL was excluded (robots.txt, a URL pattern, a folder audit with no network) are now
translated in report and inspect, and so is the “File:” line in the header of crawl and
resume. crawlforge rules | head no longer panics with “Broken pipe”: commands that never open a
socket now end quietly, like any Unix filter. crawl, list and resume keep the old behaviour on
purpose, so a server closing a connection mid-write can’t end a crawl.
Binaries built by CI for each release and a Homebrew tap are being set up.