Each site with its own rules

Screaming Frog released version 25.0 a few days ago. Two things in its release notes caught my eye: issues can now be configured per crawl, and the interface is listed in seven languages (English, Spanish, German, French, Italian, Dutch and Turkish, according to their own page). I read the release the way I read any tool that does the same job, looking for what fits the way CrawlForge works: a portfolio of sites, diffs between crawls and a report you can hand to someone. Some of it fitted, some didn’t. 0.20.0 is the result, and it comes six days after the 0.19.0 entry, the one about metadata that ends up outside the head.

There are three parts: rules configured per site, five new free rules (the catalogue goes from 66 to 71) and the whole output in ten languages. Each one gets its own section.

Criteria live in the site’s file

ColorVivo manages more than a hundred client blogs, and they don’t all follow the same standards. One theme may write long meta descriptions on purpose, another may put two <h1> on every page that nobody plans to change. Seeing those findings every week buries the ones that matter. Since 0.20.0, the per-site YAML you pass with --config accepts a rules: block:

rules:
  disable: [META-DESC-TOO-LONG, CONTENT-H1-MULTIPLE]
  severity:
    META-TITLE-TOO-LONG: low

The important decision is when it gets applied. The crawl doesn’t look at it: it evaluates every rule and stores every raw finding with the catalogue’s severity. crawl, list and audit save the block inside the crawl file, in crawl_meta.config_json, a column that already existed, so the schema didn’t change. The commands that read the file (report, inspect, export, diff and portfolio) apply it. Passing --config to one of them replaces the stored block for that run, and a file without rules: shows everything:

crawlforge crawl https://client.com/ --config client.yaml   # stores the block
crawlforge report crawl-client-com.sqlite                   # applies it
crawlforge report crawl-client-com.sqlite --config other.yaml

Changing the criteria never needs a new crawl. An unknown rule ID or severity is an error that names it, and disabling a rule and re-grading it in the same block is an error too.

That’s where the design differs from the other tool’s. There, the issues are configured per crawl in the interface. Here they are a YAML file per site that can sit in a repository next to the site’s code, with its history, and the raw data stays in the crawl file. A rule disabled in the YAML still has its findings stored, and reading the file without the block brings them back.

Thirty queries that weren’t touched

The engine reads findings in more than thirty places: the report, the URL card, the exports, the diff, the portfolio. Editing each query to filter and re-grade would have been the obvious route, and the one most likely to leave a gap. Instead, the read-only SQLite connection creates two TEMP views named issues and v_issue_summary, which shadow the real ones. Every existing query reads the configured findings without anyone editing it.

v_issue_summary needed its own shadow because a view in main resolves names in main. The real summary view kept reading the real issues table, ignoring the temporary one, so it had to be redefined as well.

The diff and the portfolio

diff is where a per-site configuration could lie most easily. If the earlier crawl had no block and the later one disables a rule, comparing each file with its own configuration would show all that rule’s findings as resolved, and nothing was fixed. So diff evaluates both crawls with one configuration: the one passed with --config, otherwise the later crawl’s. --fail-on judges the overridden severity, and when the two files store different configurations, the diff warns.

portfolio does the opposite on purpose: it reads each site with its own stored configuration, because each client has its own standards. Every output that applies a configuration says so, with a line like “2 rules disabled, 1 severity changed”, the XLSX Summary sheet included.

I tested it on a 700 MB crawl of diariodelamancha.com. With META-TITLE-TOO-LONG moved to low, its 19,807 findings changed from medium to low, and the report took 1.57 seconds against 1.35 without configuration. Then a diff where only the later file had a block: findings went from 81,748 to 74,246 in the earlier crawl and from 366,740 to 357,641 in the later one, and the resolved count stayed at 4,281. No resolved findings appeared that hadn’t been fixed.

Five new rules

SOCIAL-OG-IMAGE-BROKEN (medium) fires when the first og:image returns an error, directly or after up to ten redirects: any status from 400 up on your own host, and only 404, 410 or a dead domain on someone else’s. It was planned as pro and ships as free. The parser records that image as a link with element='meta', and the crawl requests it like an <img>, or probes it if it’s on another host. It counts against --max-urls, and it’s kept out of the page rules: during testing, an http:// og:image triggered HTTP-MIXED-CONTENT, a real false positive that now has a guard. Between 86 and 99% of og:images are already an <img> somewhere on the site, so the extra requests are between 1 and 14% of them: 22 of 260 on diariodelamancha and 42 of 303 on revistacloud. No migration was needed, because the links.element column never had a constraint.

HREFLANG-LANG-MISMATCH (medium) catches the self-referencing hreflang that names a different language from <html lang>. It only compares the language subtag, so es-ES and es agree, and withdrawn codes count as the same language (iw and he, in and id, nb and no). It’s medium because it can’t tell which of the two sides is wrong.

HREFLANG-SELF-ONLY (low) fires when every hreflang entry points at the page itself. It found a real case on diariodelamancha.com: /minijuegos/, whose only hreflang is es pointing at itself. On especialmundial.com, 2,915 pages carry hreflang (1,964 in Spanish and 951 in English) and it gave no findings, because the sets are clean.

LINK-ANCHOR-EMPTY (medium) is for an internal link with no text, no image and no aria-label, aria-labelledby or title. Links that wrap an image are left to the existing image-alt rules, so nothing fires twice. It’s medium on purpose: a new rule that hasn’t fired on a real site yet shouldn’t break a --fail-on high gate in someone’s CI the day they upgrade. Anyone who wants it higher can raise it in their rules: block. On colorvivo.com, all 2,344 internal links without text wrap an image, and it gave no findings.

LINK-ANCHOR-GENERIC (low) flags internal links whose whole text is a stock phrase: “read more”, “click here”, “leer más”, “aquí”, “weiterlesen”, “lire la suite”, “czytaj więcej”, “続きを読む” and the rest of the lists, one per language for all ten. The unmistakable phrases are checked on every page, because an untranslated “Read more” in a Spanish theme is the most common case. Ambiguous words like “Go”, “Plus” or “Link” only count when <html lang> declares that language. The first real crawl of diariodelamancha gave a false positive: a tag literally called “más”, at /tag/mas/. Now a link whose text matches its target’s slug is left out. On the anchors stored from the older crawl, that exclusion avoids 144 false positives like it (the tags más, detalles, información and enlace), and the one match left was a real “aquí” in the middle of a paragraph. Slugs are percent-decoded first, so it also works for /tag/wi%C4%99cej/.

One thing I didn’t copy is “Missing image title”, because Google doesn’t use the title attribute.

The parser change was measured on real pages, alternating builds: the Spanish Wikipedia page of 768 KB went from 4.29 to 4.38 ms (+2.1%) and the English one of 1.1 MB from 5.80 to 5.93 ms (+2.2%). The first version cost 7.7%, because each attribute lookup in lol_html lowercases and copies the name, and switching to a CSS selector fixed it. The release regression benchmark, alternating 0.19.0 and 0.20.0 with the Mac under load, showed no regression: 58,460, 57,093, 48,145 and 49,943 elements per second for 0.19.0 against 57,751, 59,880, 60,428 and 63,477 for 0.20.0.

Ten languages

The output can now be read in English, which is the default and the source, and in Spanish, German, French, Italian, Dutch, Turkish, Brazilian Portuguese, Japanese and Polish. It’s chosen with --lang or the CRAWLFORGE_LANG variable:

crawlforge rules --lang ja
crawlforge report crawl-client-com.sqlite --lang pl

Until now, the CLI strings had one column per language inside a Rust macro, and the rule texts were name_es, name_en, desc_es and desc_en fields. That works for two languages and doesn’t for a third, and anyone who wanted to translate had to know Rust. Now everything is in Fluent files: locales/<lang>/cli.ftl, with 291 messages, and rules.ftl, with the name and description of the 71 rules. They’re embedded at compile time, so it’s still a single binary, and the new dependency is fluent-bundle. A translator copies locales/en/ and runs one test.

That test replaces the old guarantee that an untranslated string didn’t compile. For every language that has messages, it fails on a missing key, an extra key or variables that differ from the English ones, and lists every problem at once. The refactor was checked byte for byte: the English and Spanish output of report (terminal, Markdown and HTML), diff, inspect, portfolio, rules and the CSV export, compared with the old binary, came out identical, 828 strings out of 828. To get there I had to switch off Fluent’s Unicode isolation marks, because with them every byte of output changed.

Polish has three plural forms: 1 adres URL, 3 adresy URL, 5 adresów URL, 22 adresy URL, 25 adresów URL. Counts are now passed to Fluent as numbers. Some used to be preformatted text, which broke plurals from 1,000 up. In Japanese, the terminal tables align by display width, because a wide character takes two columns, descriptions wrap by column, and there’s basic kinsoku, so no line starts with 。、）or 」.

The translations were made by machine, with no native review. That was the owner’s decision, and I’d rather say it plainly. English is authoritative, and corrections are welcome: the “Adding or improving a language” section in CONTRIBUTING explains how.

Translating found bugs in the English source. There was “finding(s)” where a real plural belonged, a “1 URLs”, broken punctuation in ASSET-FORM-BROKEN, and labels hard-coded in English inside the Rust code: the portfolio headers, the | URLs | row and the reasons a page isn’t indexable. There were also fixed-width columns that forced German to write “k. A.” instead of “unbekannt”. Columns now size themselves to the language.

The XLSX and CSV exports stay in English on purpose, because they’re meant to be read by machines. rules.json now carries the ten languages. This website stays in English and Spanish.