crawlforge portfolio takes a directory of crawl files and answers two questions. The first is the
one you expect: what changed on each site since last time, ordered by impact, with new critical and
high findings at the top.
The second is the one that justifies the command existing. On how many sites does each rule fire?
── Failing across the portfolio ─────────────
A rule firing on most sites is rarely content: it is usually a shared template or
plugin — one fix that serves them all.
medium CANON-CROSS-DOMAIN 3 of 5 sites
critical HTTP-NO-HTTPS 2 of 5 sites
medium INDEX-DEEP-PAGE 1 of 5 sites (2 inconclusive)That column changes what the work is. A rule firing on 9 of 12 sites is almost never content. It is a shared template, a plugin everybody installed, or a deployment habit, and the one fix that closes it serves nine clients at once. Without the aggregate, that rule shows up as one more line in twelve separate reports, each read in a different week, and nobody ever puts them side by side.
Where the trap was
That number has a way of lying that does not show on the page.
Some rules can only conclude anything if the link graph is complete. INDEX-ORPHAN-PAGE, the rule
for pages that exist and that nothing links to, is the obvious one: you cannot claim a page is orphaned
without having seen every other page that might have linked to it. The same holds for pages buried
too deep, and for sections cut off from the rest of the site.
When a crawl is truncated, those rules never run. The engine switches them off deliberately, because a graph full of holes makes them fire on their own holes: on one site it reported 39 pages out of 40 as too deep, and the 39 were the URL cap rather than the site. Truncation is cheap to produce: hitting the URL limit, running out of time, a Ctrl-C halfway through the afternoon. A list-mode crawl has no graph either, by definition: it only ever saw the URLs you handed it.
So in any given file in your directory, those rules may never have executed at all.
Counting that as “clean here” is lying with arithmetic. The missing value becomes a zero, the zero goes into a count, and the count prints with the same air of measured fact as everything around it. In a single report that error costs you one site. In an aggregate it multiplies by the number of files in the directory, and it does it in the reassuring direction: the rule looks less widespread than it is.
Which is why the panel tracks three states per rule and site rather than two. Fires, does not fire, and could not be evaluated:
medium INDEX-DEEP-PAGE 1 of 5 sites (2 inconclusive)One confirmed, two unknown. The line is uglier and it is the only version you can read without being misled. A bare “1 of 5” invites you to close the tab. “1 of 5, and on two of them I do not know” sends you back to re-crawl those two, which is the correct next move.
The other two warnings go on top
There are two more ways to build a false aggregate out of files that are individually fine, and both of them print in the panel header rather than in a footnote nobody reads.
Different rule catalogs. If one site was crawled in April and another last week, the catalog has probably grown in between. A rule can be “missing” from the older site because it did not exist when that crawl ran, and comparing those two files rule by rule states something untrue. The panel warns, naming the versions it found.
Crawls spread over time. A portfolio whose files span three weeks is not a snapshot of anything. The “what changed” section would be covering a different period on each site, and the reader draws a trend out of a blend. The panel always states its date range, and when the oldest and newest crawls are more than a week apart it says so in plain words.
There is a third decision from the same family. A file that cannot be opened — not a crawl, another program’s database, a schema newer than the binary — is set aside in its own section with the reason, and the panel is produced anyway. One bad file must not cost you the other eleven, and it must not quietly vanish from the counts either.
Where the caution comes from
None of this is theoretical. On 1 August, crawling a real client site with the command line already finished, the orphan-page rule returned 1,912 findings, of which 1,867 were images.
WordPress publishes an image sitemap. Those uploads are internal, they are declared in a sitemap, and
no page links them with an anchor, because an image is used with <img src>. All three conditions
the rule asked for were met. The one the rule is named after, being a page, was never asked. They
all came out at high severity, on the CMS we work with most, in one of the rules that most
differentiates the product.
What that week left behind was not the fix, it was the fright. A number that looks like data can be built on a badly posed question, and the report prints it just as straight either way. Multiply that by a hundred sites on one screen and it stops being a false positive: it becomes a panel you no longer believe, which is the one outcome that makes the whole tool worthless.
What you can take from it
If you maintain anything that aggregates results from several sources — monitoring dashboards, code quality reports, client scorecards — the question worth asking it is this: what does a missing value look like in here?
If the answer is “a zero”, the panel is lying somewhere and you do not know where. A zero and a gap render identically in a table and mean opposite things, and in an aggregate that silence does far more damage than in a standalone report, because nobody is going to open the source files to check.
The test is quick. Feed a broken source into your aggregate on purpose — a crawl cut short, a host reporting nothing, a repository whose analysis failed — and look at what prints. If the totals have not moved and nothing anywhere says something is missing, you have found the bug.