Fifty domains that counted as one site

The previous entry fixed the title of a report for a list of URLs that mixes several sites. It closed with something that stayed as it was, on purpose, because it wasn’t a bug in a title but a product question. The portfolio panel showed the same two-site list like this:

── Portfolio panel ──────────────────────────
  1 site · crawls from 2026-10-12 to 2026-10-12
  ...
  high      ASSET-IMG-EMPTY-ALT-LINK       1 of 1 site

The row in the table already named both sites. The header said one.

The panel counted files. That was right while one crawl file meant one site, and it stopped being right in 0.11.0, when a list started auditing every domain it contains. Nobody went back to the panel to check what that meant for its numbers.

Why it wasn’t just a matter of wording

If a header says “1 site” when there are two, you could call that a cosmetic problem. The same number had two other jobs, though.

The first is the column that gives the panel its point. portfolio exists to answer what fails on all your sites at once. If a rule fires on nine of twelve blogs, that’s almost never content. It’s the theme or a plugin they share, and one fix covers all of them. With files as the unit, a list of fifty blogs crawled into one file shows up as “1 of 1 site” for every rule it has, and the template problem you came looking for disappears.

The second is the tier’s cap. Pro allows ten sites in the panel, and the check counted files. Fifty domains in one list went through as one site. No one is paying for a plan yet, so nobody was getting anything they shouldn’t, but a cap that gives way to a text file with one URL per line doesn’t cap anything.

That entry left it as a decision, and the decision was to count sites.

One row per file, counts per site

The obvious way to count sites would be one row per site in the glance table, splitting a multi-site file into its domains. I didn’t go that way. The panel’s “what changed” section compares each file with the .prev.sqlite next to it, and that diff is a single file pair. Splitting the rows would mean splitting the diffs too, and that’s a different, larger job.

So a multi-site list is still one row, labelled with all its sites as in 0.14.0, and the counts add up its sites. The header, the “fires on N of M sites” column, the “N of M sites have a previous crawl” line and the cap all count sites now. The same list from before:

── Portfolio panel ──────────────────────────
  2 sites · crawls from 2026-10-12 to 2026-10-12
  ...
  high      ASSET-IMG-EMPTY-ALT-LINK       1 of 2 sites

It says 1 of 2 because all three findings in that file are on colorvivo.com and none are on especialmundial.com, and the panel has to know that. A rule that fires in a multi-site file doesn’t fire on every site in it. So the findings are split by the site of the URL they hang from, using the same rule as the engine: host and port.

That split runs in SQL, cutting the authority out of the stored URL. One of our crawls of diariodelamancha.com has more than 278,000 findings, and the panel shouldn’t parse every one of those URLs to count to three. It only runs for files that audited more than one site. A single-site file keeps going the way it always did.

Two holes I found in my own first version

The first version of that query had two bugs. I spotted them reading it back, before writing a single test.

Some findings belong to the site and not to a URL. INDEX-SITEMAP-MISSING is one, and so is INDEX-SECTION-DISCONNECTED (our crawl of colorvivo.com has one). They’re stored with no URL, so there’s nothing to cut. The query counted them as one more site, an empty one, and a rule with findings both on pages and at site level would have come out on one site more than it really was.

The other one was worse. A finding can point at any row of urls, external ones included, and HTTP-404-EXTERNAL, the broken outbound link, does exactly that: its finding hangs from the dead external URL, not from the page that links to it. The query counted that URL’s host as a site, so every broken link to another domain would have added a site to the spread.

Now only the sites the file audited count, the same ones its title names. A rule whose findings match none of them, as happens with site-level findings, counts once. That can undercount a spread in some odd case, but it can’t invent one, and in a table meant for deciding where to spend the morning, undercounting is the safer way to be wrong.

Incomplete crawls needed the same change. A list never builds the full link graph, so the rules that need it can’t be evaluated there, and the panel says “inconclusive” for those sites instead of counting them as clean. It used to count that per file. It now counts it per site: on a three-site list where an orphan page shows up on one site, the rule fires on 1 and is inconclusive on 2.

Breaking it four times

There are two new tests. One builds a three-site list plus an ordinary file and checks the total, four sites, along with the spread of two rules and an eleven-site list that doesn’t fit under Pro’s cap. The other checks the inconclusive count per site.

Then I broke each piece and watched it fail. With the total going back to counting files, the first test went red. With every rule in a file counting as one site, red. Without the filter on external hosts, red. The old inconclusive count turned the second test red. One test covers three of those, and it fails on a different line each time.

In passing: Ubuntu 26

From October 19, GitHub’s ubuntu-latest runner moves to Ubuntu 26, and the first push after that could break for reasons that have nothing to do with us. Before this release I ran the full CI on a throwaway branch with the ubuntu-26.04 image: tests, the rule fixtures and the performance regression. All of it passed. The branch is deleted.

This release doesn’t touch the engine or the rules, only the command line. The pre-publish regression came in at 64,858 elements per second, with the Mac’s load average around 20 when I checked right afterwards, and verificar.sh passed in full.

Corrected the same day: the first version of this entry said no rule hangs findings from external URLs. HTTP-404-EXTERNAL does. I’d checked crawl files from August, made before external links were probed at all. The code was right; the sentence wasn’t.