Two sites, one name in the title

The entry for 0.11.0, the one about a list with two sites, ended with a short section on what still came from the first URL in the file. There were two things. The CRAWLFORGE_AUTH credential is only sent to the first line’s host, and the report is titled after it. The manual said so in both languages, and at the time that felt like enough.

It wasn’t. A limitation written down in the manual is still a report that says the wrong thing, and the manual isn’t what travels. The HTML file does.

What the title said

This morning I crawled a three-line list with 0.13.0: two pages of especialmundial.com and the home page of colorvivo.com. Since 0.11.0 the engine treats both domains as audited sites, so the crawl file holds findings for both. The report opened like this:

# Audit of https://especialmundial.com/

Mode `list` · started 2026-10-12 10:18:50

Everything under that heading mixes the two sites: the totals, the findings, the example URLs. A colleague who gets the HTML reads “Audit of especialmundial.com” and then finds colorvivo pages in the list of problems. Either they think the tool got confused, or they start fixing the wrong site. Neither one is good.

The credential and the title had been put in the same sentence, and they shouldn’t have been. The credential is a choice. When a list mixes sites, deciding which of them gets a password would be guessing, so it stays with the first line on purpose. The title isn’t a choice, it’s a fact about the file: which sites were audited. The crawl already knew the answer.

Where the list of sites comes from

The crawl file has no column listing the sites. The crawl_meta table has a base_url, and in a list that’s the first line. I considered adding a column with the full set, but a new column means a migration, and adding a migration touches four places in the code. It also wouldn’t help the files that already exist, which would keep showing the old title until someone crawled again.

The data was already there. In a list crawl, the URLs from the file are written at depth 0, and they’re the only rows at that depth. Whether a row is internal has been decided against every line of the list since 0.11.0. So the sites in a list are the distinct hosts of the internal rows at depth 0, in the order they were written, which is the order of the file. That’s a query any version of the file can answer.

The file I’d crawled with 0.13.0 that morning, opened with the new binary, now gets this:

# Audit of 2 sites: especialmundial.com, colorvivo.com

I didn’t have to crawl it again. Lists crawled with 0.11.0 or later get the new title. A list from before 0.11.0 is a different case: back then the other domains were marked external and weren’t audited, so the query finds one site and the title stays what it was. That title was right for that file.

A list with three hundred domains

A list can carry many sites, and a title that names three hundred of them isn’t a title anyone reads to the end. It names up to three and counts the rest:

# Audit of 5 sites: s1.example, s2.example, s3.example and 2 more

The number uses each language’s thousands separator, like the rest of the tool. In Spanish it says «Auditoría de 5 sitios: … y 2 más».

Then there’s ports. Since 0.10.0 the engine treats ejemplo.es and ejemplo.es:8443 as two different sites, because a staging server on another port isn’t the production site, and the title counts them the same way. https://ejemplo.es:443 is the same site as https://ejemplo.es written out in full, so it counts once. Both cases have a test, because a title that disagreed with the engine about how many sites there are would be a new way of getting it wrong.

The portfolio panel had the same problem

crawlforge portfolio puts several crawl files in one table, one row per file, and each row was labelled with its base_url. A list that mixed sites showed up as its first one. The row now uses the same list as the title:

URLs  index.  crit  high   med   low  info  crawled     site
 131       2     0     1     1     1     0  2026-10-12  especialmundial.com, colorvivo.com  (list crawl)

One thing is left as it was, and I’d rather say so here than have someone find it. The panel’s header still says “1 site”, because it counts files. The site cap in the plans counts files too. Whether a list with fifty domains counts as one site or fifty is a product question, not a bug in a title, and it shouldn’t change quietly as a side effect of this fix.

The tests

There are five new tests. Four check titles: two sites, five sites, two ports on one host, and the panel row. The fifth checks that a list with a single site, written once plain and once with :443, keeps the title it always had.

With the function that reads the sites switched off, the first four went red. The fifth stayed green, which is expected, since it guards against the opposite mistake: counting a site twice. Then I tried that mistake on purpose. I removed the line that skips sites already in the list, and the single-site test failed along with the two-site one.

The query only takes internal rows at depth 0, and I checked that condition the same way. Removing is_internal from the query turned the two-site test red, because the test leaves an external row at depth 0 behind. The comment I’d written next to it said that row was the destination of an external redirect. When I went into the engine to confirm it, it turned out that externals are written with no depth at all. The condition protects the query from a case that doesn’t happen today, and the comment now says that.

This release doesn’t touch the engine or the rules, only the command line. The pre-publish regression ran at 67,457 elements per second, and verificar.sh passed in full.

The credential is now the only thing that comes from the first line, and that one is deliberate.