A list with two sites and 121 stray requests

The 0.10.0 entry ended with a debt, in the part about what wasn’t there yet: list mode treated only the domain of the first URL in the file as internal. We said changing it in that release would have been bad timing, since it was already moving findings around, and that the manual would say so until we lifted it.

0.11.0 lifts it. This entry covers what was actually happening, because it was worse than the manual said.

What it did

crawlforge list urls.txt audits exactly the URLs in the file and nothing else. It’s the mode for reviewing the forty landing pages of a campaign, and it’s also the one we use to compare against another tool on equal terms. To decide what’s yours and what isn’t, the engine needs to know which site is “your site”. In crawl mode that’s obvious: the start URL’s site. List mode made the same call with the first line, and it worked as long as the file held a single domain.

That’s not our use case. We run more than a hundred blogs, and the natural thing is one file with the pages that matter from each. To measure it I built a twelve-URL list: six from especialmundial.com first, then six from colorvivo.com. With 0.10.0:

colorvivo.com        is_internal=0  done      6
colorvivo.com        is_internal=0  skipped  122  (121 with a status)
especialmundial.com  is_internal=1  done      6
especialmundial.com  is_internal=1  skipped  199  (0 with a status)

The six colorvivo pages were downloaded and audited, but stored as external. The 122 colorvivo pages they link to, which weren’t in the list, got the treatment of a link to a foreign site: a HEAD request each to check that it answers. That’s 121 requests against our own server that nobody asked for. The 199 from especialmundial, which was “the site”, were recorded without being fetched, which is what this mode promises.

Those probes produced two ASSET-BROKEN findings, broken resources used on six pages:

/cdn-cgi/scripts/5c5dd728/cloudflare-static/email-decode.min.js   404
/cdn-cgi/scripts/7d0fa10a/cloudflare-static/rocket-loader.min.js  404

These are the scripts Cloudflare injects to hide email addresses and to defer the rest of a page’s scripts. They return 404 to a bare request without cookies, and the engine already ignores /cdn-cgi/ paths on a site of your own on purpose: it records them and doesn’t fetch them. Colorvivo wasn’t “the site”, so the filter didn’t apply, and the report showed two errors that don’t exist. The manual never mentioned this. It said the other domains were treated as foreign, and the concrete damage only showed up when we crawled, as it usually does.

Two questions that used to be one

The obvious fix is to replace “the first line’s domain” with “any domain in the list”. For deciding what gets fetched and what gets probed, that’s enough. But “internal” was doing two jobs, and with one site both always gave the same answer.

The first job is deciding whether a URL belongs to one of the audited sites. That decides whether it’s fetched, whether it’s probed, and how it’s stored. The second is deciding whether a link stays inside the site of the page that carries it. That one feeds each page’s internal link count and rules like INDEX-NOFOLLOW-INTERNAL, which warns when you put nofollow on a link to your own site.

With two sites in the list, a link from blog A to blog B points at an audited URL and also leaves A. Counted as internal, every nofollow between two blogs of the same portfolio would raise a warning, and some portfolios do that on purpose. To a search engine A and B are two sites even when one person owns both, and the tool should see it the same way.

So there’s now one object with the two questions kept apart. With a single site, which covers crawl and the audit of a dist/ folder, it takes exactly the same path as before. The same list with 0.11.0:

colorvivo.com        is_internal=1  done      6
colorvivo.com        is_internal=1  skipped  122  (0 with a status)
especialmundial.com  is_internal=1  done      6
especialmundial.com  is_internal=1  skipped  199  (0 with a status)

No probes against our own pages, and the two ASSET-BROKEN findings are gone. Every other finding matches between the two versions.

The scary number that wasn’t ours

The pre-publish check reported 68,760 elements per second. 0.10.0 did 108,585. After last time, when a 24% drop got past the test, I didn’t trust a green run.

The reasonable suspect was the new code. The “is this an audited site?” check runs once per link on every page, and it’s the same function that cost 4% a month ago. That’s why the single-site case keeps the old comparison that allocates nothing, and only a list with several domains goes through a hash table.

Comparing against last month’s number proves nothing, though. What works is building both versions and measuring them on the same machine, in the same half hour, alternating the order:

original  60,625    new  61,730
original  63,400    new  60,039

A tie within the noise. The Mac had a load average of 23 while I measured, and that accounted for the whole drop. The changelog figure is a reference, but the measurement has to come from the same day and the same machine.

What still comes from the first URL

Two things stay tied to the first line on purpose. The CRAWLFORGE_AUTH credential is only sent to its host, because picking which of several sites gets a password would be guessing. And the report is titled after it. The manual says so in both languages.

Both new tests were seen failing before we trusted them: with the fix removed, each block of checks fails on its own. They set up two sites on one test server, since to the engine 127.0.0.1 and localhost are different domains even though both end up on the same machine. That’s 1,054 tests passing.