How to check that your crawler is not lying to you

An SEO crawler has a curious problem: it has no way of knowing whether what it extracts is correct. It can have thousands of tests, all green, because tests compare its output against what its author expected. If the author misunderstood something, the test enshrines the mistake.

The only way out is checking against another implementation. We did, and found a defect that had been there for months.

Crawling is no good for comparing

The first thing anyone thinks of is crawling the same site with both tools and seeing whether they agree. It does not work.

Each crawler decides where to start, which links to follow, when to stop, how to normalise URLs and what to do with parameters. Two crawls of the same site produce different sets of URLs, and what you end up comparing is two journeys rather than two extractions. Almost every difference you see will be about scope.

The right way is list mode: a file of URLs and the instruction to crawl exactly those. Most serious tools have it, and it brings an extra benefit — it does not depend on licence limits by URL count, because you choose how many go in.

The setup

We pulled 300 URLs from a previous crawl. All internal, all returning 200, all HTML:

sqlite3 crawl.sqlite "SELECT url FROM urls
  WHERE is_internal = 1 AND status_code = 200
    AND content_type LIKE 'text/html%'
  ORDER BY url LIMIT 300;" > list.txt

The ORDER BY url is not cosmetic: without it, two runs give you different sets and you lose the ability to repeat the test.

Then both tools against that same file. Ours:

crawlforge list list.txt --concurrency 4 --out cf.sqlite

And the other in headless mode, which usually exists even when it is barely documented. Two things cost us time: it insists on absolute paths, and the CSV it exports carries its headers in the interface language. With the application in Spanish, the columns are called Dirección and Título 1.

Comparing field by field

Six fields: status code, title, meta description, H1, canonical and indexability. Eighteen hundred comparisons.

Normalising whitespace before comparing is essential. One tool may collapse consecutive spaces and the other may not, which fills your result with differences that are not differences:

def norm(s):
    return re.sub(r"\s+", " ", (s or "").strip())

The first pass gave us 290 canonical differences, which was too good to be true. And it was not: we were reading a column that did not exist in that CSV, so every canonical was being compared against an empty string. The column was called Elemento de enlace canónico 1.

With that fixed, the real result:

OK  status           0 of 300
OK  title            0 of 300
OK  description      0 of 300
≠   h1               1 of 300
OK  canonical        0 of 300
OK  indexability     0 of 300

The difference

One, in the home page H1:

<h1>Agencia Especializada en WordPress<br />con +25 años de experiencia</h1>

We returned Agencia Especializada en WordPresscon +25 años de experiencia.

The <br /> splits the element’s content into two text nodes. Our parser accumulated them and joined them with nothing in between, so the words at the boundary ran together. The other tool got it right.

The same defect affected anchor text: <a>Read<em>more</em>here</a> came out as Readmorehere.

Why 810 tests missed it

The fixtures we had written for headings looked like this:

<h1>A normal heading</h1>
<h2>Another heading</h2>

Textbook HTML. No element inside, no <br>, no styling <span>. And since the test compared our output against what we expected, and we expected what the code produced, everything lined up.

The fix was inserting a separator when each text node closes; the extra space is cleaned up by normalisation that already happened afterwards. Four lines.

Writing the test for it, we made the same mistake again. The first version was:

<a href="/x">Read <em>more</em></a>

With a space before the <em>. That test passes with the bug in place, because the space is already in the text, so it proved nothing. Remove it and it fails as it should.

What you can claim afterwards

Rerunning the test after the fix: zero differences across 1,800 comparisons.

That does not mean our extraction is correct. It means it agrees with another mature implementation, which is a different thing and rather more useful than it sounds: if you are going to switch tools, the first thing you need to know is that the data will not shift underneath your reports.

Worth saying what the test does not cover, too. It is 300 URLs of one site, all returning 200 and all HTML. It says nothing about redirects, server errors, non-HTML content, or sites that build their content with JavaScript. And agreeing on six fields does not guarantee agreeing on the rest.

Do it with yours

If you maintain anything that extracts data from web pages, this test costs twenty minutes and finds things. The whole recipe:

1. Pull a list of 200-300 real URLs, sorted, from a site with varied HTML.
2. Run it through both tools in list mode.
3. Normalise whitespace before comparing.
4. Check the exact name of every column before trusting a zero.
5. When you find a difference, go to the original HTML and decide who is right.

That last step matters: it is not always you, and it is not always them. We were the ones wrong this time, and knowing it was worth far more than the time it took to find out.