A <link rel="canonical"> can sit in the source and count for nothing. It happens when something
that doesn’t belong in the <head> comes before it: a tracking <img>, a <div>, an <iframe>,
a stray word some plugin left behind. The browser closes the head at that point and opens the body,
even though </head> is still further down, and everything after it is now in the body. Google says
so on its page about valid metadata: if you use an invalid element in the head, it ignores anything
that appears after it.
Nobody notices, because when you open the source the canonical is right where it should be. 0.19.0
adds the rule that catches it, META-OUTSIDE-HEAD. It’s high, it’s in the free tier, and the
catalogue now has 66 rules. It watches the five pieces of metadata that matter for search: the
canonical, meta robots, hreflang, the title and the description.
What the finding says
The finding names what closed the head, with the attribute that identifies it: <img src="https://…">,
<div id="…"> or the text that shouldn’t have been there. On a WordPress site it’s almost always a
plugin, and that attribute tells you which one. Knowing the canonical doesn’t count isn’t much use
if you then have to dig through the HTML by hand to find the culprit.
Unlike almost every other rule, this one also looks at pages the crawl sees as noindex. That
noindex may be one of the lost tags. If it is, Google doesn’t see it, and a page we consider
excluded may well be indexed.
Following the standard, not a list
The naive version of this rule is a list: only title, meta, link, script, style, base,
noscript and template belong in the head, and anything else closes it. That’s how Google sums
it up, but there are cases where the list gets it wrong, and they’re among the most common.
The Meta Pixel is the first one. Its install snippet puts an <img> inside a <noscript>, and plenty
of people paste it into the head. Going by the list, that <img> would close the head on every page
that installs it that way, and none of them has anything wrong. The HTML standard treats <noscript> content as text when
the browser runs JavaScript, so that <img> closes nothing. The same goes for what’s inside
<template>, <script> and <style>.
The second is less obvious. A <link> or <meta> that shows up between </head> and <body>
doesn’t stay outside: the standard puts it back in the head. What opens the body in that gap is any
other kind of element, or text that isn’t whitespace.
Then there’s the case I worried about most: WordPress printing a PHP warning before the document,
something like <br /><b>Warning</b>: Undefined variable…. That <br> comes even before <html>,
the head never opens, and every piece of metadata on the page ends up in the body. No selector
reaches that far up, so the engine checks the bytes before <html> before it starts.
An oracle for the parser
The engine reads HTML with lol_html, which streams and never builds the document tree. That’s what makes it fast, but it also means I had to write the standard’s rules about where the head ends myself. I needed something to check them against.
This website’s own node_modules has parse5, a parser that implements the full WHATWG standard, the
one browsers follow. I wrote a script that feeds it a folder of HTML and records which metadata ends
up outside the head according to it, then compared that file by file with what the rule said.
The corpus had two parts. First, 26 cases written by hand to be awkward: uppercase tags, a stray
, an <svg> with its own <title>, loose closing tags like </p>, a custom element, text
before the doctype, the PHP warning. Second, the home pages of 81 real sites: the external domains
colorvivo.com links to and the 29 WordPress sites in the ColorVivo hub, one request each.
They agree on every file but one. It’s a stray </br> in the head. The standard turns it into a
<br> and opens the body, but lol_html doesn’t report closing tags that don’t close anything, so the
engine never sees it. It’s a miss that stays quiet, not one that cries wolf, and it’s written down
in the changelog.
The oracle also caught something I hadn’t seen coming. The first version of the check before
<html> gave a false positive on decoracion2.com. Its theme opens with Internet Explorer’s
conditional comments, <!--[if IE 8]><html class="ie8">, so the first <html in the file sits
inside a comment. My code looked for that <html and cut there, splitting the comment in half. I
rewrote it to walk forward from the start, skipping whole comments, and the test for it uses that
header exactly as it is.
None of the 81 home pages had the defect, and neither did the 600 colorvivo pages I crawled or especialmundial.com. The rule has been checked against false positives, but it hasn’t found a real case yet. When it does, I’ll write it up.
What it costs
The first version registered the handler with the * selector, which is the simplest option: it
runs for every element on the page and returns straight away once the head is closed. I timed it by
parsing four real home pages, between 400 and 545 KB, alternating 0.18.0 and the new build. Parsing
went up by 8 to 13%. Even if every call returns on its first check, it’s still one call per element
in the body.
It’s enough for lol_html to call the handler only on head *, html > *, meaning the head’s
descendants and <html>’s direct children, and to look at text only inside the head. That brought
the cost down to 3 to 7%, about 0.03 milliseconds per page. The Mac was under heavy load while I
measured, so treat it as an order of magnitude. The regression benchmark that runs before every
release gave 68,519 elements per second, in line with recent versions.
I saw everything fail before release, breaking each piece and watching its test go red: the stray
text, the <noscript>, the byte order mark some editors put at the start of a file, the narrowed
selector, the check before <html>, the noindex filter and the engine wiring. CI passed on Linux,
Windows and macOS.