DUP-CONTENT-EXACT has been in the catalogue since the first release, and on a WordPress site it
almost never fires. It compares the HTML byte for byte, and a nonce, a date or a recent-posts widget
is enough to give two pages with the same text different hashes. Tag and category pages that list
the same excerpts, or two posts written over each other, always get past it.
0.18.0 adds the rule that was missing, DUP-CONTENT-NEAR. It fires when two or more indexable pages
share 80% or more of their visible text. It’s medium, and it’s in the free tier, although the
catalogue had it down as Pro. The free tier limits scale, not what the tool tells you about your own
site, and this rule is very much the second kind. The catalogue now has 65 rules.
Getting to 80% meant throwing out two things that looked settled: the algorithm in the plan, and the threshold almost everyone uses.
SimHash didn’t survive the arithmetic
The plan said SimHash: one 64-bit fingerprint per page, and two pages are near duplicates if their fingerprints differ in only a few bits. It’s cheap and takes eight bytes. I built it first, and the rule’s own fixture raised the flag: two pages sharing nearly all their text came out 4 bits apart, above the cut-off of 3 I had set.
I did the maths before moving the cut-off. Two pages sharing 90% of their text land anywhere from 6 to 12 bits apart, depending on how the differences fall. With the cut-off at 9, a crawl of 200,000 pages has about 35 pairs that match by pure chance. Keeping the noise down means lowering the cut-off, and then real duplicates slip through. With 64 bits there is no cut-off that does both.
I switched to MinHash: 128 values per page, one per permutation, one byte each. The share of values
two pages have in common estimates how much text they share. That’s 128 bytes per page in a new
column, text_minhash. The migration is a safe one: an old file opens and resumes as before, its
pages just have no signature and the rule skips them.
The text is cut into runs of three consecutive words, lowercased. Navigation, footer, scripts and
<aside> are left out, because a sidebar repeated across the site would make every page look alike.
Pages with fewer than 50 runs get no signature: with that little text, similarity means nothing.
A version that passed the tests and estimated wrong
The first MinHash did its permutations in 64-bit arithmetic. It passed every unit test. Run against
especialmundial.com, it gave one pair of pages a similarity of 0.945, and I wanted to know if that
was true. I fetched both with curl, pulled out their three-word runs with a few lines of Python
and computed the exact similarity, no estimating: 0.85.
Ten points off, right where the threshold sits, is not noise. I rewrote the permutations as 32-bit multiplications, which the processor also does four at a time, and the same pair came out at 0.851. There is now a test that generates 600 pairs of known similarity and checks two things: that the mean error stays under 0.01, and that the spread is no more than 30% above what theory predicts.
90% found nothing
The most widely used tool in the field defaults to 90% for near duplicates, so that’s where I started. On especialmundial it found nothing real. What it did find was a false positive: the list of 222 stadiums in Spanish and the same list in English are 96% alike, because proper names don’t get translated and the list is mostly names.
That one had a direct fix. The two pages declare each other as translations with hreflang, and two
pages that say they are alternates are never put in the same group.
The threshold had no direct fix. I pulled every pair from the crawl with an estimated similarity between 0.5 and 1, and checked the ones near the cut-off by hand with the exact calculation:
| Case | Similarity |
|---|---|
| The same list in two languages | 0.96 |
| The same stadium published under two slugs | 0.72 to 0.85 |
| Template pages with different data | 0.62 |
| Unrelated posts on colorvivo.com | 0.25 at most |
The real duplicates sit between 0.72 and 0.85. They’re pages of about a hundred words that repeat the stadium’s name spelled two ways, and each spelling changes several three-word runs. At 90%, none of them show up. The threshold is now 0.80, with this table written into the constant’s comment so anyone who wants to move it knows what they’d leave out.
Lowering the threshold meant splitting the signature into bands again, which is what saves comparing every page against every other. With 16 bands of 8 bytes, 5% of pairs at 0.80 were missed. With 21 bands of 6, it’s under 1%.
What it found
On especialmundial, two groups. The Bernabéu exists at /en/stadiums/estadio-santiago-bernabeu and
at …-bernabeu-madrid, and the Durival de Britto at …-durival-britto-curitiba and
…-durival-de-britto-curitiba. Each is the same page published twice, and one of each pair has to
go.
Below the threshold there are more of the same kind, and I’m not hiding them: Saint Petersburg as
saint-petersburg and st-petersburg, and the Spanish versions of those same stadiums. Some renamed
stadiums have one page per name, the Comunale and the Artemio Franchi in Florence, or the Littoriale
and the Dall’Ara in Bologna. That may well be on purpose.
On colorvivo, zero, and I believe it. Its tag pages are noindex (121 of the 130 in the sample), and
the rule only looks at indexable pages. The nine categories that are indexable don’t resemble each
other.
What it costs
The signature is computed while each page is parsed, which is the hottest part of the engine. I timed parsing a real 2,700-word article, 200 times in a row, on 0.17.0 and on 0.18.0: it goes from 0.46 to 0.57 milliseconds, 24% more. The 64-bit version cost nearly three times as much. In an HTTP crawl, where each page takes tens of milliseconds to arrive, you won’t notice. The rule itself takes 73 milliseconds over the 2,327 pages of especialmundial.
I had to take that measurement separately, because the regression benchmark that runs before every release called it a draw. Its test pages have three words and never reach the 50 runs needed for a signature, so the benchmark runs none of the new code. It’s noted now: a change that touches text gets measured on a real page.
A column that said what it wasn’t
Going over the schema turned up one more thing. The text_hash column was documented as “simhash
for near duplicates”. It never was: it’s an exact hash of the text, filled in only on Pro. I fixed
the documentation and left the column alone, since it has another job. The new signature has its
own column so an old file is never read with the wrong signature.
The rule’s tests cover the exact boundary between 25 and 26 differing signature bytes, the bands,
identical HTML that the other rule already handles, noindex pages, empty or malformed signatures,
the transitivity of groups and the hreflang exception. The ones for the bands, identical HTML,
<aside> and hreflang I saw fail by removing what they test. CI passed on Linux, Windows and macOS.