We scanned 2,165 small business websites for the www duplicate content problem. 5.0% have it

This is a prevalence study, not a guide. If you already know what the www/non-www duplicate content problem is, skip to the results.

The hypothesis was simple enough to be worth testing: a meaningful percentage of small business websites serve their site on both example.com and www.example.com with no redirect between them, and nobody involved has any idea. Not a rounding error. A real slice of the web.

We scanned 2,165 sites to find out. The answer is 5.0%. A further 6.2% we could not measure at all, and that second number turned out to be the more interesting one.

Why this problem is worth measuring

When both hostnames answer with a 200 and neither redirects to the other, search engines have two addresses serving identical content. Links pointing at one do not add to the other. It is not catastrophic and it is not a penalty, it is just quiet, permanent leakage.

What makes it interesting is that it is invisible from the outside. Nothing looks broken. Both addresses load the site. Both look right. There is no error page, no warning in any dashboard the business owner ever opens, and no moment where anybody would think to check. The failure mode is not "site is broken", it is "site has been slightly underperforming for six years."

That combination, real cost plus zero visibility, is what makes it a good candidate for being widespread.

Building the sample

Source: OpenStreetMap, via the Overpass API. Small businesses commonly carry a website tag in OSM, the data is open under ODbL, and it is explicitly designed to be queried. We looked at Yelp, YellowPages and BBB first and did not use them, because scraping them is against their terms.

Eight trades, 300 domains each, except locksmiths where the entire US pool with a website tag came to 68. 2,168 rows, 2,165 unique domains once you account for three that appear in two trades.

Sampling is a seeded shuffle, so re-running reproduces the same cohort. If a future run gives different numbers, that has to mean the web changed, not that the sample did. That property turned out to matter more than we expected.

Exclusions. Social and site-builder hosts are dropped at cohort-build time. A Facebook page has no www/apex pair to get wrong, and counting it would dilute the denominator with domains the business does not control.

Eight trades, not one, for a specific reason. Our first pass was 26 locksmith sites, and a high rate there could easily have meant we had accidentally measured one web designer's client list rather than a segment of the web.

What we checked, and what counts as broken

For each domain we requested the root and one interior URL, on both hostnames, and classified the pair.

The full breakdown, since the chart above collapses ten verdicts into three:

Root verdict	n	Meaning
redirects_to_target	716	www correctly redirects to the bare domain
target_redirects_to_counterpart	642	bare domain correctly redirects to www
unresolved	364	the counterpart has no DNS record
blocked	134	a bot wall answered. We have no observation either way
redirects_elsewhere	114	it redirects, but somewhere else entirely
duplicate_content	106	both serve, no redirect, no canonical
duplicate_canonicalised	52	both serve, but a canonical tag resolves it
no_live_page	33	the counterpart answers, but serves no page
canonical_offsite	3	canonical points at a different domain
canonical_wrong_page	1	canonical points at a different page

102 domains are broken, out of 2,023 we could both observe and count. That is 5.0%.

Two adjustments to the denominator, both of which lower the number we get to

report:

• The 134 blocked sites are excluded, because we could not observe them.

Counting an unobserved site as healthy is exactly the quiet assumption that

produced our first wrong answer.

• 8 parked domains are excluded from both sides. They serve a registrar

parking page on both hosts, which is technically duplicate content and

practically meaningless, since there is no site there to rank.

Two rows deserve explicit argument.

duplicate_canonicalised does not count as broken. Those 52 sites serve on both hosts, but both copies declare the same canonical URL, so search engines consolidate them and the ranking signal stays together. A 301 would be stronger, since a canonical is a hint that can be ignored and a redirect is a directive that cannot, and both copies still get crawled either way. But "weaker than optimal" is not "broken", and folding them in would have lifted the headline by redefining the problem rather than measuring it.

blocked does not count as either. A challenge page tells you about the WAF, not the site.

Results by trade

Electricians are nearly three times the rate of roofers, and we do not have a confident explanation. The obvious guess is that trades cluster on different web design shops and hosts, and the configuration follows the builder rather than the business. Our provider data is consistent with that and does not prove it.

Note the right-hand column, because it is the whole story of this post. Roofers have by far the most unobservable sites, and in our first run those 42 were being counted as broken.

The homepage-only bias, which is the part most checkers miss

We checked two URLs per domain deliberately. A homepage-only scan is biased in the flattering direction, because the homepage is the single page most likely to have been configured correctly. It is the page people test.

Of the 2,165 domains, 1,581 had an interior URL we could check. 1,500 of those had a clean root, and 46 of those 1,500 (3.1%) had a broken interior page anyway.

That is a site that passes every homepage-based www checker on the internet while its actual content is duplicated. It happens when the redirect was written as a rule for / rather than a path-preserving rule for everything.

Counted across the whole sample, root-or-deep breakage is 156 of 2,165, or 7.2%. (That figure is the unadjusted one, for comparability with the first run.)

Treat the 3.1% as a floor, not an estimate. We sampled one interior URL per domain. A site that breaks some interior paths and not others is very likely to have been recorded as clean.

The bugs, which are the most useful thing in this post

Every one of these produced a plausible-looking number rather than an error, and every one of them failed in the direction of good news. We are listing them because if you run this kind of scan yourself, you will hit at least two.

1. No User-Agent header. Our first version sent no UA. A meaningful number of hosts answer that with a 403, which the classifier read as "the counterpart serves nothing unusual."

2. Apex/www discovery. An early version crawled one page and reported success, silently, when it should have been walking the site.

3. First sub-sitemap only. When a site had a sitemap index, we read the first child sitemap and stopped. Again: one page crawled, no error raised.

4. Bot walls scoring as duplicate content. This is the one that mattered. A Cloudflare or WAF challenge page is served identically on both hostnames, carries no canonical tag, and returns 200. That is *byte-for-byte the signature of duplicate content* as far as a classifier is concerned. A site behind a strict WAF was recorded as broken when we had never seen its pages at all.

What bug 4 actually cost, measured

We fixed it, added a distinct blocked verdict, and re-ran the identical cohort. This is the before and after:

Of the 203 domains our first scan called broken, 91 were bot walls. Not duplicate content. Not anything. We simply had not been allowed to look. The headline went from 9.4% to 5.0% on the same cohort, same seed, same script.

The per-trade table reshuffled completely. Roofers were our headline finding at 15.3% and are now the *lowest* trade at 3.5%, because roofers had the most sites behind strict WAFs. The single most quotable number in our first draft was an artifact of the measurement.

Four domains also changed for real reasons between the two runs, one day apart: three that were unreachable or redirecting elsewhere now serve duplicate content, and one got fixed. That is the genuine background churn rate, and it is small.

The general lesson

A surprising prevalence result should be treated as a suspected bug until you have hand-checked the probe against a site you already know the answer for. Ours failed toward good news four times in a row, and each time the number it returned looked entirely reasonable. There was no error, no exception, no warning. Just a number that was too high, in the direction that made our work look more interesting.

The asymmetry is what makes this worth taking seriously rather than treating as an anecdote. An inflated finding is not just a wrong statistic. In our case it was about to become an email telling a stranger their website is broken when it is not, and there is no second chance to correct that.

Who actually holds the problem

We identified the operator of each domain from its nameserver records rather than by fingerprinting the HTML. HTML fingerprinting returned "unknown" for 88% of this cohort, because most of these are custom builds with no platform marker in the markup. That 88% was a measurement of our method, not of the web. Every domain has NS records, they cannot be absent, and they name whoever is actually in a position to add a redirect.

Run against the 102 confirmed-broken sites:

Nameserver operator	Broken sites	Share
GoDaddy	34	33.3%
Cloudflare	19	18.6%
Turbify	6	5.9%
Network Solutions	5	4.9%
Route 53	3	2.9%
everything else	35	one or two each

One provider is a third of the entire problem. If the fix has to happen one small business at a time, it does not happen. If it can happen at the provider level, a single change covers hundreds of sites.

The correction inside the correction

Our pre-fix provider table said SiteGround held 30% of the affected sites, more than anyone. That was the largest single finding in the first analysis, and it is completely wrong.

SiteGround appears zero times in the confirmed-broken list. Not once in 102. Every single SiteGround domain that looked broken was one of their bot challenges being misread, and they are 60 of our 134 blind spots for the same reason.

Having spent a while irritated at their captcha, the fair thing to report is what the corrected data actually says: SiteGround-hosted sites in this sample have the lowest rate of this problem of any major provider, which is to say none. They also ship a redirect tool in Site Tools under Domain > Redirects with an explicit 301/302 choice, free, documented, no support ticket required. On the evidence here, it works and people use it.

One caveat worth flagging for anyone relying on it: the documented flow is URL-to-URL, "forward the visitors of a specific URL to another page." That is the shape that produces the deep-page failure in the previous section, where the homepage redirects correctly and interior pages do not. If you set one up, check an interior URL rather than your homepage.

And a finding that survived

Four of the providers have the problem on their own websites. Two hosting companies and two web design agencies, each hand-verified rather than trusted to the scanner: 200 on both hostnames, no redirect between them, no canonical on either. I re-checked all four while writing this section, because after the SiteGround reversal I did not trust anything the first analysis had told me. They still have it.

We are not naming them. The point is not that those four are careless, it is the opposite. The people who build and host websites for a living have this on the site they use to advertise building and hosting websites. That is the strongest available evidence that the problem is genuinely invisible rather than a marker of an amateur site. Nobody reading this should conclude "mine was professionally built, so I am fine."

What we could not do

Only 11% of affected domains yield a contact email. 22 of 203, measured twice: once with a naive homepage scrape and once with a careful seven-path extractor handling mailto links, entity encoding and obfuscation. The careful version did not beat the naive one, because this is not an extraction problem. Spot checks: one site returns 168 bytes of HTML, another 114, both JavaScript shells with no content in the markup. One serves 106KB with zero email addresses and zero forms. The addresses are not hidden. They are not there. Contact forms and client-side rendering replaced them.

So a study like this can tell you how big the problem is and cannot, by itself, tell the people affected. That is the part we are working on now.

What Part 2 will be

A prevalence number on its own is close to worthless. Plenty of people have published "X% of sites have Y" and nothing changed as a result. The interesting question is not how many sites have it, it is what happens when you tell them.

We have emailed 22 of them so far. All 22 are confirmed still broken under the corrected probe, so the outreach list is real even though the headline moved. Part 2 will report:

• How many we managed to reach at all, and through which channel
• Of those reached, how many fixed it themselves after being told, which we expect to be the largest group and consider a completely good outcome
• How many fixed it using our platform, counted separately and never blended into the first number
• How many did nothing, which is data too

Those stay separate counts because merging them is the easy way to make our own numbers look better, and this only means anything if you can check it. Every claim we make about a specific domain being fixed is verifiable by anyone with a browser and twenty seconds.

If you want to check your own site while you wait, the tool we used for all 2,165 of these is free and needs no signup. Paste an interior URL rather than your homepage. That is where this actually breaks.

---

Disclosure: we are RedirectIQ, and we host redirects, so we have an obvious interest in people discovering they need one. Three things that follow from that, which you should hold us to. The tool linked above is free and unauthenticated. Every number here is reproducible by anyone willing to run the same scan against the same seeded cohort. And when we tell a business about this, we tell them to check with their host first, because plenty of hosts do it for free, and if yours does then you should use it.

We also published the number that made us look worse. Our first answer was 9.4% and would have been a better headline. It was wrong, we found out before anyone else did, and telling you that is worth more to us than the four points.