Ten deploy bugs that quietly cost you traffic, found on real production sites
None of these break a page. The site loads, the design is intact, nobody files a bug. They just stop pages from being found — and because there is no error to see, they can sit there for years. Every example below is a live production site, most of them run by organisations considerably larger than us, and every one was re-checked the day this was published.
We build an open-source checker called sitepreflight. To find its bugs we ran it, session after session, against sites we do not own — a different site generator each time, on the theory that our own small flat site could never exercise the interesting parts. That turned out to be true: it found two dozen bugs in our own tool. What we did not expect was how much it found on the sites themselves.
These are ten of those findings, grouped by failure class. Each one names a real site, states exactly what we observed, gives you the one-line command to run the same check yourself, and says what the fix is. We re-verified all ten on 8 August 2026 before publishing, because publishing a stale accusation about someone else's site is its own kind of dishonesty. If one is fixed by the time you read this, that is a good outcome.
One framing note before the list. Every bug here has the same shape: a machine reads your site differently than you do. You look at a page; a crawler reads a header, a tag, a status code. The gap between those two readings is where all ten of these live.
1. No sitemap and no robots.txt at all
The base case, and more common than you would guess on genuinely good sites. A crawler arriving at your domain has two conventional places to ask "what is here and what may I read" — /robots.txt and /sitemap.xml. If both are missing, discovery falls back entirely to whoever links to you.
Found on: www.11ty.dev, the site of the Eleventy static site generator. Both /robots.txt and /sitemap.xml return 404. Eleventy is a tool for building static sites, which makes it the clarifying example: whatever the reason, it is certainly not a knowledge gap.
Why it costs traffic: a sitemap is not a ranking factor and will not rescue a page nobody wants. What it does is shorten the time between publishing a page and a crawler learning it exists — and for a new or deep page with few inbound links, that gap can be indefinite.
curl -sI https://example.com/robots.txt | head -1
curl -sI https://example.com/sitemap.xml | head -1Fix: generate both at build time. Every static site generator has a plugin; every CMS has a setting. It is a ten-minute job that nobody schedules.
2. A perfectly good sitemap that robots.txt never mentions
The subtler version of bug 1, and the one we like best because of who it belongs to.
Found on: www.hubspot.com. Their robots.txt is alive and enormous — 200 OK, 12,606 bytes of carefully maintained crawl rules. The string sitemap appears in it zero times. Meanwhile www.hubspot.com/sitemap.xml returns 200 and roughly half a megabyte of XML. The map exists; the signpost does not point at it. HubSpot sells SEO software.
Why it costs traffic: less than it used to — the major engines will guess /sitemap.xml and you can submit it in Search Console directly. But the Sitemap: line is the only discovery route that works for a crawler you have never heard of and cannot log in to, and it costs one line.
curl -s https://example.com/robots.txt | grep -i sitemapFix: one line, absolute URL, anywhere in the file: Sitemap: https://example.com/sitemap.xml.
3. robots.txt points at a sitemap that is dead
Worse than not declaring one, because it looks correct in every review that consists of reading the file.
Found on two sites, two different flavours. statamic.com serves a 25-byte robots.txt with no Sitemap: line, and statamic.com/sitemap.xml returns 404 — dressed as a 75 KB styled HTML error page, which means anything checking only for "did I get a document back" sees success. Separately, www.editorx.com/sitemap.xml returns 410 Gone: a deliberate, explicit "this used to exist and will not return."
Why it costs traffic: the 404-as-HTML case is the dangerous one. It is invisible to any tool that does not check the status code, and invisible to a human who visits the URL and sees a page rather than a browser error.
curl -so /dev/null -w '%{http_code}\n' https://example.com/sitemap.xmlFix: check the number, not the page. Anything other than 200 on a URL you declared is a broken promise to every crawler that took you at your word.
4. Your sitemap lists URLs that are not yours
A sitemap is a statement about your site. Entries pointing at another host are, at best, ignored.
Found on: www.joomla.org. Their sitemap — declared in robots.txt as /sitemap, not /sitemap.xml, which is a good reminder to read the Sitemap: line rather than assume the conventional path — contains 342 URLs on www.joomla.org and five that are not: one each on downloads., volunteers., tm., showcase. and community.joomla.org. A subdomain is a separate origin as far as this file is concerned.
Why it costs traffic: those five pages get no benefit from being listed — they need to appear in the sitemap of the host they live on. This is nearly always the fingerprint of a hand-maintained sitemap or a generator pointed at the wrong base URL, so it is worth investigating as a symptom, not just patching.
curl -s https://example.com/sitemap.xml \
| grep -o '<loc>[^<]*' | cut -d/ -f3 | sort | uniq -cFix: one sitemap per origin, and check your generator's configured base URL — a mismatch there tends to be causing other problems quietly.
5. A sitemap URL that redirects to the homepage
This one is genuinely hard to see, and we only found it because fixing an earlier bug in our own checker exposed it.
A page is removed. Rather than let it 404, someone adds a catch-all redirect to the homepage — a well-intentioned move that feels tidier. But the URL stays in the sitemap. Now you are formally telling crawlers "this page exists, please index it," and every crawler that follows the invitation lands on your homepage instead. Google files that URL as a page with redirect and never indexes it.
Found on: a CMS vendor's own sitemap, in July 2026. We are not naming it, because when we re-checked before publishing, the URL was no longer in their sitemap and we could not re-verify the original finding to the standard we hold ourselves to. The class is real and worth checking; the accusation is not one we can currently support.
Why it costs traffic: a blanket redirect-to-home is worse than a clean 404 for a page that is genuinely gone. The 404 tells the crawler to drop it. The redirect keeps it in limbo and burns crawl budget on every visit.
curl -so /dev/null -w '%{http_code} %{redirect_url}\n' https://example.com/some-old-pageFix: remove the URL from the sitemap when you remove the page. Redirect to a genuinely equivalent page or 410 it; the homepage is not equivalent to anything.
6. No rel=canonical anywhere on the site
Found on three sites in three different weeks: super.so, www.joomla.org and statamic.com. On every page we checked, the count of rel="canonical" in the served HTML is zero.
Why it costs traffic: without a canonical, the engine decides for itself which URL is the real one, and it has more candidates than you think — http and https, www and apex, trailing slash and not, plus every tracking parameter anyone has ever appended to a link to you. Each variant is a separate URL until told otherwise. The canonical is how you tell it otherwise.
This one is easy to get wrong in the opposite direction too: a canonical on every page pointing at the homepage, usually from a mis-set template variable, is far worse than no canonical at all. It asks the engine to drop your entire site in favour of one page — and engines will often oblige.
curl -s https://example.com/some/page | grep -o '<link[^>]*canonical[^>]*>'Fix: self-referencing canonical on every indexable page, absolute URL. Then check a second page, not just the homepage — the homepage is the one that is always right.
7. The tag is there, and it is empty
The gap between "present" and "populated" is where a whole category of audits gives you a clean bill of health you have not earned.
Found on: www.mozilla.org/en-US/products/monitor/, which ships <meta name="description" content="">. Grep for name="description" and it is there. Read the value and there is nothing in it.
Why it costs traffic: an empty description hands the search engine the job of writing your snippet from page text. Sometimes that is fine. On a product page, the snippet is your ad copy in the only place it is guaranteed to be read, and you have declined to write it.
This one also taught us something about our own trade. Our checker used to report an empty tag as "no meta description," which is false — the owner greps, finds the tag, and stops trusting the entire report. Absent and empty are different states and a report has to say which it saw.
curl -s https://example.com/page | grep -o 'name="description" content="[^"]*"'Fix: treat an empty value as a missing one in your own tooling, and find out why the template rendered nothing — it is usually a field that is optional in the CMS and empty on more pages than the one you found.
8. A page that sells something, with no description and no canonical
Found on: www.saddlebackleather.com. On the homepage of a working ecommerce storefront, the count of name="description" is zero and the count of rel="canonical" is zero.
We flag this separately from bugs 6 and 7 because of where it is. A missing description on an archive page is housekeeping. On a page whose job is to convert a stranger who arrived from a search result, the description is the search result. It is the only sentence you control in the entire interaction, and blank is the one option worse than mediocre.
A caution from checking this one: a naive grep for name="description" can also match analytics JavaScript that happens to contain the selector string, which is how a page with no description can appear to have two. If a count surprises you, look at what actually matched.
curl -s https://example.com/ \
| grep -oE '<meta[^>]+name="description"[^>]*>'Fix: obvious. The interesting question is how it went unnoticed on a revenue page, and the answer is always the same — nobody looks at the served HTML, they look at the rendered page, and the rendered page looks fine.
9. Links, in your own navigation, to pages that 404
Found on: super.so. Running the loop below against their homepage returns five dead internal links today — /memberships, /consulting-services and three UUID-style paths, every one of them a 404 that the homepage links to by href.
Why it costs traffic: a crawler following your own navigation into a wall learns something about the site's maintenance state, and a human doing it learns something worse. A dead link on a pricing or contact route is a lost order, not just a lost ranking.
These links survive because they are usually in a shared component. Nobody navigates their own site through the footer, so the broken entry is invisible to everyone who could fix it and visible to everyone who cannot.
curl -s https://example.com/ | grep -o 'href="/[^"]*"' | sort -u \
| cut -d'"' -f2 \
| while read -r p; do
printf '%s %s\n' "$(curl -so /dev/null -w '%{http_code}' "https://example.com$p")" "$p"
done | grep -v '^200'Fix: check links from the built output on every deploy, not by hand. A crawl of your own navigation is cheap and it is the check with the highest hit rate of anything on this list.
10. The page declares its encoding and your tools ignore it
The last one is about how you read a site, and it is the reason we trust the other nine.
Found on: www.lib.ru, a large Russian library site that serves Content-Type: text/html; charset=windows-1251 — correctly declared, exactly per spec. But fetch()'s response.text() in Node always decodes as UTF-8 and ignores the declared charset. So the title arrived at our checker as 4,776 replacement characters, every text check ran on garbage, and we would have reported title and description problems on a site that has neither.
Why it matters to you: the same failure, in the other direction, is real mojibake on a real page — the café-becomes-café class of bug, which happens whenever content is written in one encoding and served as another. Both look identical in a naive tool, and only one is your problem. A tool that cannot tell them apart will confidently report the wrong one.
curl -sI https://example.com/ | grep -i '^content-type'Fix, if the page really is mojibake: serve UTF-8 and declare UTF-8. Legacy encodings are legitimate but every layer must agree, and the layer that disagrees is usually a database connection nobody has touched in a decade.
Three failures that are not your bug, and must never be reported as one
Half the bugs we fixed in our own checker were not missed detections — they were fabricated findings, and that is the failure mode to fear in any tool you let near a deploy gate. Three worth knowing about, because most link checkers still get them wrong:
- A 429 is not a dead link. Hammer a site with parallel requests and it will rate-limit you. Report that as a broken link and you have blamed the owner for your own burst. Back off, retry once, and if it still throttles, say you could not check it — do not say it is broken.
- A 403 to HEAD is not a 404. Some servers answer HEAD with 403 and the identical GET with 200. We fabricated six dead links on one enterprise site this way. Re-check every failing HEAD with a GET before you count it.
- A timeout is a fact about your network, not their site. If the host is unreachable from your CI runner but fine from everywhere else, the honest output is "could not be reached from this machine" — and it must not fail the build. A checker that red-builds on a flaky network gets switched off within a week, and then it checks nothing at all.
The general rule: a tool that reports a problem the owner can disprove in thirty seconds has spent its entire credibility, including on the nine findings that were correct. That standard is why the ten items above each name a specific observation you can reproduce with one command rather than a severity score.
How to check your own site
All ten checks above are in sitepreflight, which is MIT-licensed, has zero dependencies, and needs no signup. One command, nothing to install:
npx -y github:iamphera/sitepreflight check https://example.comIt reads your sitemap, checks each page, and exits 1 if anything fails — so it works as a deploy gate in CI, and as a GitHub Action if that is where your deploys live. It can ping IndexNow after a clean run, which notifies Bing, Yandex, Seznam and Naver. Google does not participate in IndexNow, and anything claiming otherwise is wrong.
If you would rather not run anything, the free audit on this site runs fourteen of the same on-page checks against a single URL in your browser. No account, nothing stored.
Or have it watched for you
sitepreflight watch — $5/month. We run the full check against your site every week and email you the report. If nothing is wrong, the email says so in one line. If something broke since last week, you find out from us rather than from a traffic graph three months later. Cancel any time; refund on request, no argument.