The local AEO lab: what a crawler trap taught us about AI visibility
Half a million bot requests in two days, a universal soft-404 and a business center in Elda: the first lessons from Blobic's public AEO laboratory.
Before relaunching blobic.com we looked at its traffic in Cloudflare. The numbers seemed great: hundreds of thousands of requests per day. The reality was a lesson in how AI crawlers actually behave — and why technical hygiene is the unglamorous half of AEO.
The crawler trap we found on our own domain
The previous site answered every URL — any URL — with HTTP 200 and the same homepage. No real 404s. For a crawler, that turns a domain into an infinite space of duplicate pages: every malformed relative link spawns new crawlable URLs forever. In two days, one AI crawler made over half a million requests chasing recursive garbage paths like /economia/js/css/favicon.ico.
That traffic looked impressive on a dashboard. It was actually teaching AI engines that the domain was an endless pile of identical noise. The fix was boring and decisive: a static site that returns real 404s, plus 301 redirects for the legacy URLs that still carry value — old topic feeds now point to our blog RSS, and historic local sections point to the lab pages.
What the Elda pilot keeps teaching
The lab's first case, a real business center in Elda, follows the same pattern: the blockers are rarely exotic. A bot-management layer was injecting contradictory robots rules for AI crawlers. The contact form didn't send. The sitemap listed two pages. Service pages didn't exist. None of this shows up in a content-marketing deck, and all of it caps your visibility before content even matters.
- Verify, with real requests, that GPTBot, ClaudeBot, PerplexityBot and Googlebot get a 200 on your key pages.
- Kill soft-404s: unknown paths must return 404, not a disguised homepage.
- Redirect legacy URLs with intent: feeds to feeds, sections to their modern equivalents, junk to nothing.
- Then measure prompts repeatedly — visibility is a probability, not a screenshot.
Every lab case is documented publicly — including the parts that resist, like external caches that keep showing a business's old address weeks after the website was corrected. That honesty is the point: it is the same system our partner agencies resell, shown working in daylight.