12.7%have an llms.txt file127 of 1,000 sites
15.5%block at least one AI crawler155 of 1,000 sites
46%of sites that show PHP run an unsupported version37 of 81 sites
58.4%have no structured data on the home page584 of 1,000 sites
1. Most sites block AI training, not AI search
15.5% of sites block at least one AI crawler in robots.txt, and 3.4% block every crawler. But sites block the crawlers that collect training data about twice as often as the crawlers that fetch pages for AI answers. Most owners seem to want to appear in AI answers, without giving their content away for training.
Training crawlers blocked
GPTBot (OpenAI, training)12.9%
CCBot (Common Crawl)12.3%
Bytespider (ByteDance)12%
ClaudeBot (Anthropic, training)11.6%
Google-Extended (Gemini training)11.3%
Search and answer crawlers blocked
ChatGPT-User (ChatGPT browsing)5.7%
PerplexityBot (Perplexity search)5.5%
OAI-SearchBot (ChatGPT search)5%
Claude-SearchBot (Claude search)4.2%
2. Only 1 in 8 sites has an llms.txt
127 sites (12.7%) publish an llms.txt, the short guide that tells AI tools what a site is about. 119 of them start with the heading the format expects. 10 sites have an llms.txt but also block AI crawlers in robots.txt, so the file may never be read.
3. The search basics are still missing on many sites
Uses HTTPS99.6%
Has a sitemap69.3%
Sends an HSTS header46.3%
Has structured data (JSON-LD) on the home page41.6%
Has an llms.txt file12.7%
Almost every site uses HTTPS. But 58% have no structured data on the home page, 31% have no sitemap we could find, and 54% don’t send HSTS, which tells browsers to always use the secure connection.
4. Old software is easy to spot
Most sites hide their PHP version. Of the 81 sites (8.1%) that show it, 37 run a PHP version that no longer gets security fixes (8.1 or older, down to PHP 5.3), and 2 more miss the October 2026 security fix. 17.2% of all sites show their server software version, and 88 of the 209 WordPress sites (42%) show their WordPress version publicly.
What to do this week
- Check your robots.txt. Block training crawlers if you want, but allow the search crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot), or AI answers can’t quote you. AI search checker
- Add an llms.txt that names your business, says what you do in one line, and links your key pages with a short note each. llms.txt checker and generator
- Add structured data (JSON-LD) for your business, products or services, so search and AI tools read the facts correctly. AI search readiness checklist
- Hide software versions and update PHP. A public old version number tells attackers which known bugs to try. PHP version checker
- Running an MCP server for AI agents? Check its login and tools before you share it. MCP security checker
How we did it
- Sample: 1,000 websites picked at random from the Tranco list K97GW, a research ranking of popular domains, between rank 10,000 and 500,000. We skipped infrastructure domains (CDNs, APIs), government and adult sites, and domains without a working home page. 1,518 domains were tried to reach 1,000 websites.
- Date: 17 November 2026, from one location.
- What we read, per site: the home page, /robots.txt, /llms.txt and, when robots.txt listed none, /sitemap.xml. Nothing else. About four light requests per site.
- Limits: we only see what a site shows to the public. Many sites hide their software versions, and some block unknown visitors, so the security numbers are a lower bound. Structured data was checked on the home page only.
Want this for your own site? Our team helps businesses get found and quoted by AI search. See AI search optimization.