When someone asks ChatGPT, Claude, Perplexity, Gemini or Copilot "which company should I hire for this?", the assistant can only recommend businesses it can read and understand. Many good websites are invisible to AI assistants for simple technical reasons. This checklist covers the ten things that decide whether AI search can find, understand and quote your site, in the order we check them.
Want the answer for your own site first? Run the free AI search checker; it tests all of this in about 30 seconds.
1. Let the AI search crawlers in
Each AI company uses named crawlers, and your robots.txt file decides which may read your site. The important difference is between search crawlers and training crawlers:
| Company | Search and live reading | Training |
|---|---|---|
| OpenAI | OAI-SearchBot, ChatGPT-User | GPTBot |
| Anthropic | Claude-SearchBot, Claude-User | ClaudeBot |
| Perplexity | PerplexityBot, Perplexity-User | |
| Googlebot (Search and AI Overviews) | Google-Extended | |
| Microsoft | Bingbot (Bing and Copilot) |
Blocking training crawlers is a business choice. Blocking search crawlers means the assistant cannot read your pages when a customer asks about you, so it will not quote or link to you.
2. Check your firewall and CDN, not only robots.txt
Some CDNs and firewalls can block AI bots with a single switch, and on some services that protection is on by default for new sites. A friendly robots.txt does not help if the firewall returns an error page to the crawler. Review the bot settings in your CDN or hosting panel and allow the AI search crawlers you want.
3. Make sure your content is in the HTML
Most AI crawlers read the HTML your server sends and do not run JavaScript. If your page builds its text in the browser, they see an almost empty page. Use server-side rendering or static pages for important content. A quick test: view the page source (not the inspector) and search for a sentence from your page.
4. Add structured data the right way
Structured data (schema.org JSON-LD) tells machines who you are and what a page is about: Organization or LocalBusiness, Service or Product, FAQPage, Article and BreadcrumbList. Put it in a normal <script type="application/ld+json"> tag in the HTML. We found that our own site added it with JavaScript, so crawlers that do not run JavaScript never saw it; the fix took an hour.
5. Answer the questions customers ask
Assistants look for short, direct answers. For each main service, answer in plain words: what it is, who it is for, where you work, how the process works, how long it takes and what drives the price. Question-style headings with a two- or three-sentence answer under each work well.
6. Keep your business facts consistent
Use one business name, address, phone number and list of services on your site, your Google Business Profile, LinkedIn and directories. Mixed facts make an assistant less sure which business you are, and less likely to recommend you.
7. Show proof
Case studies with real numbers, client names when you may share them, reviews, certifications and named authors with experience give assistants (and people) a reason to trust you.
8. Be in the search indexes AI uses
AI answers are built on search indexes. Verify your site in Google Search Console and Bing Webmaster Tools, submit your sitemap, and use IndexNow so Bing and other engines see new pages quickly. ChatGPT search uses Bing data, so Bing matters more than many people think.
9. Add an llms.txt file
/llms.txt is a plain text summary of your business and key pages for AI tools. It is a young, voluntary standard, so treat it as a helpful extra after the points above, not a replacement for them.
10. Measure AI visits
Your server logs show when crawlers such as ChatGPT-User or Perplexity-User read a page for a user, which means your page was used to answer someone. In Google Analytics, visits referred by chatgpt.com, perplexity.ai, claude.ai or gemini.google.com show who clicked through. Track both every month.
How UnlockLive can help
Our AEO and GEO services fix all ten points on WordPress, Shopify, Webflow, Next.js, Laravel and custom sites, then track AI visits so you can see the change. Start with the free AI search checker, or ask for a fixed quote.
Frequently asked questions
How do I know if ChatGPT can see my website?
Check that robots.txt allows OAI-SearchBot and ChatGPT-User, that your firewall does not block them, and that your main text is in the page HTML without JavaScript. The free UnlockLive AI search checker tests all three in about 30 seconds.
What is the difference between AEO and GEO?
AEO (answer engine optimization) makes your content easy to use as a direct answer. GEO (generative engine optimization) makes your business easy for AI assistants to find, trust and recommend. Both build on good SEO.
Should I block GPTBot?
That is a business choice. GPTBot collects content for training OpenAI models. Blocking it does not stop ChatGPT search, which uses OAI-SearchBot and ChatGPT-User. Many businesses block training crawlers and allow search crawlers.
Does structured data help with AI search?
Yes, when crawlers can see it. Put JSON-LD in the HTML, not injected with JavaScript, and describe your organization, services, FAQs and articles.
Is llms.txt required?
No. It is a young, voluntary standard. It can help AI tools describe your business, but crawler access, readable content and clear answers matter much more.
How we can help
- AEO & GEO ServicesAnswer engine and generative engine optimization: we make your website easy for ChatGPT, Claude, Perplexity, Gemini and Copilot to read, understand and recommend.
- Custom RAG & Enterprise Search DevelopmentProduction retrieval-augmented generation systems on your knowledge base. Hybrid search, reranking, citations, evals, and on-prem deployment.
- Next.js DevelopmentProduction Next.js 14 / 15 applications with App Router, Server Components, edge deployment, and TypeScript-first delivery on Vercel or AWS.
Talk to an engineer about your project
Tell us what you are building. We reply within one business day with a candid view on scope, approach and effort.
Book a free strategy callWritten by the UnlockLive IT engineering team. UnlockLive IT Limited works with clients through its Los Angeles headquarters and Toronto office and delivers engineering from its Dhaka delivery centre. About us
