Why Is Your Site Missing from AI Answers? A Crawl and Indexing Guide

Discovery means an engine learns a URL exists; crawling means it can fetch it; indexing means the content is processed into a search system. Citation is a separate decision to use a page for a particular answer. Success at one stage does not guarantee the next.

Write down the actual symptom: a fetch failure, an indexing issue reported by a URL inspection tool, or absence from a few AI answers. Each needs a different investigation. One manual query or a request carrying a bot name cannot establish the state of an entire website.

robots.txt gives crawl instructions to cooperative bots; it does not protect private information. Google documents that a blocked URL can still appear in results. Sensitive information needs authentication and actual access controls.

An index exclusion instruction such as noindex must be accessible to the crawler. Blocking the same page in robots.txt can prevent that instruction from being seen. Inspect both HTML meta tags and X-Robots-Tag headers, including any headers added by the CDN.

Fetch an article, its main image and robots.txt on the production domain. A 200 response containing a challenge or login screen is not the article. Check redirects, content type and whether the initial HTML contains the heading and main text.

CDN bot rules, geographic restrictions and rate limits can behave independently of application code. Use security reports and privacy-safe logs to investigate. Changing a User-Agent is only a diagnostic probe: it can be spoofed and does not authenticate a crawler.

Serve the main article and real links in accessible HTML. Answers hidden behind login, personal requests or required interactions may be unavailable to some content consumers. Do not make an image the only place where an important answer exists.

Sitemaps should list canonical, indexable URLs. Align canonical tags, language signals and internal links with the real destination. Sitemap submission supports discovery rather than commanding immediate indexing. Treat llms.txt as an optional guide, not a universal requirement or replacement for HTML and sitemaps.

Check each provider’s current documentation for crawler names and purposes. Search, user-triggered visits and training collection may use different agents. Make access decisions according to your publication policy rather than assuming every bot has the same role.

Review access for the public pages you want discovered while protecting administration, private APIs and customer data. Crawler names can change, so record the date and source of each decision. A supposedly permanent list of every AI bot would quickly become unreliable.

Record fetch results, index coverage, canonical URLs and the latest action for each URL. Fix a demonstrated technical blocker first and allow time for recrawling. Changing robots rules, content and URLs together makes cause and effect harder to identify.