What to do with crawl data
On this page
Crawl data in Tracking Insights shows the problems that stop AI crawlers from reading your pages, such as blocked requests, broken links and pages bots never find. This page explains what to do about each problem you find in the Track My Visibility application.
Most fixes are small, such as a firewall rule, a redirect or a robots.txt line. Small fixes can make a large difference, because AI models can only cite pages their crawlers can read.
How to fix blocked AI crawlers#
A large number of 403 responses means something is blocking the bots. The cause is usually a firewall, a CDN bot setting or a security plugin.
Step 1. Check your firewall and CDN#
Look for rules that block or challenge bots, such as bot protection modes or rate limits.
Step 2. Allow the bots you want#
Allow the AI crawlers you want to read your site, starting with search and user-query bots.
How to get key pages crawled#
If a key page has few or no visits, bots may not be finding it:
- Link to the page from your menu or other popular pages.
- Add the page to your XML sitemap.
- Make sure
robots.txtdoesn't block the page. Check this in Technical fixes.
How to fix missing and redirected pages#
A large number of 404 responses usually means bots are following old links:
- Update your sitemap.
- Redirect old URLs to the closest current page.
- Fix redirect chains, where one URL redirects to another that redirects again.
How to handle a sudden drop#
If a bot stops visiting, check what changed around that date. Look for a new robots.txt rule, a hosting or CDN change, or a new security setting.
How to handle training bots#
You don't have to change anything for training bots. If you don't want your content used to train AI models, you can block training bots in robots.txt.
How to check the result#
After you make a change, watch the timeline and status codes over the next few days. Successful requests should go up and errors should go down. Then check whether the fixed pages start appearing in Sources.
Frequently asked questions#
How do I unblock AI crawlers on my website?
Check your firewall, CDN bot settings and security plugins for rules that block or challenge bots, and allow the AI crawlers you want. Then check robots.txt in Technical fixes to make sure those bots aren't disallowed there.
Why are AI crawlers getting 403 errors on my site?
AI crawlers usually get 403 errors because a firewall, CDN bot protection or security plugin treats them as unwanted traffic. Allow the specific bots you want in those tools, then watch the status codes in Tracking Insights.
How do I get AI crawlers to visit my important pages?
Link to your important pages from your menu and popular pages, add them to your XML sitemap and make sure robots.txt doesn't block them. Pages that are easy to find get crawled more often.
Should I block AI training bots?
Blocking training bots is optional. It limits what future AI models learn from your content, but it doesn't stop AI search tools from citing your pages, as long as search and user-query bots stay allowed.
How long does it take to see results after fixing crawl issues?
Crawler activity usually changes within a few days of a fix, which you can see in the timeline and status codes. Changes in AI answers take longer, because AI models need to read and cite the pages again.
Related articles
Last updated 3 Nov 2026
Can't find what you need?
Email support@trackmyvisibility.com with your website, what you were trying to do and what you saw. A screenshot helps.