What Is Robots.txt?
robots.txt is a plain text file at the root of your website, such as favze.com/robots.txt, that gives instructions to automated crawlers. It tells search engine bots like Googlebot and AI crawlers like GPTBot which sections of your site they may visit.
The file uses simple rules: a "User-agent" line names the crawler, and "Allow" or "Disallow" lines list paths. It often also points to your XML sitemap so crawlers find every important page quickly.
Why Robots.txt Matters for Northern Indiana & Michiana Businesses
A single wrong line in robots.txt can hide an entire website from Google. Website rebuilds sometimes launch with a "Disallow: /" rule left over from development, and a Mishawaka business can lose its search traffic without realizing why.
robots.txt also shapes AI visibility. Businesses that want to appear in ChatGPT, Perplexity, and Google AI answers keep the doors open for those crawlers, while still blocking areas such as admin pages, carts, and internal search results.
How Robots.txt Works
- Crawlers check firstWell-behaved bots read robots.txt before crawling your site.
- Rules by crawlerYou can give different instructions to different bots or one rule set for all.
- Paths allowed or blockedDisallow lines keep crawlers out of specific folders or pages.
- Sitemap referenceA sitemap line points crawlers to your list of important pages.
- Crawling, not indexingBlocking a page from crawling does not guarantee it stays out of results; a noindex tag handles that.
A Local Example
A Goshen manufacturer relaunches its website, and its leads drop sharply over the next month. A review finds the developer left "Disallow: /" in robots.txt from the staging site. After the rule is removed and the sitemap is resubmitted in Google Search Console, Google recrawls the site and its rankings recover over the following weeks.
Getting Started With Robots.txt
Visit yourdomain.com/robots.txt in a browser and read it. Confirm there is no rule blocking your whole site, check that your sitemap is listed, and note any blocked folders so you know why each one is there.
Then use the robots.txt report in Google Search Console to confirm Google can fetch the file without errors. Decide which AI crawlers you welcome; most local businesses benefit from allowing them so AI tools can recommend their services. Keep a copy of the file before any change and recheck it after every website update or migration.
Quick Wins to Improve Your robots.txt
- Confirm your site is not blocked with "Disallow: /"
- Add your XML sitemap location to the file
- Block admin, cart, and internal search pages
- Decide which AI crawlers you want to allow
- Recheck the file after every redesign or migration
What to Watch For
- Blocking resources. Blocking CSS or JavaScript can stop Google from rendering pages correctly.
- Noindex confusion. To remove a page from results, use a noindex tag and let crawlers see it.
- Public file. Anyone can read robots.txt, so it is not a security tool.
- Case and paths. Rules are case sensitive and must match your URLs exactly.
Robots.txt FAQ
Does every website need robots.txt?
It is strongly recommended. Even a simple file that allows crawling and lists your sitemap helps crawlers.
Can robots.txt block AI crawlers?
Yes. Many AI companies publish crawler names that respect robots.txt, so you can allow or block each one.
Will robots.txt remove a page from Google?
Not reliably. Use a noindex tag or remove the page, since blocked pages can still appear if other sites link to them.
Where do I edit robots.txt?
Many platforms and SEO plugins provide an editor. Otherwise it is a file at the root of your web server.
Put It to Work
Want certainty that Google and AI tools can reach every page you want found? Riley will review your robots.txt and crawl settings.
