Robots.txt: Control Search Engine Crawling
The robots.txt file provides instructions to automated crawlers about which URL paths they may request on your website. It is one of the first technical files that should be checked when investigating crawlability.
A correctly configured robots.txt file can help prevent unnecessary crawling of certain URL patterns. However, it should be used carefully because blocking a URL in robots.txt does not mean that the URL can never appear in search results.
Important Distinction
Where Is robots.txt Located?
The robots.txt file is normally located at the root of a website.
https://example.com/robots.txtFor example, if your website is example.com, the crawler instructions should be available from the root domain rather than inside a subdirectory.
Basic robots.txt Syntax
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlThe User-agent directive identifies the crawler to which the rules apply. The Allow directive specifies paths that may be crawled, while the Sitemap directive can provide the location of your XML sitemap.
Common robots.txt Directives
| Directive | Purpose |
|---|---|
| User-agent | Specifies which crawler a rule applies to. |
| Disallow | Requests that specified URL paths should not be crawled. |
| Allow | Allows access to specified paths when applicable. |
| Sitemap | Provides the URL of an XML sitemap. |
Example: Blocking an Administrative Directory
User-agent: *
Disallow: /admin/This requests that crawlers do not crawl URLs under the /admin/ path. The exact rules required by a website depend on its architecture and which resources should be accessible to crawlers.
Common robots.txt Mistakes
Blocking Important Content
Accidentally disallowing important sections can prevent crawlers from accessing URLs you want them to discover.
Blocking the Entire Website
A broad Disallow rule can unintentionally restrict crawling across the entire site.
Using robots.txt for Noindex
Robots.txt should not be treated as a replacement for page-level indexing controls.
Incorrect Sitemap URL
A sitemap directive pointing to the wrong location reduces its usefulness.
Testing Only the Homepage
A robots.txt file can look harmless while still blocking important directories or URL patterns.
Ignoring URL Patterns
Parameter and path patterns should be reviewed carefully before adding broad crawl restrictions.
robots.txt Audit Checklist
SEOHACK Tip
Related Technical SEO Guides
Continue learning with these related Technical SEO resources.
Technical SEO Crawling
Improve crawlability and help search engines discover your pages.
Read Guide →Site Architecture
Build a logical website structure that supports SEO and users.
Read Guide →SEO URL Structure
Create clean, logical and search-friendly URLs.
Read Guide →XML Sitemaps
Help search engines discover and understand your important URLs.
Read Guide →Ready to Improve Your
Google & AI Rankings?
Analyze your website in under 60 seconds and receive a professional SEO report with Technical SEO, AI SEO, Performance, GEO, AEO and actionable recommendations.
SEOHACK Editorial Team
The SEOHACK Editorial Team creates practical, research-driven content focused on Technical SEO, AI SEO, Generative Engine Optimization (GEO), Answer Engine Optimization (AEO), Local SEO, Performance Optimization and website growth.