Technical SEO

How to Configure Robots.txt

9 min read

Your robots.txt file controls which parts of your site search engine crawlers can access. A misconfigured robots.txt can block important pages from being indexed or waste crawl budget on low-value URLs. This guide teaches you how to set it up correctly, test it, and avoid common pitfalls that harm SEO.

learn.sections.stepByStep

Understand Robots.txt Basics

Robots.txt is a plain text file at your site's root (example.com/robots.txt) that uses directives to guide crawlers. The two main directives are User-agent (which crawler the rule applies to) and Disallow (which paths to block). An empty or missing robots.txt means all crawlers can access everything.

Identify What to Block

Block URLs that waste crawl budget without providing SEO value: admin pages, internal search results, login areas, cart and checkout pages, print versions, and parameter-heavy filter URLs. Never block CSS, JavaScript, or image files that search engines need to render your pages correctly.

Write Your Robots.txt Rules

Start with User-agent: * to apply rules to all crawlers. Use Disallow for paths to block and Allow to create exceptions within blocked directories. Remember that rules are case-sensitive and use path matching with wildcards (*) and end-of-URL markers ($).

Add Your Sitemap Reference

Include a Sitemap directive pointing to your XML sitemap: Sitemap: https://example.com/sitemap.xml. This helps search engines discover your sitemap even if they haven't found it through other means. You can list multiple sitemaps if your site uses sitemap index files.

Test Before Deploying

Use Google Search Console's Robots.txt Tester to verify your rules before going live. Test specific URLs to confirm important pages are accessible and blocked pages return the expected result. A single misplaced rule can accidentally block your entire site.

Monitor Crawl Activity

After deploying, monitor the Crawl Stats report in Google Search Console to verify that crawl patterns match your intentions. Check that blocked URLs aren't appearing in search results and that important pages are being crawled at appropriate frequencies.

Pro Tips

Robots.txt blocks crawling but not indexing. If a blocked page has external backlinks, Google may still index the URL (just without content). Use noindex meta tags to prevent indexing.
Use the $ end-of-string character to block specific file types: Disallow: /*.pdf$ blocks all PDFs without affecting other URLs containing '.pdf' in the path.
Keep your robots.txt simple. Complex rules with many exceptions are hard to maintain and easy to break. If you find your robots.txt growing beyond 20-30 lines, consider using noindex tags instead.

Common Mistakes to Avoid

Blocking CSS and JavaScript files

Google needs to render your pages to evaluate them properly. Blocking CSS or JS files prevents rendering, which means Google sees a broken page. Never block resources needed for page rendering.

Using robots.txt to hide sensitive content

Robots.txt is publicly accessible -- anyone can read it. Using it to hide admin panels or private directories actually advertises their existence. Use authentication and noindex for truly private content.

Accidentally blocking the entire site

A single 'Disallow: /' under 'User-agent: *' blocks every crawler from your entire site. This can happen during development or migration. Always double-check that no broad rules are accidentally active on production.

How Keyword Kick Makes It Easy

Interactive robots.txt generator with preset templates for common CMS platforms
Site audit checks that flag robots.txt issues including blocked important resources
Crawlability analysis showing which pages are blocked and whether that's intentional

learn.sections.faq

Is robots.txt required for SEO?

No, it's not required. Without a robots.txt file, search engines will crawl everything they can find. You only need one if you want to block specific sections from crawlers or if your site is large enough that you need to manage crawl budget.

Does robots.txt prevent pages from appearing in search results?

Not reliably. Robots.txt prevents crawling, but Google may still index the URL if it finds links pointing to it. The indexed result will show the URL and title but no description. Use a noindex meta tag to fully prevent search appearance.

How often do search engines check robots.txt?

Google typically caches your robots.txt for up to 24 hours. Changes may not take effect immediately. If you need Google to re-fetch it urgently, you can use the Robots.txt Tester in Search Console to request a refresh.

learn.cta.button