Robots.txt Generator

Generate robots.txt files to control search engine crawling

Basic Settings

Website URL

Used for sitemap reference

Default Policy

Allow All Block All Custom

User Agents & Rules

User Agent

Rules

Crawl Delay (seconds)

Additional Settings

Sitemap URL

Optional: Link to your XML sitemap

Host Directive

Optional: Specify preferred domain (mainly for Yandex)

Include explanatory comments

Advertisement

Common Patterns

Path Examples

/ - Root directory
/admin/ - Admin folder
*.pdf - All PDF files
/search? - Search pages
/private/ - Private content

Important Tips

Place in website root
Use lowercase filename
Test with Google Search Console
Be careful with wildcards
Update regularly