Robots.txt Generator
Create and download a valid robots.txt file for your website. Configure user agents, allow and disallow rules, sitemaps, and more.
Use paths relative to your site root. Example: /images/, /private/ or /folder/file.html
Some crawlers respect crawl-delay. Use only if necessary.
Post a Comment