Loading…
Create and customize your robots.txt file. Review the result before reusing it in another workflow.
User-agent: * Disallow: /admin/
The robots.txt file tells search engine crawlers which URLs they can access on your site.
User-agent: Specifies which bot the rules apply to (* = all bots)
Disallow: Tells bots not to crawl the specified path
Allow: Overrides a Disallow directive for a specific path
Sitemap: Points bots to your XML sitemap