Loading…
Generate robots.txt files to control search engine crawlers. Review the result before reusing it in another workflow.
What is robots.txt?
A robots.txt file tells search engine crawlers which pages or files they can or can't request from your site.
# robots.txt generated by Multi-Tools Platform User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
Important
Upload robots.txt to the root directory of your website. Note that robots.txt is a directive, not a security measure. Malicious bots may ignore it.