If an administrator enables the Crawl feature for a tenant, the robots.txt file gives crawlers instructions about how to index the following parts of the portal:
- The Homepage
- The Reader page
- The Viewer page
The robots.txt file also provides the path to a sitemap which allows bots to retrieve URLs to the Reader page and the Viewer page.