If an administrator enables the Crawl feature for a tenant, the robots.txt file gives crawlers instructions about how to index the following parts of the portal:
-
The Homepage
-
The Reader page
-
The Viewer page
The robots.txt file also provides the path to a sitemap which allows bots to retrieve URLs to the Reader page and the Viewer page.