When an administrator enables crawl for a portal, Fluid Topics adds the following API path to the root of the portal's URL:
/sitemap.xml
The following example shows a portal's URL with the API path added:
my_fluid_topics_portal.net/sitemap.xml
Fluid Topics adds the following additional paths for different categories of content:
-
/home.xmlfor the portal's homepage. -
/pages.xml -
/unstructured/<page>.xml. -
/structured/<page>.xml.
- Fluid Topics paginates structured and unstructured documents. Each page contains 10,000 documents.
- If crawl is disabled, Fluid Topics does not generate a sitemap and replies with an HTTP 404 error if a bot or user tries to access one.
Even if an administrator restricts access to authenticated users, crawlers are still able to access the tenant's sitemap.xml file with an HTML view of public content. Only content protected by content access rules is absent from the sitemap.xml file.
If a pretty URL exists for a given document, Fluid Topics asks bots to give it preference over the canonical one when adding a URL to the sitemap.