By defining default robots meta tags, ADMIN users can control how bots crawl content on tenants where crawl is active.
Fluid Topics supports the following meta tags:
noindexnofollownoarchive
Use case
It can be useful to define default robots meta tags when deciding to index all content but not cache it for security reasons.
- Since Fluid Topics is a single-page application, and crawlers cannot always crawl such applications correctly, Fluid Topics detects crawlers based on the user-agent string. To check the configuration, it is possible to update the browser's user agent to view pages as Googlebot or Bingbot.
- There are never any robots meta tags for the most common user agents.
- Meta tags do not apply to documents that require authentication.
For details about robots meta tags, see Google's Robots meta tags documentation.
All major search engines support the robots meta tags specification.