By defining default robots meta tags, ADMIN users can control how bots crawl certain content on tenants where crawl is enabled.
Fluid Topics supports the following meta tags:
- noindex
- nofollow
- noarchive
Use case
It can be useful to define default robots meta tags when deciding to index all content but not cache it for security reasons.
- Since Fluid Topics is a single-page application and crawlers cannot always crawl such applications properly, Fluid Topics detects crawlers based on the user-agent string. To verify the configuration, end-users must update their user agent to mimic the Googlebot or Bingbot so that Fluid Topics correctly detects them as robots and sets the expected meta tags.
- There will never be any robots meta tags for the most common user agents.
For details about robots meta tags, refer to Google's Robots meta tags documentation.
All major search engines support the robots meta tags specification.