Fluid Topics recognizes the following user-agents values as crawlers:
- Any user-agent value containing the word
bot AdsBot-GoogleAhrefsBotApplebotBecomeBotbingbotBingPreviewbotcrawlerfacebookexternalhitFeedfetcherGo-http-clientGooglebotGooglebot-MobileGoogle Web PreviewJavaMediapartners-GoogleNutchPhantompython-requestsSeznamSlurpspiderTeomaThumbSniperTwicelerWeb PreviewWordPress.com mShotsYandexBotYandexMarket
Only bots whose user-agent matches one of the keywords receive pure-HTML content from Fluid Topics.
- User-agent matching is case-insensitive.
- For in-house search engines (any search engine not meant for the web), Fluid Topics provides the List maps web service to index the content of a Fluid Topics tenant.