Fluid Topics recognizes as crawlers only those bots whose user-agent is derived from the following list:
- AdsBot-Google
- AhrefsBot
- Applebot
- BecomeBot
- bingbot
- BingPreview
- bot
- crawler
- facebookexternalhit
- Feedfetcher
- Go-http-client
- Googlebot
- Googlebot-Mobile
- Google Web Preview
- Java
- Mediapartners-Google
- Nutch
- Phantom
- python-requests
- Seznam
- Slurp
- spider
- Teoma
- ThumbSniper
- Twiceler
- Web Preview
- WordPress.com mShots
- YandexBot
- YandexMarket
Fluid topics looks for these keywords in the user-agent string of a crawler request.
Keyword recognition is not case-sensitive.
Only bots whose user-agent matches one of the keywords receive pure-HTML content from Fluid Topics.
Example of how Fluid Topics processes a Swiftbot request
If a request comes with its user-agent set to Swiftbot, Fluid Topics will detect it as a bot since it contains the keyword bot.
For in-house search engines (any search engine not meant for the web), Fluid Topics provides the List maps web service to index the content of a Fluid Topics tenant.