Fluid Topics recognizes as crawlers only those bots whose user-agent is derived from the following list:
-
AdsBot-Google -
AhrefsBot -
Applebot -
BecomeBot -
bingbot -
BingPreview -
bot -
crawler -
facebookexternalhit -
Feedfetcher -
Go-http-client -
Googlebot -
Googlebot-Mobile -
Google Web Preview -
Java -
Mediapartners-Google -
Nutch -
Phantom -
python-requests -
Seznam -
Slurp -
spider -
Teoma -
ThumbSniper -
Twiceler -
Web Preview -
WordPress.com mShots -
YandexBot -
YandexMarket
Fluid topics looks for these keywords in the user-agent string of a crawler request.
Keyword recognition is not case-sensitive.
Only bots whose user-agent matches one of the keywords receive pure-HTML content from Fluid Topics.
Example of how Fluid Topics processes a Swiftbot request
If a request comes with its user-agent set to Swiftbot, Fluid Topics will detect it as a bot since it contains the keyword bot.
For in-house search engines (any search engine not meant for the web), Fluid Topics provides the List maps web service to index the content of a Fluid Topics tenant.