List of supported crawlers - Fluid Topics - 3.9

Fluid Topics Configuration and Administration Guide

Category
Reference Guides
Audience
public
Version
3.9

Fluid Topics recognizes as crawlers only those bots whose user-agent is derived from the following list:

  • AdsBot-Google
  • AhrefsBot
  • Applebot
  • BecomeBot
  • bingbot
  • BingPreview
  • bot
  • crawler
  • facebookexternalhit
  • Feedfetcher
  • Go-http-client
  • Googlebot
  • Googlebot-Mobile
  • Google Web Preview
  • Java
  • Mediapartners-Google
  • Nutch
  • Phantom
  • python-requests
  • Seznam
  • Slurp
  • spider
  • Teoma
  • ThumbSniper
  • Twiceler
  • Web Preview
  • WordPress.com mShots
  • YandexBot
  • YandexMarket

Fluid topics looks for these keywords in the user-agent string of a crawler request.

Keyword recognition is not case-sensitive.

Only bots whose user-agent matches one of the keywords receive pure-HTML content from Fluid Topics.

Example of how Fluid Topics processes a Swiftbot request

If a request comes with its user-agent set to Swiftbot, Fluid Topics will detect it as a bot since it contains the keyword bot.

For in-house search engines (any search engine not meant for the web), Fluid Topics provides the List maps web service to index the content of a Fluid Topics tenant.