List of supported crawlers - Fluid Topics - 4.3

Fluid Topics Configuration and Administration Guide

Reference Guides

Fluid Topics recognizes as crawlers only those bots whose user-agent is derived from the following list:

  • AdsBot-Google

  • AhrefsBot

  • Applebot

  • BecomeBot

  • bingbot

  • BingPreview

  • bot

  • crawler

  • facebookexternalhit

  • Feedfetcher

  • Go-http-client

  • Googlebot

  • Googlebot-Mobile

  • Google Web Preview

  • Java

  • Mediapartners-Google

  • Nutch

  • Phantom

  • python-requests

  • Seznam

  • Slurp

  • spider

  • Teoma

  • ThumbSniper

  • Twiceler

  • Web Preview

  • mShots

  • YandexBot

  • YandexMarket

Fluid topics looks for these keywords in the user-agent string of a crawler request.

Keyword recognition is not case-sensitive.

Only bots whose user-agent matches one of the keywords receive pure-HTML content from Fluid Topics.

Example of how Fluid Topics processes a Swiftbot request

If a request comes with its user-agent set to Swiftbot, Fluid Topics will detect it as a bot since it contains the keyword bot.

For in-house search engines (any search engine not meant for the web), Fluid Topics provides the List maps web service to index the content of a Fluid Topics tenant.