Index Fluid Topics with Coveo - Fluid Topics

Overview

It is possible to integrate Fluid Topics with the Coveo search engine. Once the integration is set up, the Coveo search engine uses its generic REST API source to perform the following actions:

Crawl content in the Knowledge Hub.
Index the crawled content on Coveo servers.

- Each API crawl counts for one call to list maps, and one additional call per topic for each map.
- For more information about how crawling works, refer to Introduction to crawling.

Procedure

Users with the ADMIN role can follow the steps in this section to configure a Coveo instance for integration with Fluid Topics.

Prerequisites

Before configuring the Coveo instance, it is necessary to meet the following prerequisites:

It is necessary to have a Coveo license before starting this procedure.
Make sure that none of the content access rules configured for the portal prevent bots from crawling content that should be indexed.
Create an API key with the KHUB_ADMIN role.

Coveo cannot crawl Fluid Topics sandbox environments.

Create a Generic REST API source

Authorized administrators can create a generic REST API source for a Coveo instance as follows:

In the Coveo instance, go to the Content > Sources menu.
Select the Add source button.
Select the Generic REST API source type.
In the Authentication section, paste the API key (see Prerequisites) in the API key authentication field.
Download the content of the example configuration file provided as attachment in the Attachments tab of the Reader page sidebar.
Paste the content of the example configuration file in the JSON Configuration window .
Select Save (and not Save and build).

It is necessary to complete the steps in the next section to finalize the creation of the source.

For more information, refer to Coveo's documentation about adding or editing a generic REST API source.

Create Coveo fields

Coveo fields are necessary to retrieve metadata from the portal. Authorized administrators can configure Coveo fields as follows:

Go to Content > Fields.
Select Add field.
Define a Field name. This can be a title, a date, the value of an ft:originId metadata, etc.
Define a Type.
Select Add field.

Create Coveo mappings

Coveo mappings are necessary to retrieve metadata from the portal. Authorized administrators can configure Coveo mappings as follows:

Go to Content > Sources.
Select the Generic REST API source.
In the More menu, select Manage mappings.
In the Common tab, select the Add button, then select Mapping.
In the window that appears, choose Select an option and define the target field to map.
In the Rules field, define a metadata key for Fluid Topics to return as shown in the following example:
```
%[my_metadata_key] 
```
Select Add mapping.
Select Save and rebuild source.

When defining a value for the Rules field, it is necessary to format it exactly as it will appear in the JSON response for the endpoint.

Example

A Coveo source configuration file in JSON format is available in the Attachments tab of the Reader page sidebar.

The file describes the API that crawls content and defines the following information:

The order of calls to make
How to handle paginated queries
Authorization
How to interpret the JSON responses

The following lines show an example of a file that indexes all maps, topics and unstructured documents:

{
  "Services": [
    {
      "Url": "https://my.fluidtopics.net/coveo",
      "Paging": {
        "PageSize": 25,
        "OffsetStart": 1,
        "OffsetType": "page",
        "Parameters": {
          "Limit": "per_page",
          "Offset": "page"
        }
      },
      "Endpoints": [
        {
          "Path": "/api/khub/maps/search",
          "Method": "POST",
          "Headers": {
            "Authorization": "Bearer @ApiKey",
            "Ft-Calling-App": "coveo/index"
          },
          "PayloadJsonContent": "{}",
          "RefreshEndpoints": [
            {
              "Method": "POST",
              "DateFormat": "yyyy-MM-dd",
              "QueryParameters": {
                "from": "@RefreshDate"
              }
            }
          ],
          "ItemType": "Structured",
          "ItemPath": "results",
          "Uri": "%[mapUrl]",
          "ClickableUri": "%[readerUrl]",
          "ModifiedDate": "%[metadata[?(@.key=='ft:lastEdition')].values[0]]",
          "Title": "%[title]",
          "SubQueries": [
            {
              "Path": "/api/khub/maps/%[coveo_parent.raw.mapId]",
              "Method": "GET",
              "Headers": {
                "Authorization": "Bearer @ApiKey",
                "Ft-Calling-App": "coveo/index"
              },
              "Body": "%[description]",
              "Metadata": {
                "id": "%[id]",
                "language": "%[lang]",
                "lastEdition": "%[lastEdition]",
                "lastPublication": "%[lastPublication]",
                "filetype": "html"
              }
            },
            {
              "Path": "/api/khub/maps/%[coveo_parent.raw.mapId]/rights",
              "Method": "GET",
              "Headers": {
                "Authorization": "Bearer @ApiKey",
                "Ft-Calling-App": "coveo/index"
              },
              "Metadata": {
                "accessLevel": "%[accessLevel]",
                "groups": "%[groups]"
              }
            }
          ],
          "SubItems": [
            {
              "Path": "/api/khub/maps/%[coveo_parent.id]/topics",
              "Method": "GET",
              "Headers": {
                "Authorization": "Bearer @ApiKey",
                "Ft-Calling-App": "coveo/index"
              },
              "ItemType": "Topic",
              "Uri": "%[coveo_url]/api/khub/maps/%[coveo_parent.id]/topics/%[id]",
              "ClickableUri": "%[coveo_url]%[readerUrl]",
              "Title": "%[title]",
              "ModifiedDate": "%[coveo_parent.lastEdition]",
              "Metadata": {
                "id": "%[id]",
                "language": "%[coveo_parent.language]",
                "lastEdition": "%[coveo_parent.lastEdition]",
                "lastPublication": "%[coveo_parent.lastPublication]"
              },
              "SubQueries": [
                {
                  "Path": "%[coveo_parent.raw.contentApiEndpoint]",
                  "Method": "GET",
                  "Headers": {
                    "Authorization": "Bearer @ApiKey",
                    "Ft-Calling-App": "coveo/index"
                  },
                  "IsBinaryBody": true
                }
              ]
            }
          ]
        },
        {
          "Path": "/api/khub/documents/search",
          "Method": "POST",
          "Headers": {
            "Authorization": "Bearer @ApiKey",
            "Ft-Calling-App": "coveo/index"
          },
          "PayloadJsonContent": "{}",
          "RefreshEndpoints": [
            {
              "Method": "POST",
              "DateFormat": "yyyy-MM-dd",
              "QueryParameters": {
                "from": "@RefreshDate"
              }
            }
          ],
          "ItemType": "Unstructured",
          "ItemPath": "results",
          "Uri": "%[documentUrl]",
          "ClickableUri": "%[viewerUrl]",
          "ModifiedDate": "%[metadata[?(@.key=='ft:lastEdition')].values[0]]",
          "Title": "%[title]",
          "SubQueries": [
            {
              "Path": "/api/khub/documents/%[coveo_parent.raw.documentId]",
              "Method": "GET",
              "Headers": {
                "Authorization": "Bearer @ApiKey",
                "Ft-Calling-App": "coveo/index"
              },
              "Body": "%[description]",
              "Metadata": {
                "id": "%[id]",
                "language": "%[lang]",
                "lastEdition": "%[lastEdition]",
                "lastPublication": "%[lastPublication]",
                "filetype": "%[mimeType]",
                "filename": "%[filename]"
              }
            },
            {
              "Path": "/api/khub/documents/%[coveo_parent.raw.documentId]/rights",
              "Method": "GET",
              "Headers": {
                "Authorization": "Bearer @ApiKey",
                "Ft-Calling-App": "coveo/index"
              },
              "Metadata": {
                "accessLevel": "%[accessLevel]",
                "groups": "%[groups]"
              }
            },
            {
              "Path": "/api/khub/documents/%[coveo_parent.raw.documentId]/content",
              "Method": "GET",
              "Headers": {
                "Authorization": "Bearer @ApiKey",
                "Ft-Calling-App": "coveo/index"
              },
              "IsBinaryBody": true
            }
          ]
        }
      ]
    }
  ]
}

Certain fields in this file are formatted as follows:

html "foo": "‰[bar]"

This is an internal Coveo syntax that refers to an attribute of the object that Coveo's Generic Rest API connector creates.

Known limitations

Items deleted from Fluid Topics will not be deleted from Coveo. It is necessary to regularly schedule a rescan in Coveo to remove deleted items.