Using Lexicon

Lexicon enables you to train entities for Natural Language Processing (index entities) and use NER with index entities to validate user input with a list of values predefined for your specific situation.

This section describes how to use the Lexicon in DRUID versions 5.15 and higher.

Prerequisite

You must have at least one entity defined in the system used by the bot. The entity you want to index should have a flat data type field (a field of type other than Entity or Entity List).

To use NER with index entities, follow these steps:

Step 1. Link your entity with an external data source

Connect to an external data source. Create a connector action that brings the values from an external data source and stores them in your entity.

When creating the connector action, make sure to select your entity as both the Request and Response Entity and select the Is Response Collection option.

As you are using the integration tasks only to read data from the external data source, do not provide any input arguments on the request. In the response, map the field from the data source to the DRUID entity field. You will index your entity by the field you map here.

Note: If you are using NER on multi-language bots and you expect language specific values from the external database, make sure that on the Response you map the language specific values (returned from DB fields) to DRUID language-specific field clones.

Example:

For more information on how to connect to external data sources, see Integrations.

Step 2: Index the entity

Select the bot and from the NLU menu, click Configurations. The NLU Configurations page appears by default on the General tab.

Click the Entities tab. The Entities page will load all entities used by the respective bot (any entity which is related to another bot-related element, like flows, connector actions, etc).

Add additional entities on the bot

In case an additional entity relation needs to be added (an entity which is not yet used in any other bot-related element), the Add Bot Entity Relation button can be used to add a new empty line in the table.

Index an entity

To index an entity for NLP, click on the edit icon displayed inline on the right. A pop-up appears by default on the Parameters tab.

You can set specific NER parameters for the selected entity. For more information, see Entity NER-specific Parameters.

In the pop-up, click on the Lexicon tab and tap on Uses Lexicon.

Click on the Data Sources section header.

From the Training sources drop-down, select the connector action that will pull the selected field values from your external data source (i.e., database, web service, etc.).

From the Matching Fields drop-down, select the field you used to map the data from your external data to DRUID entity field.

Note: If you're using an entity on multi-language bots, make sure to select the main field that has language specific field clones (E.g., index the entity [[Account]] by the field Name, where the field [[Account]].Name has the option Holds multilanguage records option enabled.

Now you have to train NER for the entity, that is creating an entity index.

When you train the entity, DRUID calls the connector action which brings from the data source the list of values which needs to be indexed and saves them in a special DRUID index associated to your entity.

Manually train the entity

To manually create an entity index, in the Data Sources section, click the Train button.

After the training completes, click the Save button.

An entity index is created by the system for the selected entity.

Hint: If the external data source updates periodically, we recommend you to schedule the entity index training.

Schedule the entity index training

To schedule the training of the entity index, in the Data Sources section, tap on Is scheduler enabled.

Set Start date and End date or tick No end date if you do not want to set an end date.

Select the Frequency (to run on specific day(s) of the week) or the Daily frequency: schedule the job to occur once at a specific hour (in the format HH:MM:SS), or the job runs every specified minutes / hours within a given time frame (hours).

Click the Save button at the bottom of the window to save the scheduled job.

The bot-entity relation is created in the system and the entity is trained for NLP.

Step 3. Use indexed values on flows

Note: To use the indexed values on flows in DRUID version prior 5.0, you need to include the entity in an umbrella entity, an entity which has a field of type Entity referencing the indexed entity. On flows, in Input mapping and in flow Training Phrases, use the field of your umbrella entity which references the indexed entity.

Once you added all the needed entities to your bot, select solution and on the main menu click Flows. Select a flow and go to the Flow Editor page, click the Training Phrases section and add a complete training phrase (including sample value for the indexed entity to be extracted).

In the training phrase, select the word / sequence of words that is sample value.

Click on the selected word(s). The Entity Extraction pop-up appears.

Click in the Target Entity / Entity field and select the entity that will store the extracted value.

Click the Save button. The pop-up closes and the sample value is highlighted in green and marked with the label Entity.

You can also use the indexed entity on flow steps with input mapping (when you want to do input mapping with NER) and for intent recognition.

Hint: If your entity field holds multilanguage records, specify the main entity field (that is, the multilingual entity field) in your training phrases and the Flow Engine will extract the language specific value to the appropriate field clone based on the [[ChatUser]].Language.

After the ML model is trained, you can test a phrase with values for the intended entities (either in Bot Designer / Details or in the Flows section). The DRUID ML engine will identify the correct intent and extract the values from the indexed entity.

Whenever the bot reaches the flow, it extracts the user input and stores it in the entity you provided in the flow utterance (or in the language specific field clone) and validates that value against the list of indexed values. If the value is found, it is available in the conversation flow and you can further use it on the flow based on your needs.