GreenKey

Complete Example: A Request For Directions




Let's create a complete example to demonstrate how to customize Discovery to interpret transcripts. For this example, we will create an interpreter that can create directions for a user.

Step 1: Create a "custom" directory

We will need to create a directory to hold our custom "intents" and "entities" definition files. You can name this directory anything you like, but the convention is to name this directory, custom.

Inside of your custom directory, you will need a directory called entities. This will store all of our entity definition files. You will also need a file called intents.json to store your intents definitions.

Navigate to the project directory where you will be running your command to launch Discovery. Then run the following commands:

mkdir -p custom/entities
cd custom
touch intents.json

We now have our custom folder set up to start customizing Discovery.

Step 2: Create an entity

A user would expect different directions to an address depending on their means of transportation. The first bit of information we will therefore need from the user is their means of transportation. We will be looking for words like, "bicycle", "car", "train". An individual piece of information like this is labeled an "entity".

To get Discovery to understand this entity, we must create an entity definition file. Let's do that now.

cd entities
touch transportation_type.py

In this newly created file, add the following code:

transportation_type.py

TRANSPORTATION_METHOD = {
  'label': 'TRANSPORTATION_METHOD',
  'values': (
    'foot',
    'walk',
    'walking',
    'bicycle',
    'bike',
    'bus',
    'train',
    'car',
  )
}

TRANSPORTATION_PATTERNS = [[['TRANSPORTATION_METHOD']]]

ENTITY_DEFINITION = {
  'extraTokens': (TRANSPORTATION_METHOD,),  # the trailing comma is needed to make this a tuple
  'patterns': TRANSPORTATION_PATTERNS,
}

The above entity definition file tells Discovery to tokene any words that it finds that happen to match a value in the TRANSPORTATION_METHOD dictionary. Discovery will then look through the tokenized words it finds to see if any match the patterns specified in patterns. In this case, the only pattern we are looking for is of a single occurrence of a TRANSPORTATION_METHOD.

Step 3: Create an Intent Definition

Before Discovery can go and find any instances of the transportation_type entity in our audio files, we must first associate this entity with an intent. We do this in the previously created file, intents.json.

Open this file and add in the following json:

{
  "intents": [
    {
      "label": "directions",
      "entities": ["transportation_type"]
    }
  ]
}

This tells Discovery that you will be looking for an intent of type directions which contains an entity of type transportation_type. The label you use in this file to label the entity must be the exact same string that you used to name your entity file. We named our file transportation_type.py, so the entity we put in the "entities" array must be transportation_type.

Note that this file must be valid json, so don't use any trailing commas and use double quotes!

Step 4: Giving Our Custom Definitions To Discovery

This is technically all we need to start using Discovery to detect intents and entities. Let's upload this to Discovery now.

Navigate to the parent directory of your newly created custom directory. Let's create a shell script to lauch our container called run_discovery.sh

touch run_discovery.sh

Inside the file, add these contents:

run_discovery.sh

#!/bin/bash

docker run --rm -d \
  --name=discovery \
  -e GKT_API="https://scribeapi.greenkeytech.com/" \
  -e GKT_USERNAME=$GKT_USERNAME \
  -e GKT_SECRETKEY=$GKT_SECRETKEY \
  -p 1234:1234 \
  -v "$PWD/custom":/custom \
  docker.greenkeytech.com/discovery && \
  echo "Discovery instance running on http://localhost:1234"

The line -v "$PWD/custom":/custom \ is what mounts our custom definitions into the Discovery container.

Before we start the container, verify that your file structure looks like this:

your_project_directory
|   run_discovery.sh
|
└───custom
    │   intents.json
    │
    └───entities
        │   transportation_type.py

Let's start Discovery by running sh run_discovery.sh After the container finishes booting up, we are ready to test out our custom entity.

Step 5: Testing Out Our Container

Send the following request from the command line.

curl -X POST http://localhost:1234/process \
     -H "Content-type: application/json" \
     -d '{"intents":["directions"], "transcript": "I need directions to get to the bank by car"}'

You should get this response back from Discovery:

{
  "intents": [
    {
      "entities": [
        {
          "label": "transportation_type",
          "matches": [
            [
              {
                "endTimeSec": 0.00,
                "lattice_path": [
                  [9, 0]
                ],
                "probability": 1.00,
                "startTimeSec": 0.00,
                "value": "car"
              }
            ]
          ]
        }
      ],
      "label": "directions",
      "probability": 1.0
    }
  ],
  "interpreted_quote": {}
}

Step 6: Adding Entities For An Address

A street address is generally composed of a building number and a street name, and optionally a direction for the street name and street type. Let's create entities for all of these pieces. Navigate to your entities/ directory inside of your custom folder and make the following files.

cd custom/entities/
touch address_number.py direction.py street_name.py street_type.py cleaning_functions.py

Add the following code to the files you just created.

address_number.py

# creates numbers consisting of a single digit up to a number consisting of five digits
ADDRESS_NUMBER_PATTERN = [[['NUM'] for _ in range(i)] for i in range(1, 6)]

ENTITY_DEFINITION = {
  'patterns': ADDRESS_NUMBER_PATTERN,
  'spacing': {
    'NUM': '', # setting spacing to '' will format numbers without spaces between them
  }
}

direction.py

from .cleaning_functions import capitalize

DIRECTION = {
  'label': 'DIRECTION',
  'values': ('north', 'south', 'east', 'west'),
}

ADDRESS_NUMBER_PATTERN = [[['DIRECTION']]]

ENTITY_DEFINITION = {
  'patterns': ADDRESS_NUMBER_PATTERN,
  'extraTokens': (DIRECTION,),
  'extraCleaning': {
    'DIRECTION': capitalize,
  }
}

street_name.py

from .cleaning_functions import capitalize

STREET_NAME = {
  'label': 'STREET_NAME',
  'values': (
    'abbott',
    'baker',
    'cabrini',
    'dakin',
    'eastgate',
    'fairbanks',
    'garfield',
    'monroe',
  )
}

STREET_NAME_PATTERN = [[['STREET_NAME']]]

ENTITY_DEFINITION = {
  'patterns': STREET_NAME_PATTERN,
  'extraTokens': (STREET_NAME,),
  'extraCleaning':
    {
      'STREET_NAME': capitalize,
    },
}

street_type.py

from .cleaning_functions import capitalize

STREET_TYPE = {
  'label': 'STREET_TYPE',
  'values': (
    'alley',
    'avenue',
    'street',
    'boulevard',
    'way',
    'ave',
    'place',
    'highway',
    'lane',
    'drive',
    'route',
    'square',
    'road',
    'expressway',
  )
}

STREET_TYPE_PATTERN = [[['STREET_TYPE']]]

ENTITY_DEFINITION = {
  'patterns': STREET_TYPE_PATTERN,
  'extraTokens': (STREET_TYPE,),
  'extraCleaning': {
    'STREET_TYPE': capitalize,
  }
}

cleaning_functions.py

def capitalize(wordList, spacer):
  """ Capitalize all words of a token type and space them appropriately """
  return spacer.join(word.capitalize() for word in wordList) if isinstance(wordList, list) else wordList.capitalize()

You will notice that we made a file called cleaning_functions that contains a function to capitalize words. This will help us format the output that we get from Discovery. We simply add an extraCleaning property to our entity definition specifying which tokens should receive the cleaning function. Any matching words will receive the formatting when they are returned.

Step 7: Adding Compound Entities To Our Intent Definition

We would like to be able to combine the previously created entities together to create an address. We do this with compound entities. Open your intents.json file and replace the existing contents with the following contents:

intents.json

{
  "intents": [
    {
      "label": "directions",
      "entities": ["transportation_type"],
      "composite_entities": [
        {
          "label": "complete_address",
          "spacing_threshold": 1,
          "component_entity_patterns": [
            ["address_number", "street_name"],
            ["address_number", "street_name", "street_type"],
            ["address_number", "street_name", "street_type"],
            ["address_number", "street_name", "street_name", "street_name"],
            ["address_number", "direction", "street_name"],
            ["address_number", "direction", "street_name", "street_type"],
            ["address_number", "direction", "street_name", "street_type"],
            ["address_number", "direction", "street_name", "street_name", "street_name"]
          ]
        }
      ]
    }
  ]
}

This tells Discovery to look for a specified combination of components to make up a new compound entity called complete_address. Let's try this out.

Step 8: Sending A Request With An Address

If you still have you container running from step 4, stop it, and start it again now that we have changed our definitions.

docker stop discovery
sh run_discovery.sh

After the container boots up, send the following request from the command line.

curl -X POST http://localhost:1234/process \
     -H "Content-type: application/json" \
     -d '{"intents":["directions"], "transcript": "I need directions to fifty five um west monroe by bus"}'

You should get this response back from Discovery:

{
  "intents": [
    {
      "entities": [
        {
          "label": "complete_address",
          "matches": [
            [
              {
                "endTimeSec": "0.00",
                "lattice_path": [[4, 0], [5, 0], [7, 0], [8, 0]],
                "probability": "1.00",
                "startTimeSec": "0.00",
                "value": "55 West Monroe"
              },
              {
                "endTimeSec": "0.00",
                "lattice_path": [[4, 0], [7, 0], [8, 0]],
                "probability": "1.00",
                "startTimeSec": "0.00",
                "value": "50 West Monroe"
              },
              {
                "endTimeSec": "0.00",
                "lattice_path": [[5, 0], [7, 0], [8, 0]],
                "probability": "1.00",
                "startTimeSec": "0.00",
                "value": "5 West Monroe"
              },
              {
                "endTimeSec": "0.00",
                "lattice_path": [[4, 0], [5, 0], [8, 0]],
                "probability": "1.00",
                "startTimeSec": "0.00",
                "value": "55 Monroe"
              },
              {
                "endTimeSec": "0.00",
                "lattice_path": [[4, 0], [8, 0]],
                "probability": "1.00",
                "startTimeSec": "0.00",
                "value": "50 Monroe"
              },
              {
                "endTimeSec": "0.00",
                "lattice_path": [[5, 0], [8, 0]],
                "probability": "1.00",
                "startTimeSec": "0.00",
                "value": "5 Monroe"
              }
            ]
          ]
        },
        {
          "label": "transportation_type",
          "matches": [
            [
              {
                "endTimeSec": "0.00",
                "lattice_path": [[10, 0]],
                "probability": "1.00",
                "startTimeSec": "0.00",
                "value": "bus"
              }
            ]
          ]
        },
        {
          "label": "direction",
          "matches": [
            [
              {
                "endTimeSec": "0.00",
                "lattice_path": [[7, 0]],
                "probability": "1.00",
                "startTimeSec": "0.00",
                "value": "West"
              }
            ]
          ]
        },
        {
          "label": "street_name",
          "matches": [
            [
              {
                "endTimeSec": "0.00",
                "lattice_path": [[8, 0]],
                "probability": "1.00",
                "startTimeSec": "0.00",
                "value": "Monroe"
              }
            ]
          ]
        },
        {
          "label": "address_number",
          "matches": [
            [
              {
                "endTimeSec": "0.00",
                "lattice_path": [[4, 0], [5, 0]],
                "probability": "1.00",
                "startTimeSec": "0.00",
                "value": "55"
              },
              {
                "endTimeSec": "0.00",
                "lattice_path": [[4, 0]],
                "probability": "1.00",
                "startTimeSec": "0.00",
                "value": "50"
              },
              {
                "endTimeSec": "0.00",
                "lattice_path": [[5, 0]],
                "probability": "1.00",
                "startTimeSec": "0.00",
                "value": "5"
              }
            ]
          ]
        }
      ],
      "label": "directions",
      "probability":
        1.0
    }
  ],
  "interpreted_quote": {}
}

The top result for complete_address is "55 West Monroe". Success!

Step 9: Customizing Your Return JSON From Discovery

Depending on your needs, the json response we just received might be more information than you need from Discovery. We can greatly simplify the return json by creating a schemas.json file to create our own return format. Create a custom schema definition by doing the following.

cd custom
touch schemas.json

Open your newly created schemas.json file and add the following content:

schemas.json

{
  "schemas": [
    {
      "label": "directions",
      "return_json": {
        "destination_address": "{complete_address}",
        "transportation_method": "{transportation_type}"
      }
    }
  ]
}

Now, modify your run script to contain USE_CUSTOM_JSON_SCHEMA="True", SUPPRESS_DEFAULT_OUTPUT="True", and SCHEMA_ENTITY_REPLACEMENT_POLICY="Best". It should look like the following:

run_discovery.sh

#!/bin/bash

docker run --rm -d \
  --name=discovery \
  -e GKT_API="https://scribeapi.greenkeytech.com/" \
  -e GKT_USERNAME=$GKT_USERNAME \
  -e GKT_SECRETKEY=$GKT_SECRETKEY \
  -e USE_CUSTOM_JSON_SCHEMA="True" \
  -e SUPPRESS_DEFAULT_OUTPUT="True" \
  -e SCHEMA_ENTITY_REPLACEMENT_POLICY="Best" \
  -p 1234:1234 \
  -v "$PWD/custom":/custom \
  docker.greenkeytech.com/discovery && \
  echo "Discovery instance running on http://localhost:1234"

Restart your container to try out the custom json schema definition.

docker stop discovery
sh run_discovery.sh

After the container boots up, resend the following request from the command line to see how the output has changed.

curl -X POST http://localhost:1234/process \
     -H "Content-type: application/json" \
     -d '{"intents":["directions"], "transcript": "I need directions to fifty five um west monroe by bus"}'

Your return json should look like the following:

{
  "destination_address": "55 West Monroe",
  "transportation_method": "bus"
}