Broken Alexa Interaction Model Slot Extraction When Using Dialog Management

I’m probably not the first one to notice this but it’s worth documenting for posterity. It’s also one of many minor inconsistencies in Alexa behavior that I hope will be fixed in the near future.

On one of my projects, we are integrating our Microsoft Bot Framework bot with an Alexa Interaction Model that includes Dialog Management. It’s a very interesting effort and allows us to get us intimately familiar with how the Alexa Interaction Model works, versus Microsoft’s LUIS, a system that we have much more experience with.

Let me set the stage. We want to be able to train an Intent that requires multiple custom slot types. For the sake of the example, let’s say we need three slots: a number, a custom slot comprised of two values (buy/sell) and a custom free text slot indicating the name of a product the user is buying. This last slot is the problem area I’d like to focus on. This slot is fairly free form and the set of values may change any time, so we cannot simply populate all the possible values into the slot type definition. We have no problems with Alexa detecting an unseen slot value based on what we can assume is a common machine learning-based model. However, once we enable Dialog Management on the intent, the slot is no longer recognized!

Jumping In

Let’s say we want to create an intent called RegisterNeedIntent. This lets the user inform our skill when she wants to sell or purchase an amount of some product. For example, the user may want to buy thirty two pounds of butter or sell thirty two tulips. We will create three slot types: a custom slot type for the buy/sell distinction, the number slot to capture the amount of the product and a custom slot type representing the product in question.

The interaction model looks something like this:


{
  "interactionModel": {
      "languageModel": {
          "invocationName": "product catalog",
          "intents": [
              {
                  "name": "AMAZON.CancelIntent",
                  "samples": []
              },
              {
                  "name": "AMAZON.HelpIntent",
                  "samples": []
              },
              {
                  "name": "AMAZON.StopIntent",
                  "samples": []
              },
              {
                  "name": "RegisterNeedIntent",
                  "slots": [
                      {
                          "name": "Amount",
                          "type": "AMAZON.NUMBER"
                      },
                      {
                          "name": "Product",
                          "type": "Product"
                      },
                      {
                          "name": "Action",
                          "type": "Action"
                      }
                  ],
                  "samples": [
                      "{Action} {Product}",
                      "{Action} {Amount} pounds of {Product}",
                      "{Action} {Amount} {Product}",
                      "{Action} {Amount} packages of {Product}"
                  ]
              }
          ],
          "types": [
              {
                  "name": "Action",
                  "values": [
                      {
                          "name": {
                              "value": "sell"
                          }
                      },
                      {
                          "name": {
                              "value": "buy"
                          }
                      }
                  ]
              },
              {
                  "name": "Product",
                  "values": [
                      {
                          "name": {
                              "value": "flying squirrel"
                          }
                      },
                      {
                          "name": {
                              "value": "hammer"
                          }
                      }
                  ]
              }
          ]
      }
  }
}

Simple enough. We considered using the AMAZON.SearchQuery slot type but it will not work because it cannot be combined with other slot types. When we build this model, the user can enter an utterance like buy twenty boxes of notebooks. Even with this small of amount of sample utterances, Alexa recognizes that the word notebooks is a Product slot. See the intent object passed into the skill below.


{
  "name": "RegisterNeedIntent",
  "confirmationStatus": "NONE",
  "slots": {
    "Product": {
      "name": "Product",
      "value": "notebooks",
      "resolutions": {
        "resolutionsPerAuthority": [
          {
            "authority": "amzn1.er-authority.echo-sdk.amzn1.ask.skill.a20631d9-6a23-413b-8bdc-68ea3c702a10.Product",
            "status": {
              "code": "ER_SUCCESS_NO_MATCH"
            }
          }
        ]
      },
      "confirmationStatus": "NONE"
    },
    "Action": {
      "name": "Action",
      "value": "buy",
      "resolutions": {
        "resolutionsPerAuthority": [
          {
            "authority": "amzn1.er-authority.echo-sdk.amzn1.ask.skill.a20631d9-6a23-413b-8bdc-68ea3c702a10.Action",
            "status": {
              "code": "ER_SUCCESS_MATCH"
            },
            "values": [
              {
                "value": {
                  "name": "buy",
                  "id": "0461ebd2b773878eac9f78a891912d65"
                }
              }
            ]
          }
        ]
      },
      "confirmationStatus": "NONE"
    },
    "Amount": {
      "name": "Amount",
      "value": "20",
      "confirmationStatus": "NONE"
    }
  }
}

The only caveat is that although the Action slot type has a resolution with status ER_SUCCESS_MATCH, the resolution for Product is ER_SUCCESS_NO_MATCH. Fair enough! notebooks does not exist in the Interaction Model. This is actually great. At this point, our skill code receives the raw user Product input, validates it against some known database and we’re in business. So far, this works like LUIS. LUIS’ slots equivalent, entities, do draw a distinction between dictionary look ups of known values, know as List Entities, and machine learned Simple Entities. In essence, Amazon slots act as a combination of those two entity types.

Here is Where It Breaks

We know we want to include more slot types down the road. We also know that we want all those slots to be collected by the entity. Perfect use case for Dialog Management. The following Interaction Model is a result of the addition of dialogs and prompts.


{
  "interactionModel": {
      "languageModel": {
          "invocationName": "product catalog",
          "intents": [
              {
                  "name": "AMAZON.CancelIntent",
                  "samples": []
              },
              {
                  "name": "AMAZON.HelpIntent",
                  "samples": []
              },
              {
                  "name": "AMAZON.StopIntent",
                  "samples": []
              },
              {
                  "name": "RegisterNeedIntent",
                  "slots": [
                      {
                          "name": "Amount",
                          "type": "AMAZON.NUMBER",
                          "samples": [
                              "{Amount}",
                              "I want {Amount} units"
                          ]
                      },
                      {
                          "name": "Product",
                          "type": "Product",
                          "samples": [
                              "{Product}"
                          ]
                      },
                      {
                          "name": "Action",
                          "type": "Action",
                          "samples": [
                              "I want to {Action}",
                              "{Action}"
                          ]
                      }
                  ],
                  "samples": [
                      "{Action} {Product}",
                      "{Action} {Amount} pounds of {Product}",
                      "{Action} {Amount} {Product}",
                      "{Action} {Amount} packages of {Product}"
                  ]
              }
          ],
          "types": [
              {
                  "name": "Action",
                  "values": [
                      {
                          "name": {
                              "value": "sell"
                          }
                      },
                      {
                          "name": {
                              "value": "buy"
                          }
                      }
                  ]
              },
              {
                  "name": "Product",
                  "values": [
                      {
                          "name": {
                              "value": "flying squirrel"
                          }
                      },
                      {
                          "name": {
                              "value": "hammer"
                          }
                      }
                  ]
              }
          ]
      },
      "dialog": {
          "intents": [
              {
                  "name": "RegisterNeedIntent",
                  "confirmationRequired": true,
                  "prompts": {
                      "confirmation": "Confirm.Intent.1103891303624"
                  },
                  "slots": [
                      {
                          "name": "Amount",
                          "type": "AMAZON.NUMBER",
                          "confirmationRequired": false,
                          "elicitationRequired": true,
                          "prompts": {
                              "elicitation": "Elicit.Slot.1103891303624.102254088595"
                          }
                      },
                      {
                          "name": "Product",
                          "type": "Product",
                          "confirmationRequired": false,
                          "elicitationRequired": true,
                          "prompts": {
                              "elicitation": "Elicit.Slot.1103891303624.804409414770"
                          }
                      },
                      {
                          "name": "Action",
                          "type": "Action",
                          "confirmationRequired": false,
                          "elicitationRequired": true,
                          "prompts": {
                              "elicitation": "Elicit.Slot.1103891303624.279514542098"
                          }
                      }
                  ]
              }
          ]
      },
      "prompts": [
          {
              "id": "Elicit.Slot.1103891303624.102254088595",
              "variations": [
                  {
                      "type": "PlainText",
                      "value": "How many units do you want to buy?"
                  }
              ]
          },
          {
              "id": "Elicit.Slot.1103891303624.804409414770",
              "variations": [
                  {
                      "type": "PlainText",
                      "value": "Which product would you like to {Action} ?"
                  }
              ]
          },
          {
              "id": "Elicit.Slot.1103891303624.279514542098",
              "variations": [
                  {
                      "type": "PlainText",
                      "value": "Do you want to buy or sell?"
                  }
              ]
          },
          {
              "id": "Confirm.Intent.1103891303624",
              "variations": [
                  {
                      "type": "PlainText",
                      "value": "Do you want to {Action} {Amount} units of {Product} ?"
                  }
              ]
          }
      ]
  }
}

Looks good. When we run this model, it works but only for values we have declared in the Product slot (for example, buy one hammer). If we use the same utterance as before, we get the following intent from Alexa.


{
    "name": "RegisterNeedIntent",
    "confirmationStatus": "NONE",
    "slots": {
        "Product": {
            "name": "Product",
            "confirmationStatus": "NONE"
        },
        "Action": {
            "name": "Action",
            "value": "buy",
            "resolutions": {
                "resolutionsPerAuthority": [
                    {
                        "authority": "amzn1.er-authority.echo-sdk.amzn1.ask.skill.a20631d9-6a23-413b-8bdc-68ea3c702a10.Action",
                        "status": {
                            "code": "ER_SUCCESS_MATCH"
                        },
                        "values": [
                            {
                                "value": {
                                    "name": "buy",
                                    "id": "0461ebd2b773878eac9f78a891912d65"
                                }
                            }
                        ]
                    }
                ]
            },
            "confirmationStatus": "NONE"
        },
        "Amount": {
            "name": "Amount",
            "value": "20",
            "confirmationStatus": "NONE"
        }
    }
}

My expectation is we would get the raw value and the same resolution result stating there was no match. Consequently, the Dialog.Delegate would simply ask to fill the slot if the resolution was unmatched. In our case, we could have code in our skill to resolve the right value from an external database on the raw user input. If there was a match in our database, we would update the intent with the right resolved product with a confirmation, and delegate the dialog engine to fill in additional missing slots. Instead, we have a completely empty slot and no idea what the user said. Sigh.

We are not completely blocked; but the voice experience is hindered as we can not collect multiple slots in one utterance if it leverages Dialog Management.