4 Tips to Make Better Prompts

I have blogged about sentiment detection and relaxed prompts before. I have recently put the two together and came up with a good recipe for handling prompts. Let me show you why I needed it and how I dealt with it.

Curve Ball

The bot framework can throw you a curve ball if you’re not careful selecting your prompts choices:

I have vs. I have not

It is actually trying to be smart. The bot is not sure but believes with 61% confidence that the user said I have received it. And a clear opposite to the positive option - I have not received it - would match with even stronger 83% score. WAT.

For simple yes/no choices, the bot will try a regex:

1
2
EntityRecognizer.yesExp = /^(1|y|yes|yep|sure|ok|true)(\W|$)/i;
EntityRecognizer.noExp = /^(2|n|no|nope|not|false)(\W|$)/i;

It has a few tricks to help more complicated cases:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
// [excerpt from EntityRecognizer.ts]
// value - one of the choices given to the Prompts
// utterance - the user's response
// both are trimmed and lower cased.

var tokens = utterance.split(' ');

if (value.indexOf(utterance) >= 0) {
score = utterance.length / value.length;

} else if (utterance.indexOf(value) >= 0) {
score = Math.min(0.5 + (value.length / utterance.length), 0.9);

} else {
var matched = '';
tokens.forEach((token) => {
if (value.indexOf(token) >= 0) {
matched += token;
}
});
score = matched.length / value.length;
}

61% is the result of computing 'Ireceivedit'.length/'I have received it'.length. The alternative I have not received it gives us an even better score as more tokens find a match.

Tip #1: Spend some time thinking about how you formulate your choices not to get trapped by the fuzzy match logic.

Ambiguity

Since I built my first chatbot last year, I often find myself consulting EPAM‘s clients on the technology and the approach, and go as far as helping their teams get off the ground building one.

My go-to technique to bootstrap the conversation is to ask a client to document their imaginary conversation with the bot as if it existed. I can get a lot from this simple exercise. I can recommend a proper delivery channel. It will help decide if they need a custom built NLU service or can get by with LUIS or API.ai. I will also use their dialogue to educate them about what’s easy with the commoditized AI and what’s not. Asking the bot to do three things at once, for example, may sound very natural, but will likely be a lot harder to handle.

I also try to disambiguate the prompts:

Bot >> Did you receive my email? [yes/no]
User >> no
Bot >> Did you check your spam folder?

I would instead have the bot say:

Bot >> Please check your spam folder. Did you find the email there? [yes/no]

This way the bot has no problem understanding what the positive yes and the negative no mean.

Tip #2. If the bot needs to ask a yes/no question, make it a yes/no question. Unless, of course, you want to spend time building smarter brains for your bot

Another example:

[IT Support, locked account scenario]

Bot >> You can either wait 15 minutes and try again
Bot >> or you can reset your password to unlock your account now
Bot >> What would you like to do?

You can give the user two mutually exclusive options and lock the prompt, but if you are like me and prefer to keep the prompts more open and relaxed, you might want to change the bot’s prompt to:

Bot >> You can either wait 15 minutes and try again
Bot >> or you can reset your password to unlock your account now
Bot >> Would you like to reset your password? [yes/no]

Last example:

[end of the dialog]

Bot >> Great! Anything else I can help you with?

It’s a very natural prompt, but I suggest you don’t let your bot ask it this way. Not unless you’re ready to handle an arbitrary reply. Instead, have the bot say something like:

Bot >> Great! I am glad I was able to help

Tip 3. Don’t solicit feedback from your user that you are not equipped to handle.

Sentiment

If you follow the first three tips, you are very likely to have more yes/no prompts in your dialogs than other binary questions. EntityRecognizer does a good job with a simple regex but you may want to dial it up a notch with sentiment detection.

The idea is simple. Prompt the user with a yes/no question but do it in a relaxed manner. Let the user answer with whatever they feel like if they don’t use the buttons. Then, let the Bot Framework try to understand if it was a yes or a no. If not successful, turn to sentiment detection and treat a positive expression as a yes and a negative as a no. And finally, if sentiment detection comes back inconclusive, re-prompt the user and this time lock the choices to yes/no.

Here’s a reusable macro:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
// [module sentiment.js]
// github link to the full implementation is provided below

module.exports = {
detect: function (text, language = 'en', threshold = 0.05) {
},

confirm: function (question, reprompt) {
return [
// Step 1. Relaxed yes/no prompt via Prompts.confirm
(session, args, next) => {
builder.Prompts.confirm(session, question,
{
listStyle: builder.ListStyle.button,
maxRetries: 0 // <-- no re-prompt
})
},
// Step 2. Try Sentiment detection as an alternative
(session, args, next) => {
if (args && typeof(args.response) !== 'undefined') {
// The bot framework recognized a 'yes' or a 'no'
next(args);
} else {
// Turn to sentiment detection
this.detect(session.message.text)
.then(response => next(response))
.catch(error => {
console.error(error);
next();
});
}
},
// Step 3. Re-prompt if needed
(session, args, next) => {
if (args && typeof(args.response) !== 'undefined') {
// We have a yes/no
next(args);
} else {
// Inconclusive. Need to re-prompt.
reprompt = reprompt ||
'I am sorry, I did not understand what you meant. ' +
'See if you can use the buttons ' +
'or reply with a simple \'yes\' or \'no\'. ';

session.send(reprompt);

builder.Prompts.confirm(session, question,
{
listStyle: builder.ListStyle.button
// <-- maxRetries is not set, re-prompt indefinitely
})
}
}
]
}
};

And now we can easily use it in our dialogs thanks to the spread syntax:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
const sentiment = require('./sentiment');

bot.dialog('/addToCart', [
function (session, args, next) {
// ...
},

// <-- using the macro we've just created
...sentiment.confirm('Would you like to see a few recommendations?'),

// <-- next waterfall step will receive a proper yes/no
function (session, args, next) {
if (!args.response) {
session.endDialog('Alright');
} else {
showRecommendations(session);
}
}
]);

I am using this technique in my e-commerce chatbot example and here’s a link to the full sentiment.js

Tip #4: Make your prompts handling smarter with sentiment detection but be ready to lock the user into a yes/no decision if sentiment detection comes back inconclusive.


Sentiment detection is not without traps either:

1
2
3
4
5
6
7
> const sentiment = require('./app/sentiment');
undefined

> sentiment.detect('no, thanks!')
Promise { <pending> }

> SENTIMENT: 0.941164496065538 in no, thanks!

That was a very positive no, apparently :)

Cheers!

Smarter Conversations. Part 4 - Transcript

A bot that one of our teams is working on has the following functional requirement:

Dialog reaches a point where chatbot is no longer able to help. At this point, a transcript of the conversation will be sent to a mailbox.

Capturing a transcript requires that we keep track of all messages that are sent and received by the bot. The framework only keeps track of the conversations’ current dialogs stack. I already showed you guys how to build a simple history engine and give the bot the breadcrumbs of the entire conversation. Let’s see how we can record a transcript.

Option 1. Events (first attempt)

UniversalBot extends the node.js’s EventEmitter and will produce a number of events as it processes incoming and outgoing messages. We can subscribe to send and receive, for example:

1
2
3
4
5
6
7
8
9
10
11
bot.on('send', function(event) {
if (event.type === 'message') {
// ToDo: record in the transcript journal
}
});

bot.on('receive', function(event) {
if (event.type === 'message') {
// ToDo: record in the transcript journal
}
});

There’s a little caveat that I want to bring up before I show you how to get to the conversation’s session in the event handler.

send and receive are emitted before the bot runs through the middleware stack. In general, an exception in one of the middleware components should not break the chain, but if you want to only capture messages that were actually dispatched to the user, you would subscribe to outgoing that files after the middleware chain.

Let’s now add the journaling logic.

First attempt:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
const transcript = function (session, direction, message) {
session.privateConversationData.transcript = session.privateConversationData.transcript || [];
session.privateConversationData.transcript.push({
direction,
message,
timestamp: new Date().toUTCString()
});

// NOTE 1: I will explain this line in details and show you
// that it doesn't actually do what you might think it does
session.save();
};

bot.on('incoming', function (message) {
if (message.type === 'message') {

// NOTE 2: loadSession() warrants an in depth explanation as well
bot.loadSession(message.address, (error, session) => {
transcript(session, 'incoming', message.text);
});
}
});

bot.on('outgoing', function (message) {
// ... (same as incoming, will refactor later)
});

NOTE 1. session.save()

It’s very important to understand how the bot handles the session data. The default mechanism is MemoryBotStorage that stores everything in memory and works synchronously. Your bot would default to it if you used the ConsoleConnector. You are a lot more likely to use the ChatConnector that comes with external persistence implementation. It will be reading and saving data asynchronously. Please also note that everything you put on session (e.g. session.userData) is JSON serialized for storage. Don’t try keeping callback functions around on the session.dialogData, for example.

The next very important thing to understand is that session.save() is asynchronous as well. It’s actually worse. It’s delayed via setTimeout(). The delay is configurable via autoBatchDelay and defaults to 250 milliseconds. The bot will auto-save all session data as part of sending the messages out to the user which it does in batches. The delay is built into the batching logic to ensure the bot doesn’t spend extra I/O cycles when it feels like sending multiple messages. Calling session.save() just triggers the next batch.

You can remove the delay:

1
2
3
4
const bot = new builder.UniversalBot(connector, {
persistConversationData: true,
autoBatchDelay: 0 // <-- the default is 250
});

The batching will still be asynchronous though. You can also bypass the batching altogether and instead of session.save() call session.options.onSave() directly, but you can’t work around the asynchronous nature of how the data is saved by the ChatConnector.

NOTE 2. bot.loadSession()

This method is not part of the documented public API and there’s probably a good reason for it. The bot framework doesn’t keep the sessions around. Session objects are created on demand and discarded by the GC when the request/response cycle is over. In order to create a new session object, the bot needs to load and deserialize the session data which as you just have learned happens asynchronously.

If you run the code I showed you, you will only see the outgoing messages on the transcript.

The incoming messages are swallowed and overwritten by the asynchronous and delayed processing.

Option 1. Events (second attempt)

There’s one event in the incoming message processing pipeline that is different from all others - routing. An event handler for routing is given a session object that the bot framework has just created to pass on to the selected dialog. We can transcript without having to load our own session instance:

1
2
3
bot.on('routing', function (session) {
transcript(session, 'incoming', session.message.text);
});

The routing event is the last in the chain of receive -> (middleware) -> incoming -> routing.

There is no equivalent to routing on the way out though. No event in send -> (middleware) -> outgoing chain is given the session object. There is a good reason why. Sending the messages out happens after the bot finished saving the session data.

While it’s sad that we don’t have an equivalent of routing in the outbound pipeline, knowing that session data is complete prior to bot framework dispatching the messages out makes me feel good about re-saving it. We don’t risk overwriting anything important like call stack or other session data.

Second attempt:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
const transcript = function (session, direction, message) {
session.privateConversationData.transcript = session.privateConversationData.transcript || [];
session.privateConversationData.transcript.push({
direction,
message,
timestamp: new Date().toUTCString()
});

// no need to explicitely save() for the incoming
if (direction === 'outgoing') {
session.save();
}
};

bot.on('routing', function (session) {
transcript(session, 'incoming', session.message.text);
});

bot.on('outgoing', function (message) {
if (message.type === 'message') {
bot.loadSession(message.address, (error, session) => {
transcript(session, 'outgoing', message.text);
});
}
});

This time it works as expected but is not free of side effects. The bot.loadSession() on the way out is still asynchronous and prone to interleaving. If your bot starts sending multiple messages and especially doing so asynchronously in response to receiving external data via a Promise, for example, you may find yourself not capturing all of it.

Option 2. Middleware

Another way of intercepting incoming and outgoing messages is to inject a custom middleware. The middleware is called in between receive and incoming, and also in between send and outgoing:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
bot.use({
send: function (message, next) {
if (message.type === 'message') {
// ToDo: record in the transcript journal
}
next(); // <-- I will explain in NOTE 3 below
},
receive: function (message, next) {
if (message.type === 'message') {
// ToDo: record in the transcript journal
}
next(); // <-- I will explain in NOTE 3 below
}
});

NOTE 3. next()

Middleware that you inject via bot.use() form a stack that is processed synchronously and in order. The bot framework does it via a recursive function that self-invokes. Every invocation notifies the next middleware in the chain and will eventually call the main processing callback. This is a nice way to keep running down the list even when one errors out as it will self-invoke in a catch block. I suggest that you guys take a closer look at UniversalBot.prototype.eventMiddleware if you’re interested. So if we don’t call next(), the chain will not continue and the bot will never receive the message.

We can use this feature to our advantage. If we chain next() onto the direct call to session.options.onSave(), we can ensure that the chain continues after the successful journaling of the transcript. No chance to have them all interleave and overwrite one another, though it probably takes longer before it gets to the user:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
const transcript = function (session, direction, message, next) {
session.privateConversationData.transcript = session.privateConversationData.transcript || [];
session.privateConversationData.transcript.push({
direction,
message,
timestamp: new Date().toUTCString()
});

session.options.onSave(next);
};

const journal = (direction) => (message, next) => {
if (message.type === 'message') {
bot.loadSession(message.address, (error, session) => {
transcript(session, direction, message.text, next);
});
} else {
next();
}
};

bot.use({
send: journal('outgoing'),
receive: journal('incoming')
});

You can also combine the two techniques and use routing event for incoming messages and only use send middleware to capture the outgoing traffic. Just make sure that you don’t do session.save() for the incoming. Here’s a gist.

Option 3. External Joural

I don’t know how stable is session.options.onSave() and bot.loadSession(). Neither one is part of the official public API so use at your own risk.

You can also roll your own transcript service and safely call it asynchronously from the send and receive event handlers. What I like about using session.privateConversationData is that I need no custom infrastructure and can easily discard the transcripts if I don’t use them. The bot framework will take care of it for me.

It would be nice though if bot framework gave me a routing-like event for the outbound pipeline that would fire before saving of the data. This way I would be able to nicely record the transcript without disrupting the flow of things, and wouldn’t risk relying on internal implementation detail that can easily go away in the next version.

Smarter Conversations. Part 3 - Breadcrumbs

This post continues the smarter conversations series and today I would like to show you how to keep track of the conversation flow and help your bot remember and reason about it. Previously, in part 1, I showed how to add sentiment detection to your bot and in part 2 I explored ways to keep your dialogs more open.

In part 1 I used the following dialog to illustrate why you might want to be able to detect expressed sentiment:

User >> I’m looking for screws used for printer assembly
Bot >> Sure, I’m happy to help you. 
Bot >> Is the base material metal or plastic?
User >> metal
Bot >> [lists a few recommendations]
Bot >> [mentions screws that can form their own threads]
User >> Great! I think that's what I need
Bot >> [recommends more information and an installation video]

The highlighted phrase is not an expression of a new intent, not an answer to the bot’s prompt. It’s a positive emotional reaction to the perfectly timed recommendation about thread forming screws. We were able to capture it and present to our bot as an intent:

1
2
3
4
5
6
7
bot.dialog('affirmation', [
function (session, args, next) {
// ...
}
]).triggerAction({
matches: 'Affirmation'
});

Unlike other intents, however, the Affirmation intent can’t be fulfilled without knowing what came before it. Wouldn’t it be great if the bot had access to the conversation’s breadcrumbs? If it could reason about what was talked about before?

History Engine

While the bot framework doesn’t keep the history of triggered intents and actions beyond the active dialog stack, it’s not hard to build a simple history engine that would take care of it.

Probably the easiest way to do it is via the onSelectRoute hook:

1
2
3
4
5
6
7
8
9
10
11
12
13
// ...
const bot = new builder.UniversalBot(connector);

bot.onSelectRoute(function (session, route) {
session.privateConversationData.history = session.privateConversationData.history || [];
session.privateConversationData.history.push(route.routeData.action);
session.save();

// Don't forget to call the default processor.
// While the "on" syntax suggests that it's an event handler,
// the onSelectRoute actually replaces the default logic with yours
this.defaultSelectRoute(...arguments);
});

The route.routeData.action is the name of the dialog that is about to be triggered. Here’s how your bot would use it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
const affirmations = {
'*:productLookup': (session, args, next) => {
// handle positive reaction right after product lookup
},
'*:howToOrder': (session, args, next) => {
// handle positive reaction right after ordering tips
}
}

bot.dialog('affirmation', [
function (session, args, next) {
const history = session.privateConversationData.history || [];

// The last step in the history is the one currently being executed
const affirmationFor = history[history.length - 2];
const action = affirmations[affirmationFor];

if (!action) {
session.endDialog();
} else {
action(session, args, next);
}
}
]).triggerAction({
matches: 'Affirmation',
onSelectAction: function (session, args, next) {
// keep the interrupted dialog on the stack
session.beginDialog(args.action, args);
}
});

It’s important to note that if you are using the IntentDialog, you won’t see onSelectRoute triggered for your intent.matches(). This is because the matching is handled by the dialog, not the routing system. I stopped using the IntentDialog bound to / in favor or recently added global recognizers and triggers and will soon upgrade my ecommerce bot.

Relaxed Prompt

I wanted to share one more technique that I recently discovered and started using a lot to keep my prompts more open, more relaxed.

In the product selection dialog, for example, you may find yourself giving your user a set of options to choose from and also an option to forego the selection:

...
Bot >> Would you like to look at one particular brand? 
Bot >> [lists a few brand choices as buttons] 
User >> No, thank you

The answer no, thank you is not one of the brand options and I wouldn’t render it as such either. I would like the bot to accept one of the options given and consider everything else not picked up by any other recognizer as a no, thank you answer.

All we need to do, apparently, is to make sure the bot doesn’t reprompt if it receives a wrong answer and is ready for an alternative response:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
bot.dialog('brands', [
function (session, args, next) {
// products that match previous user's selections
const products = session.privateConversationData.products;
// distinct list of brands
const brands = [...new Set(products.map(p => p.brand))];

// will come in handy when processing the response
session.dialogData.brands = brands;
session.save();

builder.Prompts.choice(session,
'Would you like to look at one particular brand?',
brands,
{
listStyle: builder.ListStyle.button,
maxRetries: 0 // <-- No re-prompt
}));
},
function (session, args, next) {
// either one of the options provided, or something else
const reply = (args.response && arg.response.entity) ||
session.message.text;

const brands = session.dialogData.brands;

if (brands.includes(reply)) {
// continue with a list filtered down by the selected brand
} else {
// "no, thank you". continue with a full list
}
}
]);

That’s it for today. Next time I will show you how to keep a full history of a conversation and be ready to send a transcript to the customer support agent when the bot gets stuck.

Integrating Bot Framework with api.ai

My go-to NLU service for all the bot prototypes that I build with Microsoft Bot Framework is LUIS. This time, however, I needed to build a bot that would speak a language that LUIS doesn’t understand yet. I needed my bot to speak Russian.

api.ai

The Bot Framework comes with built-in support for LUIS but it’s not hard to build your own intent recognizer.

It probably took me under ten minutes to sign up for api.ai, orient myself with the tool, and train an agent that would understand one intent and extract one entity out of it. Their web interface is very slick, very intuitive to navigate.

I didn’t set up any events or actions, didn’t configure webhook fulfillments, and didn’t use the one-click integrations. All I needed my api.ai agent to do was to recognize the intent and extract the entity. Everything else in my case is done by the Bot Framework.

I could now send the request with my user’s utterance to api.ai and receive a JSON payload back:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
{
"id": "42384260-8f60-4473-9e69-1dab4b286fa6",
"timestamp": "2017-03-23T13:46:34.812Z",
"lang": "ru",
"result": {
"source": "agent",
"resolvedQuery": "хочу купить кофеварку",
"action": "",
"actionIncomplete": false,
"parameters": {
"product": "кофеварка"
},
"contexts": [],
"metadata": {
"intentId": "a407b3f7-5874-4d97-b261-e3564d8dfc4d",
"webhookUsed": "false",
"webhookForSlotFillingUsed": "false",
"intentName": "buyCoffeeMaker"
},
"fulfillment": {
"speech": "",
"messages": [
{
"type": 0,
"speech": ""
}
]
},
"score": 1
},
"status": {
"code": 200,
"errorType": "success"
},
"sessionId": "af9eb509-77cb-402b-a32c-d28f7d8d3aa2"
}

Recognizer

api.ai comes with an SDK for pretty much any platform you will want to use it on. I build bots with node.js and they had the npm package for me:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
const apiai = require('apiai');
const app = apiai(process.env.APIAI_TOKEN);

module.exports = {
recognize: function (context, callback) {
const request = app.textRequest(context.message.text, {
sessionId: `${Math.random()}`,
language: 'ru-RU'
});

request.on('response', function (response) {
const result = response.result;

callback(null, {
intent: result.metadata.intentName,
score: result.score,
entities: Object.keys(result.parameters)
.filter(key => !!result.parameters[key])
.map(key => ({
entity: result.parameters[key],
type: key,
score: 1
}))
});
});

request.on('error', function (error) {
callback(error);
});

request.end();
}
};

And that’s it. My bot speaks Russian now.

Smarter Conversations. Part 2 - Open Dialogs

This post continues the smarter conversations series and today I would like to explore ways of keeping your dialogs open. Previously, in part 1, I showed how to add sentiment detection to your bot.

Waterfall

Prior to 3.5.3, the dialog routing system in the Bot Framework was not very flexible.

Imagine the following dialog:

User >> I’m looking for screws used for printer assembly
Bot >> Sure, I’m happy to help you. 
Bot >> Is the base material metal or plastic?
User >>I don't know. Does it matter?

The question that the bot asks about the material will likely be handled as a waterfall:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
const builder = require('botbuilder');

bot.dialog('productLookup', [
function (session, args, next) {
// ...
builder.Prompts.choice(session, 'Is the base material metal or plastic?',
['metal', 'plastic'],
{ listStyle: builder.ListStyle.button });
},
function (session, args, next) {
const material = args.response.entity;
// ...
}
]);

The user’s response is neither metal nor plastic and the bot would simply reprompt:

Reprompt

The builder.Prompts.choice opens up a new dialog that gets pushed onto the stack and that’s what receives the next message. We will take a closer look in a minute.

Trigger Actions

The routing system was reworked in 3.5.3 and it came with a few important enhancements.

First, you no longer need the IntentDialog to recognize your users’ intents. The UniversalBot now inherits from Library and has its own set of global recognizers:

1
2
3
4
5
6
7
8
9
10
const bot = new builder.UniversalBot(connector);

// custom recognizers
const smiles = require('./app/recognizer/smiles');
const sentiment = require('./app/recognizer/sentiment');

// set up global recognizers
bot.recognizer(smiles);
bot.recognizer(sentiment);
bot.recognizer(new builder.LuisRecognizer(process.env.LUIS_ENDPOINT));

Second, the dialogs can now define trigger actions and be picked up even while another dialog’s prompt is being processed.

1
2
3
4
5
6
7
bot.dialog('affirmation', [
function (session, args, next) {
// ...
}
]).triggerAction({ // <-- this right here
matches: 'Affirmation'
});

If our bot had an intent recognizer that could understand that the user asked a question instead of answering the metal vs. plastic question, and if we had a way to handle it, we could break out of the waterfall using the triggerAction technique. In part 3 I will show you how a simple history engine can help you attach a sentiment like that to what was happening previously in the conversation and how your bot can intelligently handle such a diversion.

Routing and Callstack

Bot Framework maintains a callstack of the triggered dialog actions. When user utterance triggered the productLookup dialog, the stack only had one item coming into the first function of the waterfall:

1
*:productLookup

The builder.Prompts.choice adds another one:

1
2
*:productLookup
BotBuilder:Prompts <--

In the routing system that came before 3.5.3, the next message would land onto BotBuilder:Prompts, would upset the choice validation logic, and would trigger a reprompt. The newer version does a much better job.

First, the UniversalBot runs the incoming message through the set of global recognizers. Then the default routing mechanism runs three parallel searches - global actions, stack actions, and active dialogs. In doing so it collects all matching route results and scores them. The best route will then be selected and executed.

Handling Interruptions

The default behavior of launching a new dialog via its triggerAction is to clean up the callstack and start fresh. You can do two things to handle the interruption.

First, you can override the default behavior with onSelectAction. Instead of resetting the callstack you can add the newly triggered dialog on top of it. This would return the conversation back to where it was interrupted at when the newly triggered dialog finishes:

1
2
3
4
5
6
7
8
9
10
11
12
bot.dialog('affirmation', [
function (session, args, next) {
// ...
}
]).triggerAction({
matches: 'Affirmation',

// <-- overwrite how the dialog is launched
onSelectAction: function (session, args, next) {
session.beginDialog(args.action, args);
}
});

And you can also attach the onInterrupted handler to the dialog that could be interrupted and message the user about what’s happening.

Open All The Way

And if that was not flexible enough, you can define your own dialog’s behaviors by overwriting begin, replyReceived, and even recognize on your dialogs:

1
2
3
4
5
6
7
8
bot.dialog('custom', Object.assign(new builder.Dialog(), {
begin: (session) => {
session.send('I am built custom');
},
replyReceived: (session) => {
session.endDialog();
}
}));

I will sure come back to this technique when I show you how to drive your dialogs from metadata and not code. Comes very handy when building product recommendation bots. Stay tuned!

Smarter Conversations. Part 1 - Sentiment

This post starts a series of short articles on building smarter conversations with Microsoft Bot Framework. I will explore detecting sentiment (part 1), keeping the dialog open-ended (part 2), using a simple history engine to help the bot be context-aware (part 3), and recording a full transcript of a conversation to intelligently hand it off to a human operator.

Affirmation

Imagine the following dialog:

User >> I’m looking for screws used for printer assembly
Bot >> Sure, I’m happy to help you. 
Bot >> Is the base material metal or plastic?
User >> metal
Bot >> [lists a few recommendations]
Bot >> [mentions screws that can form their own threads]
User >> Great! I think that's what I need
Bot >> [recommends more information and an installation video]

It’s not hard to train an NLU service like LUIS to see a product lookup intent in the first sentence. A screw would be an extracted entity. Following a database lookup, the bot then clarifies an important attribute to narrow the search down to either plastic or metal screws and presents the results.

The highlighted sentence that follows is a positive affirmation. It is not an intent that needs to be fulfilled, not an answer to the question asked by the bot. And yet it presents an opportunity for a smarter bot to be more helpful, act as an advisor.

Sentiment

Text Analytics API is part of the Microsoft’s Cognitive Services offering. The /text/analytics/v2.0/sentiment endpoint makes it a single HTTP request to score a text fragment or a sentence on a scale from 0 (negative) to 1 (positive).

I decided to make the expressed sentiment look like an intent for my bot and so I built a custom recognizer:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
const request = require('request-promise-native');

const url = process.env.SENTIMENT_ENDPOINT;
const apiKey = process.env.SENTIMENT_API_KEY;

module.exports = {
recognize: function (context, callback) {
request({
method: 'POST',
url: `${url}`,
headers: {
'Content-Type': 'application/json',
'Ocp-Apim-Subscription-Key': `${apiKey}`
},
body: {
"documents": [
{
"language": "en",
"id": "-",
"text": context.message.text
}
]
},
json: true
}).then((result) => {
if (result && result.documents) {
const positive = result.documents[0].score >= 0.5;

callback(null, {
intent: positive ? 'Affirmation' : 'Discouragement',
score: 0.11 // <-- just above the threshold
});
} else {
callback();
}
}).catch((reason) => {
console.log('Error detecting sentiment: %s', reason);
callback();
});
}
};

Context

Now I can attach a dialog that would be triggered when the bot detects an affirmation and no other intent scores higher. The default intent threshold is 0.1 and that’s why a detected sentiment is given 0.11.

1
2
3
4
5
6
7
8
9
10
11
const sentiment = require('./app/recognizer/sentiment');

bot.recognizer(sentiment);

bot.dialog('affirmation', [
function (session, args, next) {
// ...
}
]).triggerAction({
matches: 'Affirmation'
});

Unlike other intents, however, a detected sentiment is not enough to properly react to on its own.

The bot needs to understand the context to properly react to an affirmation or discouragement expressed by a user. The bot also needs to be able to handle an interrupted dialog if an affirmation (or an expression of frustration) came in the middle of a waterfall, for example.

I will get to it in part 2. Stay tuned.

Ecommerce Chatbot

I have published a short screencast about the chatbot that I built and have also shared the code on github.

Enjoy!

Understanding Date Ranges in Your Chatbot

When your chatbot performs tasks of a personal assistant like scheduling meetings or generating reports, you need to make sure it can understand dates and date ranges.

Step 1. Resolve

LUIS has a set of pre-built entities to recognize date and time (builtin.datetime). It will understand when your users say tomorrow, October 1st or next week, for example, and will convert that to a date or a duration. Couple examples:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
// tomorrow
"resolution": { "date": "2016-11-20" }

// last quarter
"resolution": { "date": "XXXX-Q4" }

// last year
"resolution": { "date": "2015" }

// last two years
"resolution": { "duration": "P2Y" }

// last week
"resolution": { "date": "2016-W45" }

// past three weeks
"resolution": { "duration": "P3W" }

// this month
"resolution": { "date": "2016-11" }

// last ten months
"resolution": { "duration": "P10M" }

Unfortunately, the only quarter-based duration LUIS understands right now is last quarter. It doesn’t recognize this quarter, next quarter, or plurals like last three quarters.

As you can see, the resolutions are indicative, use different formats, and need to be parsed to get converted to dates and date ranges.

Step 2. Parse

When LUIS detects a datetime entity (e.g. tomorrow) it will send back the resolution along with the extracted entity itself (the word tomorrow in this case).

First, I try to understand what time span the user asked about:

1
2
const span = 
['day', 'week', 'month', 'quarter', 'year'].find(s => entity.match(s));

Then I parse the dates and durations with moment:

1
2
3
4
5
6
7
8
9
10
11
12
13
const moment = require('moment');

// date
const resolved = resolution.date.replace('XXXX', moment().year());
const date = moment(resolved, ['YYYY-MM-DD', 'YYYY-Q', 'YYYY-W', 'YYYY']);

// duration
const duration = moment.duration(resolution.duration);
const sign = ['last', 'past', 'previous'].some(p => entity.match(p)) ? -1 : +1;
const date = moment().add(sign * duration.as('hours'), 'hours');

// normalized result
return date.startOf(span || 'day');

Step 3. Understand

Now we have the date representing the beginning of the period the user asked about. If today was Friday 11/18, for example, and you asked for last three weeks, the date would be Sun, Oct 23 (weeks start on Sunday in US unless you use isoweek with moment).

One date is not enough though for utterances like:

1
please generate a service cost report for the last two weeks

Your report generation service/API is likely to require a date range.

LUIS can also understand numbers spelled as digits like 2 or 5 or spelled as words like two or five. A phrase like last two weeks will produce two entities:

1
2
3
4
5
6
7
8
9
10
11
12
13
"entities": [
{
"entity": "two",
"type": "builtin.number"
},
{
"entity": "last two weeks",
"type": "builtin.datetime.duration",
"resolution": {
"duration": "P2W"
}
}
]

Last thing I need to do to understand the range, is to extract the number and do the date math:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
const moment = require('moment');
const builder = require('botbuilder');

const numbers = {
'one': 1,
'two': 2,
'three': 3,
// you got the idea
};

// the entity here is the 'builtin.number'
const range = builder.EntityRecognizer.parseNumber(entity)
|| numbers[entity]
|| 1;

const end = moment(date)
.add(range, span)
.subtract(1, 'day')
.endOf('day');

And that’s it. Now last three weeks is understood as 10/23 - 11/12. And last quarter will be 10/1 0:00 - 12/31 23:59.

Intent Recognizers For Your Chatbot

Two weeks ago I attended API Strat in Boston where I gave a talk on cognitive APIs and conversational interfaces and showed and explained an e-commerce chatbot that I built. My presentation is on slideshare. I have learned a lot about chatbots and now I feel an urge to write about it.

Skype conversation excerpt

Intents

My bot is using the intent dialog from the Microsoft Bot Framework:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
const bot = new builder.UniversalBot(...);
const intents = new builder.IntentDialog(...);

intents.matches('Greeting', '/welcome');
intents.matches('ShowTopCategories', '/categories');
intents.matches('Explore', '/explore');
intents.matches('ShowProduct', '/showProduct');
intents.matches('AddToCart', '/addToCart');
intents.matches('ShowCart', '/showCart');
intents.matches('Checkout', '/checkout');
intents.matches('Reset', '/reset');
intents.matches('Smile', '/smileBack');
intents.onDefault('/confused'); // no intent recognized

bot.dialog('/', intents);

bot.dialog('/confused', [
function () {
session.endDialog('Sorry, I didnt understand you');
}
]);

The intent dialog associates a user’s intent like Explore or Checkout with a specific dialog that knows how to respond.

It feels very much like routing in a web framework where given a specific URL pattern, the request will be routed to a controller that knows how to handle it.

Users don’t spell out their intents like that though. And so the first thing my bot needs to do is to learn to recognize them. The simplest way to trigger a dialog handler in response to a users utterance is by matching it with a regex. A more sophisticated logic requires an intent recognizer.

Intent Recognizers

An intent recognizer is basically a service that can understand users’ utterances. Given a text message it will return a list of intents that it inferred from it along with supporting entities. Here’s how it looks in LUIS (language understanding service from Microsoft):

LUIS intents and entities

The Explore intent was recognized along with two supporting entities that I trained it for. Here’s another way of looking at it:

1
2
3
4
5
curl -v "https://api.projectoxford.ai/luis/v2.0/apps/{app-id}" 
-H "Content-Type: application/json"
-H "Ocp-Apim-Subscription-Key: {subscription-key}"
-G
-d "q=I am looking for touring bikes. Do you have some?"

And the response:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
{
"query": "I am looking for touring bikes. Do you have some?",
"topScoringIntent": {
"intent": "Explore",
"score": 0.9994699,
"actions": [
// ...
]
},
"entities": [
{
"entity": "touring",
"type": "Detail",
"score": 0.9710912,
// ...
},
{
"entity": "bikes",
"type": "Entity",
"score": 0.943606555,
// ...
}
],
// ...
}

Microsoft Bot Framework comes with built-in support for LUIS in the form of LuisRecognizer

Custom Recognizers

Not every thing your users say has to be sent to a natural language service to extract the intent. Buttons and tappable images can post back bot-specific commands like /show:123456789, for example, that you can easily recognize with a regex. Also, if you want your bot to smile back at a smile sent to it, you don’t need to train a linguistic model either.

It turns out, building your own recognizer is not hard at all. I have built a few for my e-commerce bot and here’s how it works.

First, know that the Bot Framework supports sending a message through a number of recognizers at the same time. You can chain them or run them all in parallel:

1
2
3
4
5
6
7
8
9
10
const intents = new builder.IntentDialog({
recognizers: [
commands,
greeting,
smiles,
new builder.LuisRecognizer(process.env.LUIS_ENDPOINT)
],
intentThreshold: 0.2,
recognizeOrder: builder.RecognizeOrder.series
});

The recognizer itself is a very simple interface with only one method - recognize. Here’s how you would detect a smile, for example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
module.exports = {
recognize: function(context, callback) {
const text = context.message.text;
const smiles = text.match(/<ss type="(\w+?)">(.+?)<\/ss>/);

if (smiles) {
callback.call(null, null, {
intent: 'Smile',
score: 1,
entities: [
// smiles[1] and smiles[2]
// have the details you need to smile back
]
});
} else {
callback.call(null, null, {
intent: null,
score: 0
});
}
}
};

And here’s another one that understands commands:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

const commands = {
parse: function (context, text) {
const parts = text.split(':');
const command = parts[0];

const action = this[command] || this[command.slice(1)];
if (!action) {
return unrecognized;
} else {
return action.call(this, context, ...parts.slice(1));
}
},
// ...
}

module.exports = {
recognize: function (context, callback) {
const text = context.message.text;

if (!text.startsWith('/')) {
callback.call(null, null, unrecognized);
} else {
callback.call(null, null, commands.parse(context, text));
}
}
};

That’s it for now but there is more to come. Stay tuned!