Promiseland and Async/Await Kingdom

Last night I finally got a chance to publish the remaining setup scripts for my E-Commerce Chatbot. A few days ago, I added the script to load up products and variants into Azure Search and now also the catalog and historical transactions for Azure Recommendations. I basically had to script what I originally did as a one-off with curl.

Training the recommender model takes time and when you create a new recommendation build, it won’t be ready right away. I wanted my script to wait and keep polling the API until the training has finished. The whole script is basically a serious of asynchronous HTTP requests so I wired it all up as a chain of promises:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
sdk.model.list()
.then(({ models }) => {
// ...
}).then(() => {
return sdk.model.create(modelName, description);
}).then(() => {
return sdk.upload.catalog(...);
}).then(() => {
return sdk.upload.usage(...);
}).then(() => {
return sdk.build.fbt(...)
}).then(result => {

// <--
// ToDo: need to wait until the training is finished
// <--

}).then(() => {
console.log(`Set RECOMMENDATION_MODEL to ${model.id}`);
console.log(`Set RECOMMENDATION_BUILD to ${build.buildId}`);
}).catch(error => {
console.error(error);
});

You can see the full listing here.

Promiseland

Here’s how I implemented the wait-and-see:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
// ...
}).then(build => {
const check = (timeout) => new Promise((resolve, reject) => {
setTimeout(() => sdk.build.get(model.id, build.buildId)
.then(response => {
if (!['NotStarted', 'Running'].includes(response.status)) {
console.log(`Build training finished: ${response.status}`);
resolve();
} else {
console.log(`Training is ${response.status}. Wait 30 seconds...`);
resolve(check(30000));
}
})
.catch(reject), timeout);
});

return check();
}).then(() => {
// ...
});

It’s basically a recursive promise. The function in the main then() will return a promise that will always resolve unless there’s an error, but the key is in what it will resolve with and how it runs. The function that the returned promise is wrapped around schedules itself via setTimeout() and exits the stack frame. Then, when the response is received, it will either resolve and signal that the training has complete, or it will resolve with another promise that will recursively repeat this process again. That another promise will basically insert itself into the main then chain and it will keep waiting until it resolves. Vicious circle.

It worked nicely and I even factored out the repeater so that my code looked like this:

1
2
3
4
5
6
7
8
9
10
11
// ...
}).then(build => {
return repeater.repeat(() => sdk.build.get(model.id, build.buildId), {
delay: 30000,
until: (response) => !['NotStarted', 'Running'].includes(response.status),
done: (response) => console.log(`Build training finished: ${response.status}`),
next: (response, delay) => console.log(`Training is ${response.status}. Wait ${delay / 1000} seconds...`)
});
}).then(() => {
// ...
});

Async/Await Kingdom

I really thought that I was very cleaver but then I decided to rewrite with async/await and run with the latest node that now natively supports it

Here’s what this code became:

1
2
3
4
5
6
7
8
9
10
11
12
let trained = false;
while (!trained) {
let check = await sdk.build.get(model.id, build.buildId);

if (!['NotStarted', 'Running'].includes(check.status)) {
trained = true;
console.log(`Build training finished: ${check.status}`);
} else {
console.log(`Training is ${check.status}. Wait 30 seconds...`);
await new Promise(resolve => setTimeout(resolve, 30000));
}
}

Here. Compare both versions: Original vs. Async/Await

It’s not even funny! The code is so boring now, boring and simple. Just like it should be. No need to be clever and I bet I will know exactly what it’s doing and why when I look at it a year later.

I have officially converted.

Cheers!

How To Promisify Moltin APIs

If you’ve read my last post, then you know that I am having all kinds of geeky fun with Moltin and its APIs. Today I will show you how you can quickly promisify them all.

Moltin APIs

Moltin APIs are all asynchronous HTTP calls with very lightweight wrappers for JavaScript, Python, and many other languages. I am using it with node.js and the main pattern is fairly straightforward (look for js examples).

I am writing a batch import to create a playground product catalog so I find myself doing a lot of this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
const inventory = Promise.all(goGetTheData()); 

inventory.then((data) => {
return Promise.all(data.products.map(product => {
return new Promise((resolve, reject) => {
moltin.Authenticate(function() {
moltin.Product.Create({
// .. product attributes
}, (result) => {
resolve(result);
}, (error, details) => {
reject(details);
});
});
});
}));
}).then((products) => {
return Promise.all(products.map(p => {
return new Promise((resolve, reject) => {
moltin.Authenticate(function() {
// ...
});
});
}));
}).then((modifiers) => {
// ... you got the idea, a lot of noise
});

I wish I could instead write:

1
2
3
4
5
6
7
inventory.then((data) => Promise.all(data.products.map(p => {
return moltin.Product.Create(...);
}))).then((products) => Promise.all(products.map(p => {
return moltin.Modifier.Create(...);
}))).then((modifiers) => {
// ... a lot cleaner and more readable, isn't it?
});

Promisification

There’s moltin-util on NPM that uses Promises but it seems to introduce a new API and I would like to retain the original. Here’s what I quickly put together and I now wonder if it’s worth posting to NPM. Is it? Let me know!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
const request = require('request');
const fs = require('fs');

const promisify = (moltin) => {
const promisified = {}

const executor = (actor, action) => function () {
const args = [...arguments];

let success = (result, pagination) => {
if (result && pagination) {
result.pagination = pagination;
}

return result;
};

let error = (error, details) => details;

if (typeof (args[args.length - 1]) === 'function') {
if (typeof (args[args.length - 2]) === 'function') {
error = args.pop();
success = args.pop();
} else {
success = args.pop();
}
}

return new Promise((resolve, reject) => {
moltin.Authenticate(function () {
actor[action].call(actor, ...args,
(result, pagination) => {
resolve(success.call(null, result, pagination));
},
(err, details) => {
console.error(details);
reject(error.call(null, details));
});
});
});
};

Object.keys(moltin)
.filter(key => key !== 'options' && typeof (moltin[key]) === 'object')
.forEach(member => {
promisified[member] = {};
let actor = moltin[member];

Object.keys(actor.__proto__)
.concat(Object.keys(actor.__proto__.__proto__))
.filter(action => typeof (actor[action]) === 'function')
.forEach(action => {
promisified[member][action] = executor(actor, action);
});
});

return promisified;
}

module.exports = function (moltin) {
return promisify(moltin);
};

My code is now a whole lot cleaner and smaller too. I will soon post it on Github so stay tuned! Here is, for example, how I would go about deleting a whole bunch of products:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
const moltin = require('moltin')({
publicId: process.env.MOLTIN_PUBLIC_ID,
secretKey: process.env.MOLTIN_SECRET_KEY
});
const moltin_p = require('./promisify-moltin')(moltin);

moltin_p.Product.List(null)
.then((products) => Promise.all(products.map(p => {
console.log('Requesting a delete of %s', p.title);
return moltin_p.Product.Delete(p.id);
})))
.then((result) => {
console.log('Deleted %s products', result.length);
})
.catch((error) => {
console.error(error);
});

Sequencing Asynchronous Calls in JavaScript

I am playing with Moltin for my upcoming talk on the API Strategy conference and it generates all kinds of blog posts ideas.

Context

Moltin is the API-first (or I would even argue the API-only) commerce platform. A “new kid on the block”, a recent Y Combinator graduate with a little more than $2M in seed funding. My talk is about cognitive APIs and smarter apps and I will be using a conversational e-commerce chatbot as an example. I picked Moltin as my commerce backend because it’s ridiculously easy to get started with, requires no upfront setup, and seems to provide a simple and yet a rich API that covers all my scenarios. Plus, their free tier gives me 30,000 requests per month.

You can’t transact with a commerce platform if it doesn’t have a product catalog. Products have variants (e.g. a t-shirt can come in different sizes and different colors) and different commerce platforms approach setting up this hierarchy differently. In Moltin, you first create a main product. Then you add modifiers (in my case - color and size). And then you add variations for each modifier. Moltin will then create the actual variants matrix behind the scenes. If, for example, you add blue, red, and white variations to the color modifier and S, M, L to the size modifier, you will end up with a nine total variations (every size available in every color).

Moltin APIs

Moltin APIs are lean HTTP endpoints that understand application/x-www-form-urlencoded and multipart/form-data and return back JSON. The team also supplies lightweight wrappers for JavaScript, Python, and other languages. Here’s, for example, how the creation of a variation in JavaScript looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
const moltin = require('moltin')({
publicId: process.env.MOLTIN_PUBLIC_ID,
secretKey: process.env.MOLTIN_SECRET_KEY
});

moltin.Authenticate(function() {
moltin.Variation.Create(productId, modifierId, {
title: value
},
function(result){
// result is the successfully created variation
},
function(error, details) {
// oops
});
});

I am scripting the creation of the product catalog using Adventure Works as my sample dataset so I need to run a lot of these asynchronous callback style request in order. I do it with Promises:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
const addVariation = (productId, modifierId, value) => {
return new Promise((resolve, reject) => {
moltin.Authenticate(function() {
moltin.Variation.Create(productId, modifierId, {
title: value
},
function(result){
resolve(result);
},
function(error, details) {
reject(details);
});
});
});
};

And now I can chain all my actions with .then().

Problem

I faced an interesting challenge as I was creating the variations for my products. Here’s how I do it. First, I figure out what modifiers I need to create and then for each modifier I collect the values. The result looks something like this:

1
2
3
4
5
6
7
const mods = [{
title: 'color',
values ['red', 'blue', 'white']
}, {
title: 'size',
values ['S', 'M', 'L']
}];

Now I can recursively .map() this structure into an array of Promises each creating a required variation in Moltin:

1
2
3
4
5
6
7
8
9
10
// somewhere on the chain
.then((mods) => {
return Promise.all(_.flatMap(mods, mod => {
return mod.values.map(value => {
return new Promise((resolve, reject) => {
// create the variation in Moltin
});
});
}));
})

“Looking good”, I thought, until I found out that creating all the variations asynchronously in no particular order and maybe even in parallel confuses the logic on the Moltin side that creates the product matrix (details).

Since I can’t trigger a matrix rebuild via the API, the solution was to sequence the variants creation.

Solution

Instead of just mapping the modifiers to a list of Promises running somewhat concurrently, I I needed to chain variants creation one after another. I also needed to collect all created variations into a list for the next step in the bigger chain.

Nested reduce to the rescue. First, the addVariation now keeps track of the results:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
const addVariation = (productId, modifierId, value, bag) => {
return new Promise((resolve, reject) => {
moltin.Authenticate(function() {
moltin.Variation.Create(productId, modifierId, {
title: value
},
function(result){
bag.push(result)
resolve(result);
}
// error handling skipped
});
});
};

And the Promie.all() has to convert into a linear chain of promises each creating a single variation:

1
2
3
4
5
6
7
8
9
10
11
12
// somewhere on the chain
.then(mods => {
var variations = [];

const chained = mods.reduce((chain, mod) => {
return mod.values.reduce((chain, value) => {
return chain.then(() => addVariant(productId, mod, value, variations));
}, chain);
}, Promise.resolve());

return chained.then(() => Promise.resolve(variations));
})

Works great but I feel like it can be cleaner with Rx. If you know how to convert this to observables and not explicitly manage the collection of created variants, please drop me a line. Thanks!