I was playing with Azure Cognitive Services and figured that I would switch to my @epam.com account for the next prototype. I needed to re-sign-up. This is my story.

Login

Microsoft’s live.com OAuth can integrate with your ADFS for a single sign-on experience:

I opted in for my work account, authenticated, and went ahead to enable the services I needed.

Verification

To enable Cognitive Service APIs on this account I needed to activate my Azure subscription. Microsoft will challenge your identity twice.

First, it’s the code verification that you can do by sending yourself a text. My form was pre-populated with the Belarus country code (+375). I quickly dismissed it as probably an old attribute on my AD profile. You see, I lived in Minsk (Belarus) before I moved to the states a good while ago but not every enterprise system got the memo. No big deal. Typed in my cell phone and got the activation code.

Then it’s the identify verification by card:

My postal code is again pre-populated with the one from the past. This time, however, I wasn’t able to use my current address:

What do you do when an online form tries to outsmart you? I, personally, try to outsmart it back:

1
2
3
4
5
6
7
8

const rules = $('#PCSBodyForm').data('validator').settings.rules;

// -> Object {required: true, minlength: 6, maxlength: 6, pattern: "^[0-9]{6}$"}

rules.billingZipcode.minlength = 5;
rules.billingZipcode.pattern = '^[0-9]{5}$';

// -> Object {required: true, minlength: 5, maxlength: 6, pattern: "^[0-9]{5}$"}

Alright!

Way Too Smart

Guess what, Microsoft engineers are very diligent. The validation also runs server-side and the form comes back with an error:

Sigh… My US street address and the city of Marietta were gladly accepted. It’s the postal code format and length validation that failed. Why so serious? A better solution would probably be to ask for a country as part of the address form and validate against it. Or maybe trust the SSO challenge that I went through when logging in and just collect my card?

Anyway. I guess I will keep using my personal account with Azure for now and will wait for out IT to find the field in my profile that ties me to my home country.

I have used Cognitive Services from Microsoft (part 1) and IBM Watson Services (part 2) to read my avatar image. There are two more APIs that I would like to put to the test - Google Cloud Vision API and Clarifai.

Google Cloud Vision

I already had a developer account. To use the Cloud Vision API I only had to enable it in the console and generate myself a browser key. When you sign up, Google asks for your credit card but they promise not to charge it without your permission. They also give you $300 in free trial credit and 60 days to use it.

The API itself is clearly designed for extensibility.

It’s a single endpoint that can do different things based on your request. An image can either be sent as a binary data or as a URL to a Google Storage Bucket. You can send multiple images at once and every image request can ask for different type of analysis. You can also ask for more than one type of analysis for a given image.

Google can easily add new features without adding new APIs or changing the endpoint’s semantics. Take a look:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

const key = '<use-your-own-key>';
const url = `https://vision.googleapis.com/v1/images:annotate?key=${key}`;

fetch(url, {
    method: 'POST',
    headers: new Headers({
        'Content-Type': 'application/json'
    }),
    body: JSON.stringify({
        'requests': [{
            'image': {
                'source': {
                    'gcsImageUri': 'gs://pveller/pavelveller.jpeg'
                }
            },
            'features': [{
                'type': 'LABEL_DETECTION',
                'maxResults': 10
            }, {
                'type': 'FACE_DETECTION'
            }],
        }]
    })
}).then(function(response) {
    return response.json();
}).then(function({ responses }) {
    const labels = responses[0].labelAnnotations;

    console.log(labels.map((l) => `${l.description} - ${l.score.toPrecision(2)*100}%`))
});

Here’s what I got:

1
2
3
4
5
6
7
8

[ 
    "hair - 95%", 
    "person - 94%", 
    "athlete - 88%", 
    "hairstyle - 84%", 
    "male - 79%", 
    "sports - 72%"
]

A man who definitely cares about his hair, right? :) I am not sure where the sports and athlete bits came from. I also wonder if I would get more tags (like a microphone, for example) if I could ask for features with lower scores. The API doesn’t seem to allow me to lower the threshold. I asked for ten results but got only six back.

The face detection sent down a very elaborate data structure with coordinates of all the little facial features. Things like left eye, right eye, eyebrows, nose tip, and a whole lot more. The only thing is … you can’t see the left side of my face on my avatar.

Google also tries to detect emotions. Of all that it can see - anger, joy, sorrow, surprise - none came back with anything but VERY_UNLIKELY. You can also test an image for explicit content. Same VERY_UNLIKELY for my avatar.

Very pleasant experience but I honestly expected a little more from Google’s Vision API.

I expected more because I know Google does all kinds of crazy things with deep learning in their labs. With images as of two years go and very recently with video. Maybe as those models mature, the Cloud Vision will support more features? Time will tell.

Clarifai

The easiest setup experience by far!

I was ready to go in just a few seconds, no kidding! And it also felt like the fastest response from all the APIs I tried. Very easy and intuitive to use as well:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

const key = '<use-your-own-key>';
const url = 'https://api.clarifai.com/v1/tag';

const data = new FormData();
data.append('url', image);
data.append('access_token', key);

fetch(url, {
    method: 'POST',
    body:  data 
}).then(function(response) {
    return response.json();
}).then(function({ results }) {
    const tags = results[0].result.tag;
    const labels = [...tags.classes.keys()].map((i) => ({
        'class': tags.classes[i],
        'confidence': `${tags.probs[i].toPrecision(2)*100}%`
    }));

    console.log(labels.map((l) => `${l.class} - ${l.confidence}`));
});

Here’s what I got back:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

[
    "music - 100%", 
    "singer - 99%", 
    "man - 99%", 
    "people - 98%", 
    "competition - 98%", 
    "musician - 98%", 
    "one - 98%", 
    "concert - 98%", 
    "pop - 97%", 
    "microphone - 97%", 
    "portrait - 97%", 
    "journalist - 96%", 
    "press conference - 96%", 
    "wear - 95%", 
    "administration - 95%", 
    "television - 94%", 
    "stage - 94%", 
    "performance - 93%", 
    "recreation - 92%", 
    "festival - 92%"
]

This is actually very close! Good feature detection with various plausible scenarios spelled out based on that. I would only question the absolute confidence in music and singer :) What about a… conference and a speaker?

Clarifai has another very interesting endpoint - Feedback. I haven’t used it but it seems that you can submit your own labels back to Clarifai and help them train and fine-tune the model. It won’t be your own classifier like Watson does. Feedback seems to be a crowdsourcing mechanism to train their main shared model(s). I only wonder how it will work without you having to specify the area of the image that each new label is attached to. In case of my avatar, conference and speaker would attach to the whole image. What about more involved images? Maybe I am missing something…

There’s a lot more computer vision APIs out there. Some are more generic and some a geared towards more specialized tasks like visual product search or logo recognition. Go give it a try!

It’s fascinating what kinds of things are just one HTTP request away.