Visions In Code

I remember some years ago when responsive websites was new and fresh. Today are almost every website responsive.
Why not put a new dimension to responsive websites – How about make the website respond to the visitors emotions.

If the visitor is sad, why not cheer her up with happy content.

Or an angry user, why not ask him what is wrong.

Is it even possible? It sure is. 🙂

I got the idea from Microsoft’s VERY cool Cognitive Services, where they have this nice emotion service. There is a demo but also an SDK you can download. The service works like this: You upload an image(face), stream or URL. Then it will identify emotions from the image. Each emotion has a float value.

“anger”: 0.00300731952,
“contempt”: 5.14648448E-08,
“disgust”: 9.180124E-06,
“fear”: 0.0001912825,
“happiness”: 0.9875571,
“neutral”: 0.0009861537,
“sadness”: 1.889955E-05,
“surprise”: 0.008229999

Then I came up with the idea: What if we could get the image in real-time when a user is browsing the website. We use the webcam to get a picture and then upload it to the emotion service.

The computer/device must have a webcam but hey, most of them have that already.

Lets dive in.

1. Detect and capture face in an image by using webcam – Detect and grab face demo.
2. Read face image and get emotions by using Microsoft Cognitive Services
3. Register emotion on visitor(contact) using Sitecore xDB
4. Show content depending on visitors emotions by using personalization rules engine in Sitecore

1. Detect and capture face in an image by using webcam

We need to detect the visitors face and take a snapshot of it. This will be done in javascript. (I have been following WebRtc for a while now and found some very good examples)

Taking a picture is easy, the tricky part is to detect a face. But thanks to paulrhayes.com Experiments, we can do that.

var Habitat = Habitat || {};

jQuery(document).ready(function () {
    Habitat.EmotionAware.DomReady();
});

Habitat.EmotionAware = {

    DomReady: function () {
        this.CreateImageStreamFromVisitor(function (faceStream) {
            console.log("Registering emotion");
            jQuery.ajax(
             {
                 url: "/api/EmotionAware/RegisterEmotion",
                 method: "POST",
                 data: {
                     emotionImageStream: faceStream,
                     pageUrl: window.location.pathname
                 },
                 success: function (data) {
                     console.log(data.Message);
                 }
             });

        });

    },

    CreateImageStreamFromVisitor: function (callback) {

        window.URL || (window.URL = window.webkitURL || window.msURL || window.oURL);

        var video = document.createElement('video'),
                    canvas = document.createElement('canvas'),
                    context = canvas.getContext('2d'),
                    localMediaStream = null,
                    snap = false;

        var mediaConstraints = { audio: false, video: { width: 400, height: 320 } }
        
        //Here we start the video
        navigator.mediaDevices.getUserMedia(mediaConstraints).then(successCallback).catch(errorCallback);

        canvas.setAttribute('width', 400);
        canvas.setAttribute('height', 320);

        video.setAttribute('autoplay', true);

        //Here we are processing the video feed/stream
        function successCallback(stream) {
            video.src = (window.URL && window.URL.createObjectURL) ? window.URL.createObjectURL(stream) : stream;
            localMediaStream = stream;
            processWebcamVideo();
        }

        function errorCallback(error) {
            console.log(error.message);
        }

       
        function snapshot(faces) {
            if (!faces) {
                return false;
            }

            for (var i = 0; i < faces.length; i++) {
                var face = faces[i];

                //If a face is detected in the image, we will create a stream of the image
                if (localMediaStream) {
                    callback(canvas.toDataURL('image/jpeg', 0.5).substring(canvas.toDataURL('image/jpeg', 0.5).lastIndexOf(',')+1));
                    return true;
                }
            }

            return false;
        }

        //Here it will take a "picture" and try to find a face, if not then let's continue the loop
        function processWebcamVideo() {

            var startTime = +new Date(),
                faces;

            context.drawImage(video, 0, 0, canvas.width, canvas.height);

            faces = detectFaces();

            if (!snap) {
                snap = snapshot(faces);
            }

            // Log process time
            console.log(+new Date() - startTime);

            // And repeat.
            if (!snap) {
                setTimeout(processWebcamVideo, 50);
            } else {
                localMediaStream.stop();
            }
        }
        
        //The method will detect face/s on in the image
        //ccv is in a seperate js file 
        function detectFaces() {
            return ccv.detect_objects({ canvas: (ccv.pre(canvas)), cascade: cascade, interval: 2, min_neighbors: 1 });
        }
    }
}

I did a callback function that will return an image stream(stringbase64) and do an AJAX call to the emotion service.

Try it out yourself 🙂 – Detect and grab face demo

2. Read face image and get emotions by using Microsoft Cognitive Services

A typical api service where the controller will receive the image stream.

namespace Habitat.EmotionAware.Controllers
{
  using System.Threading.Tasks;
  using System.Web.Mvc;
  using Habitat.EmotionAware.Services;
  using Habitat.Framework.ProjectOxfordAI.Enums;


  public class EmotionAwareController : Controller
  {

    private readonly IEmotionImageService emotionImageService;
    private readonly IEmotionAnalyticsService emotionAnalyticsService;


    public EmotionAwareController() : this(new EmotionImageService(), new EmotionAnalyticsService())
    {
    }

    public EmotionAwareController(IEmotionImageService emotionImageService, IEmotionAnalyticsService emotionAnalyticsService)
    {
      this.emotionImageService = emotionImageService;
      this.emotionAnalyticsService = emotionAnalyticsService;
    }


    [HttpPost]
    public ActionResult RegisterEmotion(string emotionImageStream, string pageUrl)
    {
      if (string.IsNullOrWhiteSpace(emotionImageStream))
        return this.Json(new { Success = false, Message = "No image was received" });

      Emotions emotion = Task.Run(() => this.emotionImageService.GetEmotionFromImage(emotionImageStream)).Result;

      if (emotion == Emotions.None)
        return this.Json(new { Success = false, Message = "No emotion was detected" });

      this.emotionAnalyticsService.RegisterEmotionOnCurrentContact(emotion);

      this.emotionAnalyticsService.RegisterGoal(emotion.ToString(), pageUrl);

      return this.Json(new { Success = true, Message = emotion.ToString() });
    }


  }
}

When the controller was callled from the AJAX function it just got stuck. I could not understand why until I realized that the async task was in a deadlock. I tried by adding ConfigureAwait(false) according to Stephen Cleary’s post – Don’t Block on Async Code but no luck. I finally found the solution, I just had to wrap the call in a task 🙂
Task.Run(() => this.emotionImageService.GetEmotionFromImage(emotionImageStream)).Result;

The EmotionImageService just wraps the “Framework service” – EmotionsService (Sitecore Habitat way).

namespace Habitat.EmotionAware.Services
{
  using System;
  using System.IO;
  using System.Threading.Tasks;
  using Habitat.Framework.ProjectOxfordAI.Enums;
  using Habitat.Framework.ProjectOxfordAI.Services;
  using System.Collections.Generic;
  using System.Linq;
  using Habitat.EmotionAware.Models;
  using Habitat.EmotionAware.Repositories;
  using Sitecore.Diagnostics;

  public class EmotionImageService : IEmotionImageService
  {
    private readonly IEmotionAwareSettingsRepository emotionAwareSettingsRepository;

    private readonly EmotionAwareSettings emotionAwareSettings;

    public EmotionImageService(IEmotionAwareSettingsRepository emotionAwareSettingsRepository)
    {
      this.emotionAwareSettingsRepository = emotionAwareSettingsRepository;
    }

    public EmotionImageService() : this(new EmotionAwareSettingsRepository())
    {
      emotionAwareSettings = emotionAwareSettingsRepository.Get();
    }

    public async Task<Emotions> GetEmotionFromImage(string stringBase64Image)
    {

      if (this.emotionAwareSettings == null || string.IsNullOrWhiteSpace(emotionAwareSettings.SubscriptionKey))
      {
        Log.Error("SubscriptionKey is missing for emotion service", typeof(EmotionImageService));
        return Emotions.None;
      }

      IEmotionsService emotionsService = new EmotionsService(emotionAwareSettings.SubscriptionKey);

      MemoryStream faceImage = new MemoryStream(Convert.FromBase64String(stringBase64Image));

      IDictionary<Emotions, float> emotionRanksResult = await emotionsService.ReadEmotionsFromImageStreamAndGetRankedEmotions(faceImage);

      if (emotionRanksResult == null)
        return Emotions.None;

      return emotionRanksResult.ElementAt(0).Key;

    }
  }
}

The service will return an Emotion enum.

namespace Habitat.Framework.ProjectOxfordAI.Enums
{
    public enum Emotions
    {
        None,
        Anger,
        Contempt,
        Disgust,
        Fear,
        Happiness,
        Neutral,
        Sadness,
        Surprise
    }
}

Here is the “Framework service” – EmotionsService. The method ReadEmotionsFromImageStreamAndGetRankedEmotions will call the “SDK service” – EmotionServiceClient. The stream will then be analyzed and return a number of emotion states(Emotion) where each emotion state has a value(rank).

namespace Habitat.Framework.ProjectOxfordAI.Services
{
  using System;
  using System.Collections.Generic;
  using System.IO;
  using System.Linq;
  using System.Reflection;
  using System.Threading.Tasks;
  using Habitat.Framework.ProjectOxfordAI.Enums;
  using Habitat.Framework.ProjectOxfordAI.Models;
  using Microsoft.ProjectOxford.Emotion;
  using Microsoft.ProjectOxford.Emotion.Contract;

  public class EmotionsService : IEmotionsService, IDisposable
  {
    private readonly string subscriptionKey;

    public EmotionsService(string subscriptionKey)
    {
      this.subscriptionKey = subscriptionKey;
    }


    public async Task<IDictionary<Emotions, float>> ReadEmotionsFromImageStreamAndGetRankedEmotions(Stream imageStream)
    {
      EmotionServiceClient emotionServiceClient = new EmotionServiceClient(this.subscriptionKey);

      Emotion[] emotions = await emotionServiceClient.RecognizeAsync(imageStream).ConfigureAwait(false);

      Emotion emotion = emotions.FirstOrDefault();

      if (emotion == null)
        return null;

      return this.CalculateAndRankScoreToDictionary(emotion.Scores);

    }

    private IDictionary<Emotions, float> CalculateAndRankScoreToDictionary(Scores emotionScores)
    {
      if (emotionScores == null)
        return null;

      IDictionary<Emotions, float> emotionRankingDictionary = new Dictionary<Emotions, float>();

      foreach (PropertyInfo prop in emotionScores.GetType().GetProperties())
      {
        for (int index = 0; index < Enum.GetNames(typeof(Emotions)).Length; index++)
        {
          string emotionName = Enum.GetNames(typeof(Emotions))[index];

          if (!prop.Name.Contains(emotionName))
          {
            continue;
          }

          Emotions parsedEmotion = Emotions.None;

          Enum.TryParse(emotionName, out parsedEmotion);


          emotionRankingDictionary.Add(parsedEmotion, (float)prop.GetValue(emotionScores));

        }
      }

      //Sort and set highest value on top
      return emotionRankingDictionary.OrderByDescending(x => x.Value).ToDictionary(x => x.Key, x => x.Value);
    }

  }
}

In method CalculateAndRankScoreToDictionary I wanted to make it easier by just return enums with float values. Then I simply order the emotion states by the highest value and returns the winner 🙂

3. Register emotion on visitor(contact) using Sitecore xDB

After we have received the emotion we will register it on the visitor. I really like using the visitor tags, I know I should use facets but it is so easy working with tags 🙂
I simply add a tag called emotion and sets the “emotion value”.

namespace Habitat.EmotionAware.Services
{
  using Habitat.EmotionAware.Repositories;
  using System.Collections.Generic;
  using System.Linq;
  using Habitat.Framework.ProjectOxfordAI.Enums;
  using Sitecore.Analytics;
  using Sitecore.Analytics.Data.Items;
  using Sitecore.Analytics.Model;
  using Sitecore.Analytics.Model.Entities;
  using Sitecore.Analytics.Tracking;
  using Sitecore.Data.Items;
  using Sitecore.Diagnostics;

  public class EmotionAnalyticsService : IEmotionAnalyticsService
  {

    private readonly IAnalyticsRepository analyticsRepository;
    private readonly List<Item> emotionGoals;

    private const string GoalCategoryFolderEmotions = "Emotions";

    public EmotionAnalyticsService() : this(new AnalyticsRepository())
    {
      this.emotionGoals = this.analyticsRepository.GetGoalsByCategoryFolder(GoalCategoryFolderEmotions).ToList();
    }

    public EmotionAnalyticsService(IAnalyticsRepository analyticsRepository)
    {
      this.analyticsRepository = analyticsRepository;

    }

    private bool IsTracking()
    {
      if (!Tracker.IsActive)
      {
        Tracker.StartTracking();
      }

      if (Tracker.Current != null && Tracker.Current.Interaction != null && Tracker.Current.Interaction.CurrentPage != null)
      {
        return true;
      }

      Log.Warn("Tracker.Current == null || Tracker.Current.Interaction.CurrentPage == null", typeof(EmotionAnalyticsService));
      return false;
    }

    public ITagValue RegisterEmotionOnCurrentContact(Emotions emotion)
    {

      if (!this.IsTracking())
        return null;

      //Register the emotion on the contact
      if (Sitecore.Analytics.Tracker.Current.Contact.Tags.Find("Emotion") != null)
        return Sitecore.Analytics.Tracker.Current.Contact.Tags.Set("Emotion", emotion.ToString());
      else
        return Sitecore.Analytics.Tracker.Current.Contact.Tags.Add("Emotion", emotion.ToString());

    }

    public PageEventData RegisterGoal(string goalName, string pageUrl)
    {

      if (!this.IsTracking())
        return null;

      IPageContext pageContext = Sitecore.Analytics.Tracker.Current.Interaction.GetPages().LastOrDefault(page => page.Url.Path.Equals(pageUrl));

      if (pageContext == null)
      {
        Log.Warn($"Page {pageUrl} does not exist", typeof(EmotionAnalyticsService));
        return null;
      }

      Item goalItem = this.emotionGoals.FirstOrDefault(goal => goal.Name.Equals(goalName));

      if (goalItem == null)
      {
        Log.Warn($"Goal {goalName} does not exist", typeof(EmotionAnalyticsService));
        return null;
      }

      PageEventItem pageEvent = new PageEventItem(goalItem);

      return pageContext.Register(pageEvent);

    }
  }
}

Bonus! Why not register a goal on the page. It will be cool to find out what a visitor feels about the page.

From the Visit Detail Report, we have an example from “emotion” goals from a visitor.

Here is the “Goal and Events” from the Home page:

4. Show content depending on visitors emotions by using personalization rules engine in Sitecore

Now we have registered the emotion on the visitor. Let us use the personalization rule engine to determine what content should be presented.

By using tags we don’t need to create a custom rule, we just use a standard rule.

Here is my solution on the github – https://github.com/GoranHalvarsson/Habitat/tree/EmotionAware

I used the framework Sitecore Habitat. I really like the idea of breaking it down into components. Everything is clean. Eldblom and his team are doing a great job. There are also some great videos you can check out.

This was really fun to make and I see a lot of interesting things you can do with Emotion-Aware Computing. Not only for the visitors but also for the editors. Why not register the editors emotions while they are using Sitecore 😉

That’s all for now folks 🙂