Visions In Code

I think it’s really cool that Sitecore is using MongoDB for storing visitor data(The data is being stored when the session ends)
Adam Conn explains it well(as always) in his post, INTRODUCING THE SITECORE ANALYTICS INDEX

But would it not be cool to make a snapshot of your visitors in Real-Time? Like the good old ant farm 🙂

Real-Time for me is the use of WebSockets and here is a great post/slide presentation that explains it very well.

As usually there are some great frameworks you can use – my favorites are Pubnub and Microsoft’s SignalR – I went for the SignalR.

I also wanted to store the “real-time” data and MongoDB is indeed perfect for that, especially now with the latest MongoDB .NET Driver which is completely async.

The idea is to show what the user is doing right now and store that data in a database. I don’t want to store the visitors history data, Sitecore is doing that so well.

I will create a visitor object which will contain Sitecore’s Contact Id, some metadata and the current connections that the user is having on the website.
A connection is a page view, for instance if the user has 5 pages opened in his/her browser – that means 5 opened connections.
When the user is leaving a page the connection will be removed from the visitor object and if the user is leaving the website and have no opened connections, the visitor object will be removed from the database.

It’s all about here and now!

Here is how the visitor object will be stored in MongoDB.

So this is what I would like to to:
1. Setup SignalR and make it work with Sitecore
2. Create a real-time connection between visitors and website
3. Gather visitor data and send it to the website in real-time
4. Extract and deserialize the client data
5. Store the data in MongoDB in real-time (Like a backplane for SignalR)
6. Send data back to the client
7. Client leaves website

Let’s do it!

Setup SignalR

Here are some great tutorial on how to setup the SignalR:
Getting Started with SignalR 2
Getting Started with SignalR 2 and MVC 5

I had to do some minor tweaks/changes in order to make it work properly:

• The request validation feature in ASP.NET provides a certain level of default protection against cross-site scripting (XSS) attacks. In ASP.NET 4, by default, request validation is enabled for all requests. As a result, request validation errors might now occur for requests that previously did not trigger errors. To revert to the behavior of the ASP.NET 2.0 request validation feature add the following attribute in the httpRuntime:

   <httpRuntime requestValidationMode="2.0">
  

• Websocket error: If this request has failed with error 500 it means that the server does not support websockets. In this specific case server means the server (IIS) supports WebSockets, but ASP.NET version of your application which hosts SignalR does not support websockets. In the webconfig set targetframework to 4.5.

  <httpRuntime  targetFramework="4.5">
  

• We also need to do some minor tweaks to make it play with Sitecore.
  The problem is that Sitecore will actually try to handle the URL pattern in their routes instead of allowing the proxy to take action. You will want to make sure that your CMS has an ignore specified for the SignalR. In the web.config, update your IgnoreUrlPrefixes:

  <setting name="IgnoreUrlPrefixes" value="/sitecore/default.aspx|/trace.axd|...../signalr|/signalr/hubs" />
  

Next thing will be to create a working connection.


Create a real-time connection

First we need to create the hub, it needs to derive from the Microsoft.AspNet.SignalR.Hub class and the HubName attribute specifies how the Hub will be referenced in the JavaScript.
Never mind the IRealTimeVisitorRepository and IGeoCoordinateRepository, I will come back to them later.

[HubName("RealTimeConnector")]
public class RealTimeHub : Hub
{
    private readonly IRealTimeVisitorRepository _realTimeVisitorRepository;
    private readonly IGeoCoordinateRepository _geoCoordinateRepository;
    private readonly IHubContextService _hubContextService;

    public RealTimeHub(IRealTimeVisitorRepository realTimeVisitorRepository,
        IGeoCoordinateRepository geoCoordinateRepository,
        IHubContextService contextService)
    {
        this._realTimeVisitorRepository = realTimeVisitorRepository;
        this._geoCoordinateRepository = geoCoordinateRepository;
        this._hubContextService = contextService;
    }

    public override Task OnConnected()
    {
        this.Clients.Caller.onWhoIs();
        return base.OnConnected();
    }
}

When the hub is up and connected it will call the JavaScript method onWhoIs.

The IHubContextService contains the HubCallerContext(which is the context that is relative to the current request)

public class HubContextService : IHubContextService
{
    private readonly HubCallerContext _hubContext;

    private HubContextService(HubCallerContext hubContext)
    {
        this._hubContext = hubContext;
    }

    public HubContextService()
    {}

    public HubContextService Resolve(HubCallerContext callerContext)
    {
        return new HubContextService(callerContext);
    }

    public string ContactId
    {
        get
        {
            string value = GetCookieValue("SC_ANALYTICS_GLOBAL_COOKIE");
            return value.Substring(0,value.IndexOf("|", StringComparison.Ordinal));
        }
    }

    public string ConnectionId
    {
        get { return this._hubContext.ConnectionId; } 
    }

    public string Headers
    {
        get { return this._hubContext.Request.Headers["User-Agent"]; }
    }

    private string GetCookieValue(string cookieKey)
    {
        Cookie cookieValue;
        this._hubContext.RequestCookies.TryGetValue(cookieKey, out cookieValue);

        return cookieValue == null ? string.Empty : cookieValue.Value;
    }
    
}

Since it is the current request we can get the SC_ANALYTICS_GLOBAL_COOKIE and obtain the ContactId. This will be used when we store the data in MongoDB.
The ConnectionId is unique for that specific connection which will also be stored in the MongoDB.

To make the hub work we need to add in the Startup class and mark it with OwinStartup(SignalR requires Owin).

[assembly: OwinStartup(typeof(Sitecore8MVC.Web.Startup))]
namespace Sitecore8MVC.Web
{
    class Startup
    {
        public void Configuration(IAppBuilder appBuilder)
        {
            GlobalHost.HubPipeline.AddModule(new ErrorHandlingHubPipelineModule());

            var container = new Container();
            container.RegisterSingle<IHubContextService, HubContextService>();
            container.RegisterSingle<IGeoCoordinateRepository, GeoCoordinateRepository>();
            container.RegisterSingle<IRealTimeVisitorRepository, RealTimeVisitorRepository>();
            
            HubConfiguration config = new HubConfiguration
            {
                EnableJSONP = true, 
                Resolver = new SignalRSimpleInjectorDependencyResolver(container)
            };

           ConfigureSignalR(appBuilder, config);
        }

        public static void ConfigureSignalR(IAppBuilder app, HubConfiguration config)
        {
            app.MapSignalR(config);
        }
    }
}

I’m using SimpleInjector and the class SignalRSimpleInjectorDependencyResolver works like the SimpleInjectorDependencyResolver, it needs to derive from the Microsoft.AspNet.SignalR.DefaultDependencyResolver.

The ErrorHandlingHubPipelineModule is for tracing “hub” errors.

public class ErrorHandlingHubPipelineModule : HubPipelineModule
{
    protected override void OnIncomingError(ExceptionContext exceptionContext,
        IHubIncomingInvokerContext invokerContext)
    {
        Log.Error(String.Format("Error accessing hub {0}",exceptionContext.Error.Message), this );
            
        if (exceptionContext.Error.InnerException != null)
            Log.Error(String.Format("Error accessing hub => Inner Exception {0}", exceptionContext.Error.InnerException.Message), this);
          
        base.OnIncomingError(exceptionContext, invokerContext);
    }
}

Let’s take a look at the javascript.


Gather visitor data

Here is the JavaScript that communicates with the hub.

var RealTime = RealTime || {};

jQuery(document).ready(function () {
    RealTime.Connector.DomReady();
});

RealTime.Connector = {
    DomReady: function () {
        RealTime.Connector.Init();
    },
    Init: function () {

        if (jQuery("body").data("isinpageeditor").toLowerCase() === "true")
            return;

        var connection = $.hubConnection();

        var realTimeConnector = connection.createHubProxy("RealTimeConnector");

        realTimeConnector.on("onWhoIs", function () {

            var jsonObject = RealTime.Connector.GetUserMetaData();

            realTimeConnector.invoke("SendClientMetaData", jsonObject).done();
        });

        realTimeConnector.on("onWhereIs", function () {
            RealTime.ClientTracker.TrackPerRequest(function (position) {

                if (!position.coords)
                    return;

                var locationUserObject = {};
                locationUserObject["Coordinates"] = RealTime.ClientTracker.StringFormat("{0},{1}", position.coords.latitude, position.coords.longitude, "1");

                var jsonObject = {};
                jsonObject["Container"] = locationUserObject;

                realTimeConnector.invoke("sendClientLocationData", jsonObject).done();
            });
        });

        realTimeConnector.on("onSetClientData", function (text) {
            $("#geoData").text(text);
        });

        connection.start().done();
    },
    GetUserMetaData: function () {

        var baseUserObject = {};
        baseUserObject["Language"] = jQuery("body").data("currentlanguage");
        baseUserObject["SiteName"] = jQuery("body").data("currentsite");
        baseUserObject["IpAddress"] = jQuery("body").data("currentipaddress");
        baseUserObject["PageUrl"] = window.location.pathname;

        var jsonObject = {};
        jsonObject["Container"] = baseUserObject;

        return jsonObject;
    }

}

When method onWhoIs is called it will gather the current user data(by calling the GetUserMetaData method) and send it back(to the hub) through method SendClientMetaData.


Extract and deserialize

So we are back to the hub, where method SendClientMetaData was called from the client.

public async Task SendClientMetaData(Object jsonData)
{
    KeysAndValuesContainer visitorMetaDataContainer = JsonConvert.DeserializeObject<KeysAndValuesContainer>(jsonData.ToString());

    RealTimeVisitor currentUser = await this._realTimeVisitorRepository.Get(Context) ??
                                    await this._realTimeVisitorRepository.Create(Context, visitorMetaDataContainer);

    await this._realTimeVisitorRepository.UpdateMetaData(Context, visitorMetaDataContainer);

    if (currentUser.RealTimeConnections.All(conn =>
        conn.ConnectionId != this._hubContextService.Resolve(Context).ConnectionId))
        await this._realTimeVisitorRepository.AddConnection(Context);

    //Get geodata
    this.Clients.Caller.onWhereIs();
}

KeysAndValuesContainer is just a POCO class holding the visitors meta data(deserialized json we got from the client).

RealTimeVisitor contains the visitor data and it’s current connections.

public class RealTimeVisitor
{
    [BsonRepresentation(BsonType.ObjectId)]
    public string Id { get; set; }
    public string ContactId { get; set; }
    public IEnumerable<RealTimeConnection> RealTimeConnections { get; set; }
    public RealTimeMetaData RealTimeMetaData { get; set; }
    public DateTime CreatedDate { get; set; }
}

public class RealTimeConnection
{
    public string ConnectionId { get; set; }
    public string UserAgent { get; set; }
    public DateTime ConnectedAt { get; set; }
    [BsonElement("loc")]
    public GeoJson GeoCoordinates { get; set; }
}

public class RealTimeMetaData
{
    public Dictionary<string, string> MetaData { get; set; }
    [BsonElement("loc")]
    public GeoJson GeoCoordinates { get; set; }
    public DateTime LastUpdateDate { get; set; }
}

public class GeoJson
{
    public string Type { get; set; }
    public Double[] Coordinates { get; set; }
}

RealTimeConnection and RealTimeMetada will have the visitors coordinates and if we want to use MongoDB’s very cool feature “Geospatial Queries” we need to name the GeoCoordinates as “loc” in MongoDB.


Store the data in MongoDB

First we need to create a connection string for our new MongoDB database in the ConnectionStrings.config. Lets call it signalr 🙂
Best would be to have the MongoDB in the cloud, why not in Azure.

 <add name="signalr" connectionString="mongodb://localhost:27017/Sitecore8_signalr" />

Recently MongoDB released the 2.0 .NET Driver with Async support, how cool is that! I wanted of course to use that but Sitecore is using the previous version of the driver, so what to do? After some searching I found the solution – we just have to put the older MongoDB assemblies in a separate folder and in the webconfig update the dependent assemblies in the runtime node:

<dependentAssembly>
  <assemblyIdentity name="MongoDB.Bson" publicKeyToken="f686731cfb9cc103" culture="neutral"/>
  <codeBase version="2.0.0.828" href="bin\MongoDB.Bson.dll"/>
  <codeBase version="1.8.3.9" href="bin\Sitecore\MongoDB.Bson.dll"/>
</dependentAssembly>
<dependentAssembly>
  <assemblyIdentity name="MongoDB.Driver" publicKeyToken="f686731cfb9cc103" culture="neutral"/>
  <codeBase version="2.0.0.828" href="bin\MongoDB.Driver.dll"/>
  <codeBase version="1.8.3.9" href="bin\Sitecore\MongoDB.Driver.dll"/>
</dependentAssembly>

Now we can use the very cool stuff in the latest driver and Sitecore will still work with the previous version.

Anders Laub’s brilliant post Working with custom MongoDB collections in Sitecore 8 using WebApi helped me a lot and gave me the idea how to store the data in MongoDB.

The RealTimeVisitorRepository will handle the RealTimeVisitor in MongoDB.

public class RealTimeVisitorRepository : IRealTimeVisitorRepository
{
    private const string CollectionName = "RealTimeVisitor";

    private readonly IMongoCollection<RealTimeVisitor> _realTimeVisitorCollection;
    private readonly IHubContextService _hubContextService;

    public RealTimeVisitorRepository(IHubContextService hubContextService)
    {
        this._realTimeVisitorCollection = GetCollection<RealTimeVisitor>(ConfigurationManager.ConnectionStrings["signalr"].ConnectionString, CollectionName);

        this._hubContextService = hubContextService;
    }

    public IMongoCollection<RealTimeVisitor> Collection()
    {
        return this._realTimeVisitorCollection;
    }

    public async Task<RealTimeVisitor> Get(string id)
    {
        return await this._realTimeVisitorCollection.Find(visitor => visitor.Id == id).FirstOrDefaultAsync();
    }

    public async Task<RealTimeVisitor> Get(HubCallerContext hubContext)
    {
        return await this._realTimeVisitorCollection.Find(visitor => visitor.ContactId == this._hubContextService.Resolve(hubContext).ContactId).FirstOrDefaultAsync();
    }

    public async Task<bool> Delete(string id)
    {
        DeleteResult deleteResult = await this._realTimeVisitorCollection.DeleteOneAsync(Builders<RealTimeVisitor>.Filter.Eq(v => v.Id, id));

        return deleteResult.IsAcknowledged;
    }

    public async Task<bool> AddConnection(HubCallerContext hubContext)
    {

        FilterDefinition<RealTimeVisitor> filter = GetContactIdFilter(this._hubContextService.Resolve(hubContext).ContactId);

        UpdateDefinition<RealTimeVisitor> update = Builders<RealTimeVisitor>.Update.Push(user => user.RealTimeConnections, new RealTimeConnection()
            {
                ConnectionId = this._hubContextService.Resolve(hubContext).ConnectionId,
                ConnectedAt = DateTime.Now,
                UserAgent = this._hubContextService.Resolve(hubContext).Headers,
                GeoCoordinates = new GeoJson() { Type = "Point", Coordinates = null }
            });

        return await Update(filter, update);
    }

    public async Task<bool> UpdateGeoLocation(HubCallerContext hubContext, GeoCoordinate? geoCoordinate)
    {
        if (!geoCoordinate.HasValue)
            return false;

        FilterDefinition<RealTimeVisitor> filter = GetContactIdFilter(this._hubContextService.Resolve(hubContext).ContactId);

        UpdateDefinition<RealTimeVisitor> update = Builders<RealTimeVisitor>.Update.Set("RealTimeMetaData.loc.Coordinates",
                new BsonArray { geoCoordinate.Value.Longitude, geoCoordinate.Value.Latitude });


        return await Update(filter, update);
    }

    public async Task<bool> UpdateGeoLocationOnConnection(HubCallerContext hubContext, int indexOfRealtimeConnection, GeoCoordinate? geoCoordinate)
    {
        if (!geoCoordinate.HasValue)
            return false;

        FilterDefinition<RealTimeVisitor> filter = GetContactIdFilter(this._hubContextService.Resolve(hubContext).ContactId);

        UpdateDefinition<RealTimeVisitor> update = Builders<RealTimeVisitor>.Update.Set(
            String.Format("RealTimeConnections.{0}.loc.Coordinates", indexOfRealtimeConnection),
            new BsonArray { geoCoordinate.Value.Longitude, geoCoordinate.Value.Latitude });


        return await Update(filter, update);
    }

    public async Task<bool> RemoveConnection(HubCallerContext hubContext)
    {

        UpdateDefinition<RealTimeVisitor> updateFilter = Builders<RealTimeVisitor>.Update.PullFilter(p => p.RealTimeConnections,
                                            r => r.ConnectionId == this._hubContextService.Resolve(hubContext).ConnectionId);

        UpdateResult updateResult = await
            this._realTimeVisitorCollection.UpdateOneAsync(
                user => user.ContactId == this._hubContextService.Resolve(hubContext).ContactId, updateFilter);

        return updateResult.IsAcknowledged;
    }

    public async Task<bool> UpdateMetaData(HubCallerContext hubContext, KeysAndValuesContainer metaDataContainer)
    {
        FilterDefinition<RealTimeVisitor> filter = GetContactIdFilter(this._hubContextService.Resolve(hubContext).ContactId);

        UpdateDefinition<RealTimeVisitor> update = Builders<RealTimeVisitor>.Update.Set(user => user.RealTimeMetaData.MetaData,
            metaDataContainer.Container)
            .Set(user => user.RealTimeMetaData.LastUpdateDate,
                DateTime.UtcNow);


        return await Update(filter, update);
    }

    public async Task<RealTimeVisitor> Create(HubCallerContext hubContext, KeysAndValuesContainer keysAndValuesContainer)
    {
        //Will be moved to a factory...
        RealTimeVisitor realTimeUser = new RealTimeVisitor()
        {
            ContactId = this._hubContextService.Resolve(hubContext).ContactId,
            CreatedDate = DateTime.UtcNow,
            RealTimeConnections = new List<RealTimeConnection>()
            {
                new RealTimeConnection()
                {
                    ConnectionId = this._hubContextService.Resolve(hubContext).ConnectionId,
                    ConnectedAt = DateTime.UtcNow,
                    UserAgent = this._hubContextService.Resolve(hubContext).Headers,
                    GeoCoordinates = new GeoJson() { Type = "point", Coordinates = new []{0d,0d} }
                }
            },
            RealTimeMetaData = new RealTimeMetaData()
            {
                MetaData = keysAndValuesContainer.Container,
                GeoCoordinates = new GeoJson() { Type = "point", Coordinates = new[] { 0d, 0d } },
                LastUpdateDate = DateTime.UtcNow
            }
        };

        await this._realTimeVisitorCollection.InsertOneAsync(realTimeUser);

        return await Task.Run(() => realTimeUser);
    }

    private async Task<bool> Update(FilterDefinition<RealTimeVisitor> filter, UpdateDefinition<RealTimeVisitor> update)
    {
        UpdateResult result = await this._realTimeVisitorCollection.UpdateOneAsync(filter, update);

        return result.IsAcknowledged;
    }

    private FilterDefinition<RealTimeVisitor> GetContactIdFilter(string contactId)
    {
        return Builders<RealTimeVisitor>.Filter.Eq(c => c.ContactId, contactId);
    }


    private IMongoCollection<T> GetCollection<T>(string connectionString, string collectionName) where T : class
    {
        var url = new MongoUrl(connectionString);

        return new MongoClient(url).GetDatabase(url.DatabaseName).GetCollection<T>(collectionName);
    }

}

I know there are a lot of methods but let me go through some of them.
The actual connection to the MongoDB take place in method GetCollection.

To retrieve a RealTimeVisitor you can do it by the Object Id(Generated by MongoDB) or the Contact Id(Code wise it’s so much easier then previous versions).

await _realTimeVisitorCollection.Find(visitor => visitor.Id== id).FirstOrDefaultAsync();

You can also use FilterDefinition.
Read more here – Find or Query Data with C# Driver

In the update methods I used the FilterDefinition(query) and the UpdateDefinitionBuilder:

FilterDefinition<RealTimeVisitor> filter = Builders<RealTimeVisitor>.Filter.Eq(c => c.Id, id)

UpdateDefinition<RealTimeVisitor> update = Builders<RealTimeVisitor>.Update.Set("RealTimeMetaData.loc.Coordinates",
        new BsonArray { geoCoordinate.Value.Longitude, geoCoordinate.Value.Latitude });

UpdateResult result = await this._realTimeVisitorCollection.UpdateOneAsync(filter, update);

For the update I used method UpdateOneAsyncbut there are also UpdateManyAsync and ReplaceOneAsync.
Read more here – Update Data with C# Driver

Send data back

After the data has been stored we make a new call to the client to get his/her current location.

//Get geodata
this.Clients.Caller.onWhereIs();

Here is the client method.

realTimeConnector.on("onWhereIs", function () {
    RealTime.ClientTracker.TrackPerRequest(function (position) {

        if (!position.coords)
            return;

        var locationUserObject = {};
        locationUserObject["Coordinates"] = RealTime.ClientTracker.StringFormat("{0},{1}", position.coords.latitude, position.coords.longitude, "1");

        var jsonObject = {};
        jsonObject["Container"] = locationUserObject;

        realTimeConnector.invoke("sendClientLocationData", jsonObject).done();
    });
});

The interesting part is when we got the coordinates we call the hub and send the geocoordinates.

The coordinates will be stored in MongoDB and then reversed geocoded(by Geocoding.net).

public async Task SendClientLocationData(Object jsonData)
{
   KeysAndValuesContainer visitorMetaDataContainer = JsonConvert.DeserializeObject<KeysAndValuesContainer>(jsonData.ToString());

    if (!visitorMetaDataContainer.ContainsParamkey(KeysAndValuesContainerKeys.Coordinates))
        return;

    GeoCoordinate? geoCoordinate =
        this._geoCoordinateRepository.Get(visitorMetaDataContainer.GetValueByKey(KeysAndValuesContainerKeys.Coordinates));

    bool success = await this._realTimeUserRepository.UpdateGeoLocation(Context, geoCoordinate);

    if (!success)
        return;

    string geoData = GetGeoLocationData(geoCoordinate);

    if (String.IsNullOrWhiteSpace(geoData))
        geoData = "No address";
         
    this.Clients.Caller.onSetClientData(geoData);

}

private string GetGeoLocationData(GeoCoordinate? geoCoordinate)
{
    if (!geoCoordinate.HasValue)
        return string.Empty;

    IGeocoder geocoder = new GoogleGeocoder();
    IEnumerable<Address> geoData = geocoder.ReverseGeocode(geoCoordinate.Value.Latitude, geoCoordinate.Value.Longitude);

    return geoData.First().FormattedAddress;
}

Finally we send the geodata back by calling client method onSetClientData.

Where the client function will present the data.

realTimeConnector.on("onSetClientData", function (text) {
    $("#geoData").text(text);
});

Client leaves website

Last scenario will be when the visitor leaves the website. It will trigger OnDisconnected on the hub. The current connection will be removed from the visitor object and if no more connections, the visitor will be removed from the database.

public override async Task OnDisconnected(bool stopCalled)
{
    await this._realTimeUserRepository.RemoveConnection(Context);

    RealTimeVisitor realTimeVisitor = await this._realTimeUserRepository.Get(Context);

    if (!realTimeVisitor.RealTimeConnections.Any())
        await this._realTimeUserRepository.Delete(realTimeVisitor.Id);

    await base.OnDisconnected(stopCalled);
}

I really like the idea of working with data in real-time. The cool thing with this technique is that you can also interact or even communicate with the visitors in real-time. I will try to do some more posts around the real-time subject.

That’s all for now folks 🙂

Share this:

• Twitter
• LinkedIn
• Email

Like this:

Like Loading...

Related