Using SignalR + Service Bus + Service Fabric in harmony

Recently, I was talking to a customer about their Service Fabric hosted application and they mentioned they were having difficulty getting their live dashboard working with SignalR. Immediately I thought this might be as a result of the 'persistent client connection' not being configured properly at the load balancer... unfortunately, this problem was a little more intricate.

The pattern the customer was trying to implement was a fairly simple one. A 'dashboard' service subscribed to messages on a Service Bus topic, on receiving a new message it would use SignalR to update the dashboard in real-time.

At this point, I decided to see if I could replicate the issue they were experiencing. I created a new ASP.NET Web App hosted on Service Fabric and sprinkled on a little SignalR. At this point everything worked as expected, updates on the backend were almost instantaneously manifested in the front end, perfect. Next, I provisioned a new Service Bus instance on Azure and created a test topic and client message pump to supply dummy messages. Then I needed to make sure the Service Fabric service had the correct communication listener configured to receive said messages. Luckily there is a Nuget library that handles this called ServiceFabric.ServiceBus.Services. After installing v4.3.1 of this library, I needed to add the communication listener to my service in the CreateServiceInstanceListeners method.

protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListener()  
{
    // Service Bus message receiver
    Action<string> logAction = 
    log => ServiceEventSource.Current.ServiceMessage(this.Context, log);

    return new ServiceInstanceListener[]
    {
        // Owin endpoint for ASP.NET web connections
        new ServiceInstanceListener(serviceContext => 
            new OwinCommunicationListener(Startup.ConfigureApp,
            serviceContext, 
            ServiceEventSource.Current,
            "ServiceEndpoint")),

        // Service Bus endpoint
        new ServiceInstanceListener(serviceContext =>
            new ServiceBusSubscriptionBatchCommunicationListener(
                new Handler(logAction),
                serviceContext,
                ConfigurationManager.AppSettings["TopicName"],
                ConfigurationManager.AppSettings["TopicSubName"]),
                "StatelessService-ServiceBusSubscriptionListener")
    };
}

internal sealed class Handler : AutoCompleteBatchServiceBusMessageReceiver  
{
...
}

Full demo code for using the ServiceFabric.ServiceBus.Services package is available here

After adding the ServiceInstanceListener I quickly built the code to check everything was working OK. The code compiled fine, but after I attempted to deploy the code to my local cluster I received the following runtime exception;

Could not load type 'Microsoft.ServiceBus.Messaging.MessageClientEntity' from assembly 'Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'.

After scratching my head for a few minutes and searching around, I came across the issue on the SignalR GitHub repo. It appears a breaking change between ServiceBus v2.x and v3.x has caused a few issues. I believe the exact change was the move from .NET's older Async API to .NET's updated Task based API. Either way, SignalR still has a dependency on WindowsAzure.ServiceBus v2.x for its backplane functionality and ServiceFabric.ServiceBus.Services has a dependency on v3.x and thus are incompatible within the same binary, doh!

As far as I can tell there are a couple of possible solutions to this problem. No doubt there are probably lots more which have eluded me.

My first solution, hinges on the fact that SignalR, like most of the ASP.NET components, is open source. Therefore, you can go ahead, clone the repo, update the WindowsAzure.ServiceBus reference inside Microsoft.AspNet.SignalR.ServiceBus to v3.x, fix any subsequent errors, ensure the tests pass and then repackage for your own use. Luckily, it appears somebody else has already done this and submitted a PR. It's worth noting that this particular PR was actually rejected as it appears to include a few too many changes such as also updating the SignalR.Core dependencies. Regardless, if you are happy to take on those changes, or simply remove the excessive ones, it's a simple fix to the issue.

My second solution exploits the microservices based nature of Service Fabric. Simply create a sidecar service that will pull the messages from the topic, use the built in service communication listener to then pass those messages across to your ASP.NET application to send out via SignalR. This also allows you to scale each component out individually and increases the cohesiveness of your services. However, there is a catch. No doubt you are using Service Bus in the first place as you want reliable messaging. The inter-service communication in Service Fabric will not give you the same messaging guarantees as something like Service Bus. Therefore, it will be your responsibility to implement something at the application layer, such as a send and acknowledge protocol to ensure messages aren't lost in transit. When I have time to implement this pattern I will update this post with an example of how you might achieve this with a stateful service.

If you have any other suggestions on how to mitigate this issue, then please let me know! You can tweet/DM me @dotjson.