NetMQ + RX Streaming Data Demo App

So last time I showed you a slightly cut down demo app (well cut down based on the great work done by our friends at Adaptive, where they created the awesome demo : Reactive Trader).  The demo app we looked at last time used SignalR + RX. I also promised that there would be a part 2 of that article, where I showed how to do the same sort of thing using NetMQ  (the pure c# port of ZeroMQ the extremely popular socket library) + RX.

I am pleased to say that Doron (author of NetMQ) and I have now published this article and it is available to read at the following Url:

http://www.codeproject.com/Articles/853476/NetMQplus-RX-Streaming-Data-Demo-App-of

If you looked at the 1st part in this mini series of articles you will know that we cover things like

  • Creating a streaming API of TickerDto objects
  • We make a resilient RX stream, that may be repeated
  • We expect the client(s) to know when the server is unavailable
  • We expect the client(s) to know when the server is available again

We show all of this, just like we did with the 1st demo app (SignalR + RX), but you will also learn a few things along the way such as:

  • How to create a generic NetMQ heartbeat
  • How to create a NetMQ Pub/Sub topoloogy
  • The NetMQ Actor model
  • The NetMQ Poller
  • How to use NetMQ, which will actually put you in very good stead to use ZeroMQ in any of the other language bindings (and it has loads)
  • A few of the different types of sockets within NetMQ
  • How to actually use NetMQ in a real life project

We are hoping you will actually get quite a lot from this article. So if you do enjoy it, please leave a vote over at the codeproject web site

SignalR + RX Streaming Data Demo App

I work in finance, and at the moment work in the Foreign Exchange (FX) space, where we have lots of requirements that involve streaming of various things, such as:

  • Rates
  • Trades
  • Other financial instruments

Historically what we would have done to accomodate these requirements was to use sockets and push our own data (usually JSON or XML) down the wire, and then had some quite knarly listeners above that, where each listener would want some slightly different view of the original streaming data.

Thing is, time has never stands still in the world of software development, and new things come out all the time, literally every other day a new framework comes along that helps improve what came before it in some way or another.

One particular thing, that I think a lot of people actually misunderstood was Reactive Extensions RX, which in fairness has been around a while now. I think a lot of people think of RX as LINQ to events, which it does have. However the truth is that RX is a superb framework for creating streaming applications and the more you get into RX the more you tend to see everything as a stream, from and event, down to picking a file or a ICommand.Execute in a ViewModel say.

RX comes with many many tools in it’s arsenal, such as

  • 2 core interfaces IObservable / IObserver
  • Lots of LINQ like extension methods for IObservable
  • Time / window based operations (LINQ doesn’t really have these)
  • Concurrency schedulers
  • Stream error handling controls (Catch etc etc)
  • Ability to create IObservable from a great many other things, such as
    • IEnumerable
    • Task
    • Events
    • IObservable Factories

I personally feel that a lot of people could make use of RX to great gain, if they were to put in the time to learn about it, a bit more.

There is also another great library from Microsoft which facilitates push based technology, namely SignalR. Quite a lot has been written about SignalR, but you rarely see that from a desktop point of view.

Most people would probably associate a SignalR Hub with a web site, but there are various ways you can use SignalR, which also make it a good choice for desktop (ie non web) development. This is possible thanks to the SignalR OWIN hosting API, such that you can self host a SignalR Hub. There is also a .NET client, we can use to communicate with the hub. SignalR also has quite a lot of events within the .NET client, and guess what we can do with those events, yep that’s right we can create IObservable(s) streams out of them, which opens up the gates to use RX.

This article will show one area where RX excels, which is in the area  of streaming APIs, where we will be using RX/SignalR

 

Read the full article here : http://www.codeproject.com/Articles/851437/SignalR-plus-RX-Streaming-Data-Demo-App

Azure : Redis Cache

This is a new post in a series of beginners articles on how to do things in Azure. This series will be for absolute beginners, and if you are not one of those this will not be for you.

You can find the complete set of posts that make us this series here :

http://sachabarbs.wordpress.com/azure/

This time we will look at how to use the preview of the Azure Redis Cache.

For this one you will need access to the new Azure portal:

https://portal.azure.com/

Why Redis Cache

I have used Redis in the past when I was looking into different NoSQL options, which you can read about here :

http://sachabarbs.wordpress.com/2012/05/21/document-dbs-a-quick-look-at-some-of-them/

Redis was very easy to use an I found it very easy to store complex JSON object in it, at the time I was using the ServiceStack.Redis client libs.

Here is what Scott Guthries blog has to say about the preview Redis Cache for Azure

Unlike traditional caches which deal only with key-value pairs, Redis is popular for its support of high performance data types, on which you can perform atomic operations such as appending to a string, incrementing the value in a hash, pushing to a list, computing set intersection, union and difference, or getting the member with highest ranking in a sorted set.  Other features include support for transactions, pub/sub, Lua scripting, keys with a limited time-to-live, and configuration settings to make Redis behave more like a traditional cache.

Finally, Redis has a healthy, vibrant open source ecosystem built around it. This is reflected in the diverse set of Redis clients available across multiple languages. This allows it to be used by nearly any application, running on either Windows or Linux, that you host inside of Azure.

http://weblogs.asp.net/scottgu/azure-redis-cache-disaster-recovery-to-azure-tagging-support-elastic-scale-for-sqldb-docdb

 

Creating the Cache

image

This will launch a wizard where you can pick the name/pricing plan etc etc

It may take a few minutes to configure the cache for the first time, you can see this in the portal, as shown below

image

You may also check on the status of all your configured Azure items in the new Portal, using the BROWSE button, as shown below:

image

Once you have successfully created a Cache, you will see something like this:

image

From here you can grab the keys for your account, which we will use in the next bit of this post

Using A Created Cache

So once you have created a Redis Cache using the preview portal you will likely want to connect and use it. So lets have a look at that shall we.

We start by getting the connection details, this requires 2 steps:

1. Get the url, which is easy to get using the properties window, as shown below

image

2. Once you have copied the host name url, we need to copy the password, which you can grab from the keys area

image

Once you have these 2 bits of information, we are able to start using the Redis Azure cache.

Code Code Code

You will need to add the following NuGet package : StackExchange.Redis

NuGet command line > Install-Package StackExchange.Redis

Once you have that, its finally time to start coding, so what do you need to use the cache.

Not much as it turns out. The following is a fully working program, which stores and retrieves some basic string values, and also a fully serialized object.

NOTE : To see what types you can use with the Redis Cache read this link : http://redis.io/topics/data-types

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Newtonsoft.Json;
using StackExchange.Redis;

namespace AzureRedisCacheDemo
{
    class Program
    {
        static void Main(string[] args)
        {
            ConnectionMultiplexer connection = ConnectionMultiplexer.Connect(
                "sachabarberredistest.redis.cache.windows.net,ssl=true,password=*** YOUR ACCOUNT KEY HERE ***");

            IDatabase cache = connection.GetDatabase();

            // Perform cache operations using the cache object...
            // Simple put of integral data types into the cache
            cache.StringSet("key1", "dsyavdadsahda");
            cache.StringSet("key2", 25);

            Foo foo1 = new Foo() {Age = 1, Name = "Foo1"};
            var serializedFoo = JsonConvert.SerializeObject(foo1);
            cache.StringSet("serializedFoo", serializedFoo);

            Foo foo3 = new Foo() { Age = 1, Name = "Foo3" };
            var serializedFoo3 = JsonConvert.SerializeObject(foo1);
            cache.StringSet("serializedFoo3", serializedFoo3);


            // Simple get of data types from the cache
            string key1 = cache.StringGet("key1");
            int key2 = (int)cache.StringGet("key2");


            var foo2 = JsonConvert.DeserializeObject<Foo>(cache.StringGet("serializedFoo"));
            bool areEqual = foo1 == foo2;


            var foo4 = JsonConvert.DeserializeObject<Foo>(cache.StringGet("serializedFoo3"));
            bool areEqual2 = foo3 == foo4;


            Console.ReadLine();
        }
    }


    
    public class Foo : IEquatable<Foo>
    {
        public string Name { get; set; }
        public int Age { get; set; }

        public bool Equals(Foo other)
        {
            if (ReferenceEquals(null, other)) return false;
            if (ReferenceEquals(this, other)) return true;
            return string.Equals(Name, other.Name) && Age == other.Age;
        }

        public override bool Equals(object obj)
        {
            if (ReferenceEquals(null, obj)) return false;
            if (ReferenceEquals(this, obj)) return true;
            if (obj.GetType() != this.GetType()) return false;
            return Equals((Foo) obj);
        }

        public override int GetHashCode()
        {
            unchecked
            {
                return ((Name != null ? Name.GetHashCode() : 0)*397) ^ Age;
            }
        }

        public static bool operator ==(Foo left, Foo right)
        {
            return Equals(left, right);
        }

        public static bool operator !=(Foo left, Foo right)
        {
            return !Equals(left, right);
        }
    }
}

When you run this code you should see something like this:

image

It is a VERY simple demo, but demonstrates the cache nicely I feel.

In this example I am using JSON.Net to serialize objects to strings, which are then stored in the Redis Cache, you may have some other serializer you prefer, but this does illustrate the point of a working Redis Cache ok I feel.

 

StackExchange.Redis Pub.Sub

Redis may also be used as a pub/sub framework, which you can use as follows:

Here is a basic publisher

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Newtonsoft.Json;
using StackExchange.Redis;

namespace AzureRedisCachePublisher
{
    class Program
    {
        static void Main(string[] args)
        {
            ConnectionMultiplexer connection = ConnectionMultiplexer.Connect(
                "sachabarberredistest.redis.cache.windows.net,ssl=true,password=*** YOUR ACCOUNT KEY HERE ***");
            ISubscriber sub = connection.GetSubscriber();

            Console.WriteLine("Press a key to pubish");
            Console.ReadLine();



            sub.Publish("messages", "This is from the publisher");

            Console.ReadLine();
        }
    }

}

  And here is a basic Subscriber

 

 

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Newtonsoft.Json;
using StackExchange.Redis;

namespace AzureRedisCacheSubscriber
{
    class Program
    {
        static void Main(string[] args)
        {
            ConnectionMultiplexer connection = ConnectionMultiplexer.Connect(
                "sachabarberredistest.redis.cache.windows.net,ssl=true,password=*** YOUR ACCOUNT KEY HERE ***");
            ISubscriber sub = connection.GetSubscriber();

            sub.Subscribe("messages", (channel, message) =>
            {
                Console.WriteLine((string)message);
            });


            Console.ReadLine();
        }
    }


 }

Which when run will give you something like this:

 

image

  Anyway that is all for now, enjoy until next time.

 

 

This has barely scratched the surface of working with StackExhange.Redis, if you want to know more read the documentation here: https://github.com/StackExchange/StackExchange.Redis

PowerShell : Http Get/Post

This is another article in my on going learning/experimenting with PowerShell. This time I will show how you can use PowerShell to carry out REST operations such as GET/POST.

Now there may be some amongst you, who go why didn’t you just use WGet, which is a downloadable thing, and I could have indeed used that, except for the part that the machine I need to run this on, is so locked down that I can only use things that are already installed. So raw PowerShell it is

Here is the relevant PowerShell code, which allows 3 parameters to control the script

  • Target : The url
  • Verb : This is the http verb, GET, PUT etc
  • Content : This could be some content when doing a POST request for example

It just makes use of some very standard .NET classes namely WebRequest

[CmdletBinding()]
Param(
 
 
    [Parameter(Mandatory=$True,Position=1)]
    [string] $target,
 
    [Parameter(Mandatory=$True,Position=2)]
    [string] $verb,      
 
    [Parameter(Mandatory=$False,Position=3)]
    [string] $content
 
)
 
 
write-host "Http Url: $target"
write-host "Http Verb: $verb"
write-host "Http Content: $content"
 
 
 
$webRequest = [System.Net.WebRequest]::Create($target)
$encodedContent = [System.Text.Encoding]::UTF8.GetBytes($content)
$webRequest.Method = $verb
 
write-host "UTF8 Encoded Http Content: $content"
if($encodedContent.length -gt 0) {
    $webRequest.ContentLength = $encodedContent.length
    $requestStream = $webRequest.GetRequestStream()
    $requestStream.Write($encodedContent, 0, $encodedContent.length)
    $requestStream.Close()
}
 
[System.Net.WebResponse] $resp = $webRequest.GetResponse();
if($resp -ne $null) 
{
    $rs = $resp.GetResponseStream();
    [System.IO.StreamReader] $sr = New-Object System.IO.StreamReader -argumentList $rs;
    [string] $results = $sr.ReadToEnd();
 
    return $results 
}
else
{
    exit ''
}

Here is an example usage (assuming the above script is saved as http.ps1 somewhere):

Http.ps1 -target "http://www.google.com" -verb "GET"

Which would give the following sort of output:

image

Anyway that’s it for now, hope this helps

Azure : Blob Storage / Retrieval

This is a new post in a series of beginners articles on how to do things in Azure. This series will be for absolute beginners, and if you are not one of those this will not be for you.

You can find the complete set of posts that make us this series here :

http://sachabarbs.wordpress.com/azure/

This time we will look at how to use Azure blob storage for uploading things like files (images, word documents, whatever you like really).

Introduction To Blob Storage

What exactly is Blob Storage? Well it is actually very simple, it is Azure hosted storage that allows you to upload large amounts of unstructured data (typically binary data, so bytes) that may be shared publicly using http/https.

Typical usage may be:

  • Storing images
  • Storing documents
  • Storing videos
  • Storing music

A typical Azure blob service would make use of these components

image

Account

This would be your Azure storage account. You must create a storage account using the Azure portal, which you would do through the portal : https://manage.windowsazure.com (we will see more on this in just a minute)

Container(s)

These belong to Account, and are using to group blob(s). Each account can have unlimited number of containers, and each container may contain an unlimited number of blobs.

Blob(s)

Blobs represent the Azure hosted data that represent the originally uploaded binary data. It is the blobs that you would eventual end up loading when you share an Azure blob storage url.

There are 2 types of blob storage available

Block Blobs

These are binary blocks of up to 200GB, where you can upload 64MB at one time, Typically for a larger block, you would spit things up and upload them in chunks using multiple threads, and Azure will reassemble them making the available as a single blob.

Page Blobs

These can be up to 1TB and consist of a collection of 512 pages. You would set a maximum size when creating the page blob. I have not used these so much, and personally I think these are here to support other Azure features like Virtual Hard Drives (VHDs) which are stored as page blobs in Azure Storage.

 

Url Syntax

The actual blob url format is as shown below:

http://<storage account>.blob.core.windows.net/<container>/<blob>

 

How To Get An Azure Storage Account

The first thing you will need to do is create a storage account, this is easily achieved by using the portal. Go to the portal : https://manage.windowsazure.com, and then click new

image

Then pick storage, and go through creating a new Storage Account

image

 

Then once you have that you can open the newly created storage account, and click the dashboard, and that will show you the relevant connection strings which you may use in your application

image

Using The Storage Emulator

NOTE : If you just want to try things out without going through the process of creating a Storage Account, you can actually use the Storage Emulator, which you can do using a account connection string something like this:

<!-- TODO : This would need to change to live azure value when deployed -->
    <add key="azureStorageConnectionString" value="UseDevelopmentStorage=true;" />

Since I am writing the bulk of these posts on a train without any internet connectivity, I will be using the storage emulator in any demos in this post.

If you want to use the emulator, ensure that is running, and that you have enabled the storage emulator

image

image

 

Getting The Right NuGet Package

You will need to install this NuGet Package to work with Azure Blob Storage “WindowsAzure.Storage”

 

Creating A Container

This is done using the following sort of code

CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
    ConfigurationManager.AppSettings["StorageConnectionString"]);

CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

// get blob container
CloudBlobContainer container = blobClient.GetContainerReference("testcontainer5");
container.CreateIfNotExists();


container.SetPermissions(
new BlobContainerPermissions
{
    PublicAccess = BlobContainerPublicAccessType.Blob
});

Uploading A File To Azure Blob Storage

This is done using the following code

CloudBlockBlob cloudBlockBlob = container.GetBlockBlobReference(
    "mandrillBlobUploaedToAzure.jpg");
cloudBlockBlob.Metadata["TypeOfImage"] = "Animal";

// Create or overwrite the "myblob" blob with contents from a local file.
using (var fileStream = System.IO.File.OpenRead(@"C:\Users\User\Pictures\mandrill.jpg"))
{
    cloudBlockBlob.UploadFromStream(fileStream);
    Console.WriteLine("Blob Url : {0}", cloudBlockBlob.Uri);
}

 

Listing All Blobs In A Container

This is done using the following code

container = blobClient.GetContainerReference("sachasContainer");

// Loop over items within the container and output the length and URI.
foreach (IListBlobItem item in container.ListBlobs(null, false))
{
    if (item.GetType() == typeof(CloudBlockBlob))
    {
        CloudBlockBlob blob = (CloudBlockBlob)item;

        Console.WriteLine("Block blob of length {0}: {1}", blob.Properties.Length, blob.Uri);

    }
}

Download Blobs

This is done using this sort of code

CloudBlockBlob blockBlob = container.GetBlockBlobReference("mandrillBlobUploaedToAzure.jpg");

// Save blob contents to a file.
using (var fileStream = System.IO.File.OpenWrite(@"C:\Users\User\Pictures\XXX.jpg"))
{
    blockBlob.DownloadToStream(fileStream);
}

Blob Metadata

Blobs support metadata via a dictionary which is available using the Metadata property which is a simple key/value pair.

 

How To Add Descriptive Metadata To A Blob

This is actually very easy, you just store more data in either a SQL Azure or SQL Azure table storage where you would include the blobs Url.  Job done

 

Deleting A Blob From A Container

This is done using this sort of code

blockBlob = container.GetBlockBlobReference("mandrillBlobUploaedToAzure.jpg");

// Delete the blob.
blockBlob.Delete();

Azure : SQL Azure

This is a new post in a series of beginners articles on how to do things in Azure. This series will be for absolute beginners, and if you are not one of those this will not be for you.

You can find the complete set of posts that make us this series here :

http://sachabarbs.wordpress.com/azure/

This time we will look at how to create a new SQL server database within Azure (in a later article we will loot using Microsofts NoSQL database “DocumentDB”)

Anyway so step 1, is to open up the portal

https://manage.windowsazure.com

 

From there you can click on “SQL Databases” and choose the “Create a SQL Database” hyperlink

image

From there you need to fill in your preferences within the wizard

image

image

Once this wizard has completed you will see a new database has been created

image

IMPORTANT : When the database is created, you will need to ensure that the standard port 1433 is opened. One of the easiest ways to do that, is to use the Azure portal to query the database (even though there is no tables in the database yet)

image

This little cheat will prompt you to open up the Firewall ports, which is great, lets just let the Azure portal do this work for us

image

So once the port is open, you will be redirected to an app in the browser (Silverlight app at present), that allows you to use your connection details you chose

image

When you successfully connect you should see something like this

image

Now there is no data in the SQL database yet. We could use this Silverlight app to add some tables, and data. However I would prefer to do that in Visual Studio, so lets go back to the portal, and open the connection strings, as shown below

image

image

We are interested in the ADO .NET one, where the part I have highlighted is the important part you need

image

SO grabbing the connection address to the Azure SQL server instance, lets connect via Visual Studio, and create a table

image

Once you have a connection in Visual Studio, lets create a new table using the context menu

image

When you are happy with the table, click the “Update” button which will push the changes to Azure. This is only a demo, for a real app you would likely have some sort of scripts, or would use the Entity Framework migrations facility to manage changes

image

image

So now lets check everything worked by connecting to the Azure SQL database from SQL server management studio.

image

As we can see we see the table we just created above

image

And lets also check the Azure portal query app

image

Yep, the table looks good, there is no data there yet, as expected for a new table. So lets now turn our attention to getting some data into the new table.

Lets use a new Entity Framework model to talk to the new SQL Azure database/table we just created.

image

I went with the defaults but you can choose what you like

image

This will result in a few files being created in the demo app, such as these, as well as an entry in the App.Config file to point to the SQL Azure database instance

image

And here is some code that will do some basic CRUD operation using the Entity Framework context that was created for us.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace SQLAzureTest
{
    class Program
    {
        static void Main(string[] args)
        {

            //insert
            using (var sachaAzureSQLEntities = new SachaAzureSQLEntities())
            {
                sachaAzureSQLEntities.Orders.Add(new Order()
                {
                    //note we are spelling this wrong so we can update it later
                    Description = "Buzz Lighyear toy",
                    Quanitity = 1
                });
                sachaAzureSQLEntities.SaveChanges();



                //select
                var order = sachaAzureSQLEntities.Orders.Single(
                    x => x.Description == "Buzz Lighyear toy");
                Console.WriteLine("Order : Id: {0}, Description: {1}, Quanity {2}",
                    order.Id, order.Description, order.Quanitity);

                //update
                order.Description = "Buzz Lightyear toy";
                sachaAzureSQLEntities.SaveChanges();

                var exists = sachaAzureSQLEntities.Orders.Any(
                    x => x.Description == "Buzz Lighyear toy");
                Console.WriteLine("Buzz Lighyear toy exists :  {0}", exists);

                order = sachaAzureSQLEntities.Orders.Single(
                    x => x.Description == "Buzz Lightyear toy");
                Console.WriteLine("Order : Id: {0}, Description: {1}, Quanity {2}",
                    order.Id, order.Description, order.Quanitity);



                //delete
                sachaAzureSQLEntities.Orders.Remove(order);
                sachaAzureSQLEntities.SaveChanges();

                Console.WriteLine("Orders count :  {0}", 
                    sachaAzureSQLEntities.Orders.Count());


            }

            Console.ReadLine();
        }
    }
}

And here is the results of this against the SQL Azure instance we just created.

image

Azure : Provisioning a Virtual Machine

This is a new post in a series of beginners articles on how to do things in Azure. This series will be for absolute beginners, and if you are not one of those this will not be for you.

You can find the complete set of posts that make us this series here :

http://sachabarbs.wordpress.com/azure/

This time we will look at how to create a new Virtual Machine using the Compute element of Microsoft Azure.

This again will be quite a screen shot heavy posting (its only a couple of the posts that will be like this, a lot of the subsequent ones will be much more codey, which is probably a good thing as there is a new version of the Azure portal waiting in the wings that has a different look and feel)

Anyway so step 1, is to open up the portal

https://manage.windowsazure.com

From there you will want to create a new Virtual machine, this can be done by clicking on the “New” button in the Azure portal.

image

From here you can choose what new thing you want to create in Azure. As this post is all about Virtual Machines, we will choose to create a Virtual Machine using the Azure compute element. This can be seen in the screen shot below. Azure comes with a whole load of pre canned images where the most common ones are shown in the list. You can also choose to examine more images, which is what we will be doing in  this post. So you would use the “More images” drop down item, as shown below

image

This will take you to a wizard that allows you to choose your Virtual Machine image and setup, and username/password

image

 

image

image

image

Once you have completed the wizard a new Virtual Machine, should be listed in the Azure portal for you. This is as shown below, for this posts example.

image

You can go into the Virtual Machine and view information about it, such as its public IP address, its DNS name etc etc

image

More importantly is the “Connect” button. When you click that a new Remote Desktop item will be created and downloaded for you. You can then use that to gain access to the Virtual Machine you setup.

One word of warning though is that the Status of the Virtual Machine MUST be “Running”, the status of the Virtual Machine can be seen in the screen shot above.

image

image

Here is me connected to a Virtual Machine (a standard Windows 2008 server), that I created another time.

image

Anyway hope that helps someone, another step along the Azure highway