Azure

Azure DevOps : Setting up and pushing nuget package

So its been a while since I posted. But thought it would be good to finally add the 2nd part of this, where this time we will look at how to create and push to Azure DevOps hosted Nuget feeds.

 

Creating a AzureDevops project

The 1st step is to create a new Azure Devops project, and from there you will need to go into the newly created project, and turn on the following

  • Repos : Which allows you to host repos in Azure Devops
  • Artifacts :  Which allows you create new Nuget feeds in Azure Devops

image

 

Once you have done that you can grab the repos remote clone details. So for a new project this may be something like this

 

image

 

So for me I then cloned the repo, and created a simple .NET Standard 2.0 library, that simply adds numbers

image

From there I simply pushed up the code to the remote repo.

 

So all good so far, we should have a new project which supports feeds and has our new code in it. Lets carry on

 

Creating a Nuget feed

So now that we have some code pushed up. How do we make it available on our own hosted Nuget feed. There are a couple of steps to perform here

 

Firstly we need to create a new Feed, which is done by going into the artifacts menu, and clicking the “Create Feed” button

 

image

 

You need to give the feed a name, lets suppose I chose “foofeed2” as the name, you should see something like this, where you will now need to go into the feed settings

 

image

 

If you click on the drop down you should see that the feed is created as “project scoped”, which means that it  belongs to the project. Until very recently this was not the case and all new feeds used to be scoped at organizational level, which effects how the build definition works. This was bug in Azure Devops. Which you can read more about in this StackOverflow which I created https://stackoverflow.com/questions/58856604/azure-devops-publish-nuget-to-hosted-feed

 

This was quite a weird bug to have but it did mean that all of a sudden for any Nuget feed you created and tried to publish it would not work. This is now fixed, and I will explain the difference between pushing to a project based NuGet feed and an Organizational one later when we discuss the build definition

 

For now you should make sure that the permissions for your need look something like this. Please forgive me but I am using a screen shot here from my actual project, where above I am just showing you what a new project will look like

 

image

 

I really do urge you to read the StackOverflow page as there is some really valuable discussions in there about scoping, and what extra permissions you need to ensure are there

 

Build Pipeline

So once we have the feed set up with the correct permissions, we can focus our attention to the build side of things.

 

This is complete source code for a build pipeline that does the following

  • Nuget restore
  • Builds
  • Package
  • Push to AzureDevop feed

 

# ASP.NET Core
# Build and test ASP.NET Core projects targeting .NET Core.
# Add steps that run tests, create a NuGet package, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/dotnet-core

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  buildConfiguration: 'Release'
  Major: '1'
  Minor: '0'
  Patch: '0'

steps:

- task: DotNetCoreCLI@2
  displayName: 'Restore'
  inputs:
    command: restore
    projects: '**/MathsLib.csproj'


- task: DotNetCoreCLI@2
  displayName: Build
  inputs:
    command: build
    projects: '**/MathsLib.csproj'
    arguments: '--configuration Release' # Update this to match your need


- task: DotNetCoreCLI@2
  inputs: 
    command: 'pack'
    projects: '**/MathsLib.csproj'
    outputDir: '$(Build.ArtifactStagingDirectory)'
    versioningScheme: 'byPrereleaseNumber'
    majorVersion: '1'
    minorVersion: '0'
    patchVersion: '0'


- task: NuGetCommand@2
  displayName: 'nuget push'
  inputs:
    command: 'push'
    feedsToUse: 'select'
    packagesToPush: '$(Build.ArtifactStagingDirectory)/**/*.nupkg;!$(Build.ArtifactStagingDirectory)/**/*.symbols.nupkg'
    nuGetFeedType: 'internal'
    vstsFeed: 'nugetprojects/anotherfeed'
    publishVstsFeed: 'nugetprojects/anotherfeed'
    versioningScheme: 'off'
    allowPackageConflicts: true

 

Project Scoped Feed

image

It should be noted that this is using a “project scoped” nuget feed. See these entries ‘nugetprojects/anotherfeed’ that is the syntax you need to use when using project scoped Nuget feeds.

 

Organizational Scoped Feed

image

 

In contrast to that, if you use an organizational feed you will need to ensure you use feed settings in the YAML above that just have the name of your feed so the push step would be this instead

- task: NuGetCommand@2
  displayName: 'nuget push'
  inputs:
    command: 'push'
    feedsToUse: 'select'
    packagesToPush: '$(Build.ArtifactStagingDirectory)/**/*.nupkg;!$(Build.ArtifactStagingDirectory)/**/*.symbols.nupkg'
    nuGetFeedType: 'internal'
    vstsFeed: 'testfeed'
    publishVstsFeed: 'testfeed'
    versioningScheme: 'off'
    allowPackageConflicts: true

As I say this was a VERY recent changed introduced in Azure Devops, and you can read more in the StackOverflow post that I refer to above

 

For now lets test this pipeline using the Project scoped feed “anotherfeed” above

image

 

Consuming the feed from a new project

With all this in place we should now be able to consume the feed from a new project, so lets see how we can so that. We can go to the feed to grab how to connect to it

 

image

Then pick the VisualStudio settings (or Nuget if you want to use nuget.config to configure your sources)

 

I’ll use VisualStudio one

 

image

 

I can then set this up in VisualStudio as a new Nuget feed

image

And search for my Nuget package (I did not create proper release so you need to  “include pre-releases”

image

 

Cool all working. So that’s it for this time

Azure

Getting reacquainted with Azure Devops

At the job before this one we used to make quite heavy use of VSTS, which is now Azure Devops. I used it a bit, but I am was never the “build” guy. At my current role, I am becoming more and more architecture/devops focussed. I am what one would call a hands on Architect.

Anyway long story short we use Gitlab/Rancher/AWS/Docker where I am now, but I just wanted to keep my toe in with what is going on in Azure, so I plan on getting to know the lay of the land of some of the new tooling

 

I thought what better way to start with something dead simple, like lets create a new WebApi .NET Core app, and get it deployed to a Azure App Service. This has to be pretty simple to start with right?

 

So before we start, what I did was sign up for a free tier in Azure, and added all my stuff to a single ResourceGroup in Azure.That way I can just trash it all to keep my costs low. For those that don’t know Azure resource groups are like the uber top level container that all your other services live in.

 

Ok so lets get started

 

Step 1 : Create the app

Creating the app for me was as simple as creating a new Web project in Visual Studio 2019, Where I stuck with the standard WebApi project. I literally did not change a thing, I stuck completely with the standard ValuesController, and targeted .NET Core 2.1

 

image

 

Step 2 : Created A GitHub Repo

So now that I have created a pretty useless demo app, I created a GitHub repo, and pushed this code to it

 

Step 3 : Azure AppService

So I have this small .NET Core 2.1 demo app, so I should be able to run it on Linux. So I sign up the free tier of Azure, then created a new AppService, which looked like these where I was careful to keep to the Dev/Test tier of the size plan

 

image

Ok so once that was created I was ready to head over to devops.azure.com

Step 3 : Azure DevOps

So the 1st thing I did was create a new project, lets call it “XXX” for now

image

From there I picked “GitHub” on the Connect part of the wizard. Next I simply picked my repository from the list, and then it was time to get down with the actual Build side of the Azure DevOps pipeline.

 

When I last used VSTS it was all UI based, like everything it has all turned to YAML now. So you need to look up the various build tasks to get the available task syntax that you can use. Luckily the basic skeleton was clever enough to give a very hand link to this page : https://docs.microsoft.com/en-gb/azure/devops/pipelines/ecosystems/dotnet-core?view=azure-devops

 

From there you can grab several of the tasks you might need such as

 

– Build

– Publish

 

If you want to see all the available actions for the DotNetCoreCLI@2 task you can look here : https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops, in fact from that link you can use the treeview on the left to find all sorts of tasks to help you live the DevOps pipeline dream

 

So once I read the docs a bit I ended up with this DevOps Build pipeline configuration

 

# ASP.NET Core
# Build and test ASP.NET Core projects targeting .NET Core.
# Add steps that run tests, create a NuGet package, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/dotnet-core

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  buildConfiguration: 'Release'

steps:
- task: DotNetCoreCLI@2
  displayName: Build
  inputs:
    command: build
    projects: '**/*.csproj'
    arguments: '--configuration Release' # Update this to match your need

- task: DotNetCoreCLI@2
  inputs:
    command: publish
    projects: '**/*.csproj'
    publishWebProjects: True
    arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)'
    zipAfterPublish: True

# this code takes all the files in $(Build.ArtifactStagingDirectory) and uploads them as an artifact of your build.
- task: PublishBuildArtifacts@1
  inputs:
    pathtoPublish: '$(Build.ArtifactStagingDirectory)' 
    artifactName: 'AzureDevOpsWebApiTestApp'
  

I think this is fairly self explanatory.So with this configured, it was time to test the Build. Which worked 1st time YAY.

 

image

 

So with the Build side of the pipeline working, I turned my hand towards the Release side of the pipeline. For me this was as simple as picking this type of release

 

image

I filled in the required parameters and ran the Release from the pipeline. It also passed. YAY

 

I then headed over to my previously created App Service in Azure, grabbed the public url, and gave it a try, and it worked

 

image

 

I have to say I was pleasantly surprised at just how smoothly all of that went.

 

I will be digging into Azure DevOps a bit more, I would like to see it run some tests, and I would like to setup some private nuget feeds and publish to those and consume from them. This will be the subjects of some future posts no doubt

Azure

Azure Databricks platform exploration

Now for those that know me will know I like lots of different technology, but as soon as I saw Apache Spark I fell in love with the fact that you could take a DataSet parallize it and send it out to a cluster of worker (think linq running across a cluster).

The only thing that was not great about Apache Spark was that you kind of wanted to treat it as an Analalytics engine to do one of jobs that you could get results from after it had finished.

This was possibly with Apache Spark but it was a bit clunky.

Luckily now we have a cloud hosted platform for working with Apache Spark, which is called DataBricks.

DataBricks exposes a bunch of REST APIs which really make working with Apache Spark shine even more.

I just had to find some time to have an explore, to this end I have written this article up on it which I hope some of you may find of interest : https://www.codeproject.com/Articles/1244429/Azure-Examining-Databricks-Apache-Spark-platform

Azure

Azure WebApp : Checking the deployed WebApp file system

 

So at work we are using Azure a lot, one of things we use a heck of a lot are web apps. We have a lot of these, some full blown web sites, some simple Service Stack REST APIs. We orchestrate the deployment of these Azure WebApps via the use of standard VSTS build/release steps, and the use of some custom ARM templates.

 

Just for a quick diversion this is what ARM templates are if you have not heard of them

 

What is Azure Resource Manager

Azure Resource Manager (ARM) allows you to provision your applications using a declarative template. In a single template, you can deploy multiple services along with their dependencies. You use the same template to repeatedly deploy your application during every stage of the application life cycle.

 

This post is NOT about ARM templates, but I just thought it worth calling out what they were, incase you have not heard of them before.

 

So what is this post about?

Well as I say we have a bunch of WebApps that we deploy to Azure, which most of the time is just fine, and we rarely need to check up on this automated deployment mechanism, it just works. However as most engineers will attest to, the shit fairy does occasionally come to town, and when she does she is sure to stop by and sprinkle a little chaos on your lovely deployment setup.

 

Now I don’t really believe in fairies, but I have certainly witnessed first hand that things have ended up deployed all wrong, and I have found myself in a situation where I needed to check the following things to make sure what I think I have configured is what is actually happening when I deploy

 

  1. VSTS steps are correct
  2. VSTS variables are correct
  3. Web.Config/AppSettings.json have the correct values in them when deployed

 

Items 1 AND 2 from that list are easy as we can check that inside VSTS, that is fairly ok. However item 3 requires us to get onto the actual deployment file system of the VM running the Azure WebApp that we tried to deploy. This is certainly not possible (to my knowledge) from VSTS.

 

So how can we see what was deployed for our Azure WebApp?

So it would seem we need to get access to the filesystem of the VM running the Azure WebApp. Now if you know anything about scale sets, and how Azure deals with WebApps you will know that you can’t really trust that what is there right now in terms of VMs is guaranteed to be the exact same VMs tomorrow. Azure just doesn’t work that way. If a VM is deemed unhealthy, it can and will be taken away, and a new on will be provisioned under the covers.

 

You generally don’t have to care about this, Azure just does its magic and we are blissfully unaware. Happy days.

 

However if we do need to do something like check a deployment, how do we do that? What VM should I try and gain access too? Will that VM be the same one tomorrow? Maybe/Maybe not. So we cant really write any funky scripts with set VM host names in them, as we may not be getting the same VM to host our WebApp from one day to the next. So how do we deal with this exactly?

 

Luckily there is a magic button in the Azure Portal that allows us to do just what we want.

 

Say we have a WebApp that we have created via some automated VSTS deployment setup

 

image

 

We can open the web app, and drill into its blade and look for the Advanced Tools

 

image

 

Then select the “Go” option from the panel that is displayed in the portal. Once you have done that a new tab will open in your browser that shows something like this

 

image

 

It can be seen that opens up a Kudu portal for the selected Azure WebApp.

 

But just what is Kudu?

Kudu is the engine behind git/hg deployments, WebJobs, and various other features in Azure Web Sites. It can also run outside of Azure.

 

Anyway once we have this Kudu portal open for our selected WebApp we can do various things, the one that we are interested in for this post is the Debug Console –> PowerShell

 

image

 

So lets launch that, we then get something like this

 

image

 

Aha a file system for the WebApp. Cool. So now all we need to do is explore this say by changing directories to site\wwwroot. Then from there we could say have a look at the Web.Config (this is a standard .NET web site so no AppSettings.json for this one)

 

We could examine the Web.Config content like this say where we use the Get-Content PowerShell commandlet

 

image

 

 

 

Conclusion

So that’s it. This was a small post, but I hope it showed you that even though the VMs you may be running on from one day to the next may change, you still have the tools you need to get in there and have a look around. Until next time then….

Azure

Small Azure EventGrid + Azure Functions Demo

I am a big fan of reactive programming, and love things like RX/Akka, and service buses and things like that, and I have been meaning to try the new (preview) Azure EventGrid service out for a while.

 

To this end I have given it a little go where I hooked it up to a Azure Function and written a small article about it here : https://www.codeproject.com/Articles/1220389/Azure-EventGrid-Azure-Function-demo

Azure

Azure Service Fabric Demo App

At work we have made use of the Azure Service Fabric, and I thought it might be nice to write up some of the fun we had with that. To this end I have written an article on it at codeproject.com which you can read here : https://www.codeproject.com/Articles/1217885/Azure-Service-Fabric-demo

The article covers :

  • Service Fabric basics
  • IOC
  • Logging (Serilog/Seq)
  • Encryption of connection strings

Anyway hope you like it

I’m going to write up this big Scala thing I have been doing, then I may post some more Azure bits and bobs, adios until then

 

 

Azure

Azure : Upload and stream video content to WPF from blob storage

A while back when Azure first came out I toyed with the idea of uploading video content to Azure Blob Storage, and having it play back in my WPF app. At the time (can’t recall exactly when that was, but quite a while ago) I had some major headaches doing this. The problem stemmed from the fact that the WPF MediaElement and the Azure Blob Storage did not play nicely together.

You just could not seek (that is when you go to an unbuffered / not downloaded) to a segment of the video and try and play. It just did not work, you would have to wait for the video to download ALL the content up the point you requested.

 

There is a very good post that discusses this old problem right here : http://programmerpayback.com/2013/01/30/hosting-progressive-download-videos-on-azure-blobs/

 

Previously you had to set the Blob storage API version. Starting from the 2011-08-18 version, you can do partial and pause/resume downloads on blob objects. The nice thing is that your client code doesn’t have to change to achieve this. 

 

Luckily this is no longer a problem, so now days it is as simple as following these steps:

 

  1. Upload a video (say MP4) to Azure Blob Storage
  2. Grab the Uri of the uploaded video
  3. Use that Uri for a WPF MediaElement

 

I have created a small demo app here for you, here is what it looks like after I have uploaded a video and pressed the play button

 

image

 

The code is dead simple, here is the XAML (its a WPF app)

 

<Window x:Class="WpfMediaPlayerFromBlobstorage.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        Title="MainWindow" Height="350" Width="525" WindowState="Maximized">
    <Grid>
        <DockPanel LastChildFill="True">
           
            <StackPanel Orientation="Horizontal" DockPanel.Dock="Top">
                <Button x:Name="btnUpload" 
                        Click="BtnUpload_OnClick" 
                        Content="Pick MP4 file to upload" 
                        Width="Auto" 
                        Margin="5"
                        Height="23"/>
                <StackPanel Orientation="Horizontal" Margin="50,5,5,5">
                    <StackPanel x:Name="controls" 
                                HorizontalAlignment="Center" 
                                Orientation="Horizontal">

                        <Button x:Name="btnPlay" 
                                Height="23" 
                                Content="Play" 
                                VerticalAlignment="Center"
                                Margin="5"
                                Click="BtnPlay_OnClick" />
                        <Button x:Name="btnPause" 
                                Height="23" 
                                Content="Pause" 
                                VerticalAlignment="Center"
                                Margin="5"
                                Click="BtnPause_OnClick" />
                        <Button x:Name="btnStop" 
                                Height="23" 
                                Content="Stop" 
                                VerticalAlignment="Center"
                                Click="BtnStop_OnClick"
                                Margin="5" />

                        <TextBlock VerticalAlignment="Center" 
                                   Text="Seek To"
                                   Margin="5" />
                        <Slider Name="timelineSlider" 
                                Margin="5" 
                                Height="23"
                                VerticalAlignment="Center"
                                Width="70"
                                ValueChanged="SeekToMediaPosition" />

                    </StackPanel>
                </StackPanel>
            </StackPanel>
            <MediaElement x:Name="player" 
                          Volume="1"
                          LoadedBehavior="Manual"
                          UnloadedBehavior="Manual"
                          HorizontalAlignment="Stretch" 
                          VerticalAlignment="Stretch"
                          Margin="10"
                          MediaOpened="Element_MediaOpened" 
                          MediaEnded="Element_MediaEnded"/>
        </DockPanel>
    </Grid>
</Window>

And here is the code behind (for simplicity I did not use MVVM for this demo)

 

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;

using Microsoft.Win32;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Auth;
using Microsoft.WindowsAzure.Storage.Blob;
using Microsoft.WindowsAzure.Storage.Shared.Protocol;

namespace WpfMediaPlayerFromBlobstorage
{
    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
        private static string blobStorageConnectionString =
            "DefaultEndpointsProtocol=http;AccountName=YOUR_ACCOUNT_HERE;AccountKey=YOUR_KEY_HERE";
        private Uri uploadedBlobUri=null;


        public MainWindow()
        {
            InitializeComponent();
            this.controls.IsEnabled = false;
        }

        private async void BtnUpload_OnClick(object sender, RoutedEventArgs e)
        {
            this.controls.IsEnabled = false;
            OpenFileDialog fd = new OpenFileDialog();
            fd.InitialDirectory=@"c:\";
            var result = fd.ShowDialog();
            if (result.HasValue && result.Value)
            {
                try
                {
                    var storageAccount = CloudStorageAccount.Parse(blobStorageConnectionString);
                    CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
                    CloudBlobContainer container = blobClient.GetContainerReference("mycontainer");
                    container.CreateIfNotExists();
                    CloudBlockBlob blockBlob = container.GetBlockBlobReference("myblob");
                    container.SetPermissions(
                        new BlobContainerPermissions
                        {
                            PublicAccess =
                                BlobContainerPublicAccessType.Blob
                        }
                    );

                    using (var fileStream = File.OpenRead(fd.FileName))
                    {
                        await blockBlob.UploadFromStreamAsync(fileStream);
                        uploadedBlobUri = blockBlob.Uri;
                        this.controls.IsEnabled = true;
                        MessageBox.Show("File uploaded ok");
                    }
                }
                catch (Exception exception)
                {
                    MessageBox.Show("Ooops : " + exception.Message);
                }
            }


           
        }

        private void BtnPlay_OnClick(object sender, RoutedEventArgs e)
        {
            player.Source = uploadedBlobUri;
            player.Play();
            timelineSlider.Value = 0;
        }

        private void BtnPause_OnClick(object sender, RoutedEventArgs e)
        {
            player.Pause();
        }

        private void BtnStop_OnClick(object sender, RoutedEventArgs e)
        {
            player.Stop();
            timelineSlider.Value = 0;
        }

        private void Element_MediaOpened(object sender, EventArgs e)
        {
            timelineSlider.Maximum = player.NaturalDuration.TimeSpan.TotalMilliseconds;
        }

        private void Element_MediaEnded(object sender, EventArgs e)
        {
            player.Stop();
            timelineSlider.Value = 0;
        }


        private void SeekToMediaPosition(object sender, 
		RoutedPropertyChangedEventArgs<double> args)
        {
            int sliderValue = (int)timelineSlider.Value;
            TimeSpan ts = new TimeSpan(0, 0, 0, 0, sliderValue);
            player.Position = ts;
        }
    }
}

And there you have it, a very simple media player that allows play/pause/stop and seek from a Azure Blob Storage uploaded video.

You can grab this project (you will need to fill in the Azure Blob Storage connection string details with your own account settings) from my github account here : https://github.com/sachabarber/WpfMediaPlayerFromBlobstorage

 

NOTE : If you want more control over encoding/streaming etc etc you should check out Azure Media Services