AWS

AWS : Step functions

What are we talking about this time?

Last time we talked about the “Serverless Framework” and how it could help us deploy functions with the minimum of fuss. This time we will be looking at Step Functions.

Initial setup

If you did not read the very first part of this series of posts, I urge you to go and read that one now as it shows you how to get started with AWS, and create an IAM user : https://sachabarbs.wordpress.com/2018/08/30/aws-initial-setup/

Where is the code

The code for this post can be found here in GitHub : https://github.com/sachabarber/AWS/tree/master/Compute/SimpleStepFunction

What are Step Functions?

AWS Step Functions lets you coordinate multiple AWS services into serverless workflows so you can build and update apps quickly. Using Step Functions, you can design and run workflows that stitch together services such as AWS Lambda and Amazon ECS into feature-rich applications. Workflows are made up of a series of steps, with the output of one step acting as input into the next. Application development is simpler and more intuitive using Step Functions, because it translates your workflow into a state machine diagram that is easy to understand, easy to explain to others, and easy to change. You can monitor each step of execution as it happens, which means you can identify and fix problems quickly. Step Functions automatically triggers and tracks each step, and retries when there are errors, so your application executes in order and as expected

https://aws.amazon.com/step-functions/ up on date 23/10/18

How do I get started with Step Functions?

Ok the first thing you need to do is do the initial setup stuff at the top of this post, then we can use the AWS .NET Toolkit which we also installed as part of very first post in this series. Lets see what that looks like out of the tin.

 

We start by choosing the “AWS Serverless Application with Tests”

image

This will then give us a screen like this

image

Where we choose the “Step Functions Hello World”, so after doing that we would see something like this shown to us inside Visual Studio.

image

Lets ignore the Tests project for now, but lets have a look at the actual project and try and understand what you get out of the box.

 

state-machine.json

So as we stated above Step Functions are workflows that are able to chain functions together. Lambda functions by themselves would not know how to do this, they need some other machinery above orchestrating this. This is the job of “state-machine.json”. If we look at the out of the box example we see this file contents:

{
  "Comment": "State Machine",
  "StartAt": "Greeting",
  "States": {
    "Greeting": {
      "Type": "Task",
      "Resource": "${GreetingTask.Arn}",
      "Next": "WaitToActivate"
    },
    "WaitToActivate": {
      "Type": "Wait",
      "SecondsPath": "$.WaitInSeconds",
      "Next": "Salutations"
    },
    "Salutations": {
      "Type": "Task",
      "Resource": "${SalutationsTask.Arn}",
      "End": true
    }
  }
}

There are a couple of take away points here:

  • It can be seen that this file represents the possible states, and also expresses how to move from one state to the next
  • The other key thing here is the use of the ${…Arn} notations. These are place holders that will be resolved via values in the serverless.template. When the project is deployed the contents of state-machine.json are copied into the serverless.template (Which we will look at very soon). The insertion location is controlled by the –template-substitutions parameter. The project template presets the –template-substitutions parameter in aws-lambda-tools-defaults.json. The format of the value for –template-substitutions is <json-path>=<file-name>.

    For example this project template sets the value to be:

    –template-substitutions $.Resources.StateMachine.Properties.DefinitionString.Fn::Sub=state-machine.json

 

State.cs

using System;
using System.Collections.Generic;
using System.Text;

namespace SimpleStepFunction
{
    /// <summary>
    /// The state passed between the step function executions.
    /// </summary>
    public class State
    {
        /// <summary>
        /// Input value when starting the execution
        /// </summary>
        public string Name { get; set; }

        /// <summary>
        /// The message built through the step function execution.
        /// </summary>
        public string Message { get; set; }

        /// <summary>
        /// The number of seconds to wait between calling the Salutations task and Greeting task.
        /// </summary>
        public int WaitInSeconds { get; set; } 
    }
}

What is a state machine without state. This is the state for the state machine as defined in the state-machine.json file

 

StepFunctionTasks.cs

As we saw above we define our state machine flow in the state-machine.json file, but we still need the actual AWS Lambda functions to call. The functions can reside in any file, as long as it’s a valid Lambda Function. For the out of the box example this is the file that goes with the state-machine.json file.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Threading.Tasks;

using Amazon.Lambda.Core;


// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.Json.JsonSerializer))]

namespace SimpleStepFunction
{
    public class StepFunctionTasks
    {
        /// <summary>
        /// Default constructor that Lambda will invoke.
        /// </summary>
        public StepFunctionTasks()
        {
        }


        public State Greeting(State state, ILambdaContext context)
        {
            state.Message = "Hello";

            if(!string.IsNullOrEmpty(state.Name))
            {
                state.Message += " " + state.Name;
            }

            // Tell Step Function to wait 5 seconds before calling 
            state.WaitInSeconds = 5;

            return state;
        }

        public State Salutations(State state, ILambdaContext context)
        {
            state.Message += ", Goodbye";

            if (!string.IsNullOrEmpty(state.Name))
            {
                state.Message += " " + state.Name;
            }

            return state;
        }
    }
}

Couple of points to note there:

  • There are 2 Functions here
    • Greeting
    • Salutations
  • There is the State class passed around as state, which may be written to in the functions

 

aws-lambda-tools-default.json

This file contains the defaults that will be used to do the deployment, it also shows how to integrate the state-machine.json with the serverless.template file

{
  "Information" : [
    "This file provides default values for the deployment wizard inside Visual Studio and the AWS Lambda commands added to the .NET Core CLI.",
    "To learn more about the Lambda commands with the .NET Core CLI execute the following command at the command line in the project root directory.",

    "dotnet lambda help",

    "All the command line options for the Lambda command can be specified in this file."
  ],
  "profile":"default",
  "region" : "eu-west-2",
  "configuration" : "Release",
  "framework"     : "netcoreapp2.1",
  "s3-prefix"     : "SimpleStepFunction/",
  "template"      : "serverless.template",
  "template-parameters" : "",
  "template-substitutions" : "$.Resources.StateMachine.Properties.DefinitionString.Fn::Sub=state-machine.json",
  "s3-bucket"              : "",
  "stack-name"             : ""
}

 

Take a look at the properties.

  • profile is the AWS credentials you use to connect to AWS. 
  • region is the AWS region you are going to use to deploy to
  • configuration is what configuration that is being deployed, for example, Release and Debug.
  • framework is associated with the .NET framework you wish to use
  • s3-prefix is the  prefix for the s3 bucket used to store the deployed code artifacts
  • template is the name of AWS Cloud Formation template to use
  • template-parameters are parameters for deployment.
  • template-substitutions is for identifying what to replace within the template. We mentioned this above
  • s3-bucket is the AWS S3 bucket that will be used for the deployed artifacts
  • stack-name is the name you will see in within AWS Cloud Formation console for the deployment

 

serverless.template

image 

Is the cloud formation template that describes your infrastructure requirements. For this example that would include

  • 2 Lambda functions
    • Greeting
    • Salutations
  • An IAM role for Lambda
  • The state machine

 

Ok so now we understand a bit more about the files lets see how we can deploy this. We can use Visual Studio to do this, but for real life you would use the AWS CLI

image

Where we can just work through the wizard

image

image

image

Once its published we should be able to go into the AWS console, and have a look at a few things

 

We can look at the Step Functions console in AWS, and we should see something like this

image

Which we can drill into, and “Start Execution” to test out

image

image

So lets click the button, and see what we get

image

image

Ok cool we can see that it worked nicely. So what about all that Cloud Formation stuff, how does that fit into it all. Lets go have a look at Cloud Formation Console in AWS

image#

So that’s all looking good.

 

What about more complex examples

Ok so we have seen the out of the box example, but what else can be done using step functions?

 

Well if we refer to the documentation on the states, which shows you all the possible state types, you can quickly see we could come up with some pretty cool workflows:

 

  • Pass: A Pass state (“Type”: “Pass”) simply passes its input to its output, performing no work. Pass states are useful when constructing and debugging state machines.
  • Task: A Task state (“Type”: “Task”) represents a single unit of work performed by a state machine.
  • Choice: A Choice state (“Type”: “Choice”) adds branching logic to a state machine.
  • Wait: A Wait state (“Type”: “Wait”) delays the state machine from continuing for a specified time. You can choose either a relative time, specified in seconds from when the state begins, or an absolute end-time, specified as a timestamp.
  • Succeed: A Succeed state (“Type”: “Succeed”) stops an execution successfully. The Succeed state is a useful target for Choice state branches that don’t do anything but stop the execution.Because Succeed states are terminal states, they have no Next field, nor do they have need of an End field
  • Fail: A Fail state (“Type”: “Fail”) stops the execution of the state machine and marks it as a failure.The Fail state only allows the use of Type and Comment fields from the set of common state fields.
  • Parallel: The Parallel state (“Type”: “Parallel”) can be used to create parallel branches of execution in your state machine.

You can read more about these states, and their various parameters here : https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-parallel-state.html

 

So for now lets expand apon our simple out of the box example, and adjust it to do the following

 

  • Start with an initial state, where we examine the incoming state object, and set the “IsMale” property to 1 if the Name starts with “Mr”
  • Enter a pass state (do nothing)
  • Enter a choice state, that will either call “PrintMaleInfo” next state, if the incoming state object “IsMale” is set to 1, otherwise  if its 0 “PrintFemaleInfo” will be called, if its not 0 or 1, “PrintInfo” will be the next state called
  • PrintMaleInfo/PrintFemaleInfo/PrintInfo are all terminals states

 

Here is what the revised state-machine.json looks like

{
  "Comment": "State Machine",
  "StartAt": "Initial",
  "States": {
    "Initial": {
      "Type": "Task",
      "Resource": "${InitialTask.Arn}",
      "Next": "WaitToActivate"
    },
    "WaitToActivate": {
      "Type": "Wait",
      "SecondsPath": "$.WaitInSeconds",
      "Next": "Pass"
    },
    "Pass": {
      "Type": "Task",
      "Resource": "${PassTask.Arn}",
      "Next": "ChoiceStateX"
    },

    "ChoiceStateX": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.IsMale",
          "NumericEquals": 1,
          "Next": "PrintMaleInfo"
        },
        {
          "Variable": "$.IsMale",
          "NumericEquals": 0,
          "Next": "PrintFemaleInfo"
        }
      ],
      "Default": "PrintInfo"
    },
    "PrintMaleInfo": {
      "Type": "Task",
      "Resource": "${PrintMaleInfoTask.Arn}",
      "End": true
    },
    "PrintFemaleInfo": {
      "Type": "Task",
      "Resource": "${PrintFemaleInfoTask.Arn}",
      "End": true
    },
    "PrintInfo": {
      "Type": "Task",
      "Resource": "${PrintInfoTask.Arn}",
      "End": true
    }
  }
}

And here is the revised serverless.template file

{
  "AWSTemplateFormatVersion" : "2010-09-09",
  "Transform" : "AWS::Serverless-2016-10-31",
  "Description" : "An AWS Serverless Application.",

  "Resources" : {
    "InitialTask" : {
        "Type" : "AWS::Lambda::Function",
        "Properties" : {
            "Handler" : "MoreRealWorldStepFunction::MoreRealWorldStepFunction.StepFunctionTasks::Initial",
            "Role"    : {"Fn::GetAtt" : [ "LambdaRole", "Arn"]},
            "Runtime" : "dotnetcore2.1",
            "MemorySize" : 256,
            "Timeout" : 30,
            "Code" : {
                "S3Bucket" : "",
                "S3Key" : ""
            }
        }
    },
	"PassTask" : {
        "Type" : "AWS::Lambda::Function",
        "Properties" : {
            "Handler" : "MoreRealWorldStepFunction::MoreRealWorldStepFunction.StepFunctionTasks::Pass",
			"Role"    : {"Fn::GetAtt" : [ "LambdaRole", "Arn"]},
            "Runtime" : "dotnetcore2.1",
            "MemorySize" : 256,
            "Timeout" : 30,
            "Code" : {
                "S3Bucket" : "",
                "S3Key" : ""
            }
        }
    },
	"PrintInfoTask" : {
        "Type" : "AWS::Lambda::Function",
        "Properties" : {
            "Handler" : "MoreRealWorldStepFunction::MoreRealWorldStepFunction.StepFunctionTasks::PrintInfo",
			"Role"    : {"Fn::GetAtt" : [ "LambdaRole", "Arn"]},
            "Runtime" : "dotnetcore2.1",
            "MemorySize" : 256,
            "Timeout" : 30,
            "Code" : {
                "S3Bucket" : "",
                "S3Key" : ""
            }
        }
    },
	"PrintMaleInfoTask" : {
        "Type" : "AWS::Lambda::Function",
        "Properties" : {
            "Handler" : "MoreRealWorldStepFunction::MoreRealWorldStepFunction.StepFunctionTasks::PrintMaleInfo",
			"Role"    : {"Fn::GetAtt" : [ "LambdaRole", "Arn"]},
            "Runtime" : "dotnetcore2.1",
            "MemorySize" : 256,
            "Timeout" : 30,
            "Code" : {
                "S3Bucket" : "",
                "S3Key" : ""
            }
        }
    },
	"PrintFemaleInfoTask" : {
        "Type" : "AWS::Lambda::Function",
        "Properties" : {
            "Handler" : "MoreRealWorldStepFunction::MoreRealWorldStepFunction.StepFunctionTasks::PrintFemaleInfo",
			"Role"    : {"Fn::GetAtt" : [ "LambdaRole", "Arn"]},
            "Runtime" : "dotnetcore2.1",
            "MemorySize" : 256,
            "Timeout" : 30,
            "Code" : {
                "S3Bucket" : "",
                "S3Key" : ""
            }
        }
    },
    "StateMachine" : {
        "Type" : "AWS::StepFunctions::StateMachine",
        "Properties": {
            "RoleArn": { "Fn::GetAtt": [ "StateMachineRole", "Arn" ] },
            "DefinitionString": { "Fn::Sub": "" }
        }
    },
    "LambdaRole" : {
        "Type" : "AWS::IAM::Role",
        "Properties" : {
            "AssumeRolePolicyDocument" : {
                "Version" : "2012-10-17",
                "Statement" : [
                    {
                        "Action" : [
                            "sts:AssumeRole"
                        ],
                        "Effect" : "Allow",
                        "Principal" : {
                            "Service" : [
                                "lambda.amazonaws.com"
                            ]
                        }
                    }
                ]
            },
            "ManagedPolicyArns" : [
                "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
            ]
       }
    },
    "StateMachineRole" : {
        "Type" : "AWS::IAM::Role",
        "Properties" : {
            "AssumeRolePolicyDocument" : {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Principal": {
                    "Service": {"Fn::Sub" : "states.${AWS::Region}.amazonaws.com"}
                  },
                  "Action": "sts:AssumeRole"
                }
              ]
            },
            "Policies" : [{
                "PolicyName": "StepFunctionLambdaInvoke",
                "PolicyDocument": {
                  "Version": "2012-10-17",
                  "Statement": [
                    {
                      "Effect": "Allow",
                      "Action": [
                        "lambda:InvokeFunction"
                      ],
                      "Resource": "*"
                    }
                  ]
                }
            }]
        }
    }
  },
  "Outputs" : {
  }
}

And finally here are the revised functions that are used by the state machine

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Threading.Tasks;

using Amazon.Lambda.Core;


// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.Json.JsonSerializer))]

namespace MoreRealWorldStepFunction
{
    public class StepFunctionTasks
    {
        /// <summary>
        /// Default constructor that Lambda will invoke.
        /// </summary>
        public StepFunctionTasks()
        {
        }


        public State Initial(State state, ILambdaContext context)
        {
            state.Message = $"Hello-{Guid.NewGuid().ToString()}";

            LogMessage(context, state.ToString());


            state.IsMale = state.Name.StartsWith("Mr") ? 1 : 0;


            // Tell Step Function to wait 5 seconds before calling 
            state.WaitInSeconds = 5;

            return state;
        }

        public State PrintMaleInfo(State state, ILambdaContext context)
        {
            LogMessage(context, "IS MALE");
            return state;
        }

        public State PrintFemaleInfo(State state, ILambdaContext context)
        {
            LogMessage(context, "IS FEMALE");
            return state;
        }


        public State Pass(State state, ILambdaContext context)
        {
            return state;
        }


        public State PrintInfo(State state, ILambdaContext context)
        {
            LogMessage(context, state.ToString());
            return state;
        }


        void LogMessage(ILambdaContext ctx, string msg)
        {
            ctx.Logger.LogLine(
                string.Format("{0}:{1} - {2}",
                    ctx.AwsRequestId,
                    ctx.FunctionName,
                    msg));
        }
    }
}

So with all that in place, lets check it out in the Step Function console, and see what the definition looks like and whether it runs ok. So this is what it looks like from the console

image

And when we try and execute it in the Step Function console, we can see this

image

So when we run this, it does indeed go down the “IsMale” choice state of “PrintMaleInfo”

image

Starting An Execution Using C# Code

So being able to upload a Step Function into AWS and run it via the Step Function AWS console is cool and all, but what would be better is if we are able run it via our own code. This is quite an interesting one, as there are quite a few different ways to do this, I will discuss a few of them

 

Permissioning an IAM user with the correct privileges

So throughout this series I have been using a single IAM user I created in the very 1st post in this series, now you may not want to do this in real life, but if you want that user to be able to Start a state machine execution request, you could try and add an inline policy something like this

 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "states:*",
            "Resource": "*"
        }
    ]
}

 

Of you could just give the IAM user this policy AWSStepFunctionsFullAccess. Either of which should allow you to kick of a step function using code something like this

static void ExecuteStepFunctionUsingDefaultProfileWithIAMStepFunctionsFullAccessInIAMConsole()
{
    var options = new AWSOptions()
    {
        Profile = "default",
        Region = RegionEndpoint.EUWest2
    };

    var amazonStepFunctionsConfig = new AmazonStepFunctionsConfig { RegionEndpoint = RegionEndpoint.EUWest2 };
    using (var amazonStepFunctionsClient = new AmazonStepFunctionsClient(amazonStepFunctionsConfig))
    {
        var state = new State
        {
            Name = "MyStepFunctions"
        };
        var jsonData1 = JsonConvert.SerializeObject(state);
        var startExecutionRequest = new StartExecutionRequest
        {
            Input = jsonData1,
            Name = $"SchedulingEngine_{Guid.NewGuid().ToString("N")}",
            StateMachineArn = "arn:aws:states:eu-west-2:464534050515:stateMachine:StateMachine-z8hrOwmL9CiG"
        };
        var taskStartExecutionResponse = amazonStepFunctionsClient.StartExecutionAsync(startExecutionRequest).ConfigureAwait(false).GetAwaiter().GetResult();
    }


    Console.ReadLine();
}

 

The thing with this is you are having to add the policies to your IAM user, which is cool, but another way may be to use an existing state machine role that was created by a previously deployed Step Function (say from Visual studio deploy).

 

Assuming Step Function Role

To do this you would need to assume the step function role. This would need code something like this

static void ExecuteStepFunctionUsingAssumedExistingStateMachineRole()
{
    var options = new AWSOptions()
    {
        Profile = "default",
        Region = RegionEndpoint.EUWest2
    };

    var assumedRoleResponse = ManualAssume(options).ConfigureAwait(false).GetAwaiter().GetResult();
    var assumedCredentials = assumedRoleResponse.Credentials;
    var amazonStepFunctionsConfig = new AmazonStepFunctionsConfig { RegionEndpoint = RegionEndpoint.EUWest2 };
    using (var amazonStepFunctionsClient = new AmazonStepFunctionsClient(
        assumedCredentials.AccessKeyId,
        assumedCredentials.SecretAccessKey, amazonStepFunctionsConfig))
    {
        var state = new State
        {
            Name = "MyStepFunctions"
        };
        var jsonData1 = JsonConvert.SerializeObject(state);
        var startExecutionRequest = new StartExecutionRequest
        {
            Input = jsonData1,
            Name = $"SchedulingEngine_{Guid.NewGuid().ToString("N")}",
            StateMachineArn = "arn:aws:states:eu-west-2:XXXXX:stateMachine:StateMachine-XXXXX"
        };
        var taskStartExecutionResponse = amazonStepFunctionsClient
			.StartExecutionAsync(startExecutionRequest)
			.ConfigureAwait(false)
			.GetAwaiter()
		    .GetResult();
    }

    Console.ReadLine();
}


public static async Task<AssumeRoleResponse> ManualAssume(AWSOptions options)
{
    var stsClient = options.CreateServiceClient<IAmazonSecurityTokenService>();
    var assumedRoleResponse = await stsClient.AssumeRoleAsync(new AssumeRoleRequest()
    {
        RoleArn = "arn:aws:iam::XXXXX:role/SimpleStepFunction-StateMachineRole-XXXXX",
        RoleSessionName = "test"
    });

    return assumedRoleResponse;

}

 

The important things there are

  • The Name of the execution should be unique
  • You get the ARN for the state machine from the AWS console

 

Within A Lambda

You could imagine that you could also use some code like that first example above for the permissioned IAM user, in a Lambda that you expose using the API Gateway (though you would need to provide the key/secret in code for the IAM user used by the AmazonStepFunctionsClient, as the default profile won’t be available in the actual AWS cloud, as profiles are stored locally on your PC).

 

You will be pleased to know there is already good support for this scenario using Api Gateway directly to execute a step function, you can read more about this here : https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-api-gateway.html (this is also a good read on this topic https://stackoverflow.com/questions/41113666/how-to-invoke-aws-step-function-using-api-gateway)

 

In fact it doesn’t stop there, the serverless framework that we looked at last time, also has good support for step functions and triggering them via http. You can read more about this approach here : https://serverless.com/blog/how-to-manage-your-aws-step-functions-with-serverless/ (you may need to combine that with the code in my last article as this link assumed a Node.Js based lambda, my article however showed a C# serverless framework example)

 

See ya later, not goodbye

Ok that’s it for now until the next post

Advertisements
AWS

AWS Using Serveless Framework To Create A Lambda Function

What are we talking about this time?

This time we are going to talk about how to expose our AWS lambda function over HTTP, which is exactly the same as what I did in my last AWS article, the thing is I did it all by hand in that article. This time we are going to be talking about the “Serverless Framework”.

Initial setup

If you did not read the very first part of this series of posts, I urge you to go and read that one now as it shows you how to get started with AWS, and create an IAM user : https://sachabarbs.wordpress.com/2018/08/30/aws-initial-setup/

Where is the code

The code for this post can be found here in GitHub : https://github.com/sachabarber/AWS/tree/master/Compute/ServerlessFrameworkLambda

What Is The Serverless Framework?

Picture says a 1000ns words and all that, here is a quick intro picture to the Serverless Framework.

image

At its heart it is an abstraction layer between your code and cloud, where it is cloud agnostic, your code may not be, but serverless itself is, and can be used quite happily against Azure/AWS etc etc

You should be asking yourself just how it does this abstraction over these cloud vendors? That’s a pretty neat trick isn’t it? Why yes, lets see how it works.

Its all mainly down to this rather clever abstraction file called “serverless.yml” which tells the framework what it should provision for you, and how things communicate, and should be setup.

Here is the example one for this demo app which is a simple AWS Lambda exposes as a GET REST API using AWS API Gateway

# Welcome to Serverless!
#
# This file is the main config file for your service.
# It's very minimal at this point and uses default values.
# You can always add more config options for more control.
# We've included some commented out config examples here.
# Just uncomment any of them to get that config option.
#
# For full config options, check the docs:
#    docs.serverless.com
#
# Happy Coding!

service: ServerlessFrameworkLambda # NOTE: update this with your service name

# You can pin your service to only deploy with a specific Serverless version
# Check out our docs for more details
# frameworkVersion: "=X.X.X"

provider:
  name: aws
  runtime: dotnetcore2.1

# you can overwrite defaults here
#  stage: dev
  region: eu-west-2

# you can add statements to the Lambda function's IAM Role here
#  iamRoleStatements:
#    - Effect: "Allow"
#      Action:
#        - "s3:ListBucket"
#      Resource: { "Fn::Join" : ["", ["arn:aws:s3:::", { "Ref" : "ServerlessDeploymentBucket" } ] ]  }
#    - Effect: "Allow"
#      Action:
#        - "s3:PutObject"
#      Resource:
#        Fn::Join:
#          - ""
#          - - "arn:aws:s3:::"
#            - "Ref" : "ServerlessDeploymentBucket"
#            - "/*"

# you can define service wide environment variables here
#  environment:
#    variable1: value1

# you can add packaging information here
package:
  artifact: bin/release/netcoreapp2.1/deploy-package.zip
#  exclude:
#    - exclude-me.js
#    - exclude-me-dir/**

functions:
  hello:
    handler: CsharpHandlers::AwsDotnetCsharp.Handler::Hello

#    The following are a few example events you can configure
#    NOTE: Please make sure to change your handler code to work with those events
#    Check the event documentation for details
    events:
      - http:
          path: gettime
          method: get
          cors: true
#      - s3: ${env:BUCKET}
#      - schedule: rate(10 minutes)
#      - sns: greeter-topic
#      - stream: arn:aws:dynamodb:region:XXXXXX:table/foo/stream/1970-01-01T00:00:00.000
#      - alexaSkill: amzn1.ask.skill.xx-xx-xx-xx
#      - alexaSmartHome: amzn1.ask.skill.xx-xx-xx-xx
#      - iot:
#          sql: "SELECT * FROM 'some_topic'"
#      - cloudwatchEvent:
#          event:
#            source:
#              - "aws.ec2"
#            detail-type:
#              - "EC2 Instance State-change Notification"
#            detail:
#              state:
#                - pending
#      - cloudwatchLog: '/aws/lambda/hello'
#      - cognitoUserPool:
#          pool: MyUserPool
#          trigger: PreSignUp

#    Define function environment variables here
#    environment:
#      variable2: value2

# you can add CloudFormation resource templates here
#resources:
#  Resources:
#    NewResource:
#      Type: AWS::S3::Bucket
#      Properties:
#        BucketName: my-new-bucket
#  Outputs:
#     NewOutput:
#       Description: "Description for the output"
#       Value: "Some output value"

You can see from the commented lines, how you might configure some of the other functionality you may need, and the docs are pretty decent. You can see more examples here : https://github.com/serverless/examples

But isn’t this quite limited? No not really, for example this is what the AWS offering looks like for serverless functions. This is pretty much exactly what AWS offers without the use of Serverless Framework.

imageAnd just to contrast here is the Azure offering.

image

Now as I say all you care about is the code, and the serverless.yml file, that governs the deployment/update/rollback. Serverless Framework will deal with the cloud provider for you. But just how do you get started with this framework?

The rest of this post will talk you through that.

Installation

The Serverless Framework is a node based command line installation, as such you will need to install node if you don’t already have it, download it from https://nodejs.org/en/download/. Once you have downloaded that, simply open a Node command line window and install the Serverless Framework as follows

npm install -g serverless

Credentials

The next thing you will need to do is associate your cloud provider credentials with the Serverless Framework command line, which you can read about here : https://serverless.com/framework/docs/providers/aws/guide/credentials/

For AWS this would look something like this

serverless config credentials --provider aws --key ANN7EXAMPLE --secret wJalrXUtnFYEXAMPLEKEY

Create A Project

Ok now that you are that far is, you can now create a project. This is easily done as follows, where this one uses a AWS c# template.

serverless create --template aws-csharp --path myService

NOTE : I found that I could not include “.” in the name of my service

Changes I Made At This Point

At this point I updated the serverless.yml file to the one I showed a minute ago, and I then updated the C# function to match almost exactly with the code from my last AWS article

using Amazon.Lambda.APIGatewayEvents;
using Amazon.Lambda.Core;
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Net;

[assembly:LambdaSerializer(typeof(Amazon.Lambda.Serialization.Json.JsonSerializer))]

namespace AwsDotnetCsharp
{
    public class Handler
    {
        ITimeProcessor processor = new TimeProcessor();

        public APIGatewayProxyResponse Hello(
           APIGatewayProxyRequest apigProxyEvent, ILambdaContext context)
        {
            LogMessage(context, "Processing request started");
            APIGatewayProxyResponse response;
            try
            {
                var result = processor.CurrentTimeUTC();
                response = CreateResponse(result);

                LogMessage(context, "Processing request succeeded.");
            }
            catch (Exception ex)
            {
                LogMessage(context,
                    string.Format("Processing request failed - {0}", ex.Message));
                response = CreateResponse(null);
            }

            return response;
        }

        APIGatewayProxyResponse CreateResponse(DateTime? result)
        {
            int statusCode = (result != null) ?
                (int)HttpStatusCode.OK :
                (int)HttpStatusCode.InternalServerError;

            string body = (result != null) ?
                JsonConvert.SerializeObject(result) : string.Empty;

            var response = new APIGatewayProxyResponse
            {
                StatusCode = statusCode,
                Body = body,
                Headers = new Dictionary<string, string>
                {
                    { "Content-Type", "application/json" },
                    { "Access-Control-Allow-Origin", "*" }
                }
            };
            return response;
        }

        /// 
<summary>
        /// Logs messages to cloud watch
        /// </summary>

        void LogMessage(ILambdaContext ctx, string msg)
        {
            ctx.Logger.LogLine(
                string.Format("{0}:{1} - {2}",
                    ctx.AwsRequestId,
                    ctx.FunctionName,
                    msg));
        }
    }

  
}

Package A Project

Once you have edited your code, and potentially the serverless.yml file you need to package it, which for a C# project means running the build.cmd command in PowerShell. Internally this runs the AWS dot net command line extensions for Lambda https://github.com/aws/aws-extensions-for-dotnet-cli so you may find you also need to install those too. See my last AWS article on details about how to do that

Deploy The Function

Open the node command prompt at the place where you have your serverless.yml file, and run serverless deploy, and you should see something like this, where it is provisioning the various cloud provider items needed by your serverless.yml fileimageLet’s go and have a look what it created.  imageOk so we see 1 matching AWS function. CoolimageLet’s drill in a bit further, on the matching functionimageWe see the function does indeed have a matching API Gateway (just like in the previous AWS post I did)imageOk so now lets drill into the API GatewayimageSo far so good, we can see the resource created matches what we specified in our serverless.yml fileimageSo lets test the endpoint. Woohoo we see the time, its working, and this was significantly less hassle than the last article were I had to jump into IAM settings, API Gateway configuration, Lambda configuration, publish from Visual Studio/command line wizards. 

We have only really scratched the surface of using the Serverless Framework for Lambda/functions in the cloud, as I say the documentation is pretty good, it really is worth a try if you have not played with it before.


Beware

See ya later, not goodbye

Ok that’s it for now until the next post

 

 



Lambdas / Anonomous delegates

Nice little trick when working with Expression API

So I like writing Expression tree API magic, but I am mortal and don’t find it that easy to do, and it normally takes me a while to do this. So I am up for any help that I can get.

I just learnt a neat little trick, which is that you can apply TypeDescriptor attributes at runtime, which is cool. This one thing allows us to use a fairly standard PropertyGrid to visualize Expression trees

For example here is a simple Winforms UI I crafted with a few Expressions to visualize and each one just gets set to the PropertyGrid SelectedObject when selected from the list

image

Ok its not perfect but it is good enough to help you out

Here is the relevant code from the simple Winforms UI

Expression<Func<string, string, int>> fun1 = (s1, s2) => s1.Length - s2.Length;

TypeDescriptor.AddAttributes(typeof(Expression),
    new TypeConverterAttribute(typeof(ExpandableObjectConverter)));


propertyGrid1.SelectedObject = fun1;
AWS

AWS Lambda exposed via ApiGateway

 

What are we talking about this time?

This time we are going to talk about how to expose our AWS lambda function over HTTP.  This is actually fairly simple to do so this will not be a big post, and will certainly build on what we saw in the last post where I introduced how to create and publish a new AWS lambda function.

Initial setup

If you did not read the very first part of this series of posts, I urge you to go and read that one now as it shows you how to get started with AWS, and create an IAM user : https://sachabarbs.wordpress.com/2018/08/30/aws-initial-setup/

Where is the code

The code for this post can be found here in GitHub : https://github.com/sachabarber/AWS/tree/master/Compute/Lambda.ApiGateway.DemoApp

What is AWS API Gateway?

Amazon API Gateway is an AWS service that enables developers to create, publish, maintain, monitor, and secure APIs at any scale. You can create APIs that access AWS or other web services, as well as data stored in the AWS Cloud.

image

In practical terms, API Gateway lets you create, configure, and host a RESTful API to enable applications to access the AWS Cloud. For example, an application can call an API in API Gateway to upload a user’s annual income and expense data to Amazon Simple Storage Service or Amazon DynamoDB, process the data in AWS Lambda to compute tax owed, and file a tax return via the IRS website.

As shown in the diagram, an app (or client application) gains programmatic access to AWS services, or a website on the internet, through one or more APIs, which are hosted in API Gateway. The app is at the API’s frontend. The integrated AWS services and websites are located at the API’s backend. In API Gateway, the frontend is encapsulated by method requests and method responses, and the backend is encapsulated by integration requests and integration responses.

With Amazon API Gateway, you can build an API to provide your users with an integrated and consistent developer experience to build AWS cloud-based applications.

https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html up on date 07/10/18

Writing a Lambda that Is exposes via API Gateway

Ok so now that we know what an API Gateway is, how do we write a AWS Lambda to use it? Well as before there are APIGatewayEvents that can be used inside of a Lambda function. Lets see the relevant code shall we:

using System;
using System.Collections.Generic;
using System.Net;
using Amazon.Lambda.APIGatewayEvents;
using Amazon.Lambda.Core;
using Newtonsoft.Json;

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.Json.JsonSerializer))]

namespace Lambda.ApiGateway.DemoApp
{
    public class Function
    {

        ITimeProcessor processor = new TimeProcessor();

        /// <summary>
        /// Default constructor. This constructor is used by Lambda to construct
        /// the instance. When invoked in a Lambda environment
        /// the AWS credentials will come from the IAM role associated with the
        /// function and the AWS region will be set to the
        /// region the Lambda function is executed in.
        /// </summary>
        public Function()
        {

        }

        public APIGatewayProxyResponse FunctionHandler(
            APIGatewayProxyRequest apigProxyEvent, ILambdaContext context)
        {
            LogMessage(context, "Processing request started");
            APIGatewayProxyResponse response;
            try
            {
                var result = processor.CurrentTimeUTC();
                response = CreateResponse(result);

                LogMessage(context, "Processing request succeeded.");
            }
            catch (Exception ex)
            {
                LogMessage(context, 
                    string.Format("Processing request failed - {0}", ex.Message));
                response = CreateResponse(null);
            }

            return response;
        }

        APIGatewayProxyResponse CreateResponse(DateTime? result)
        {
            int statusCode = (result != null) ?
                (int)HttpStatusCode.OK :
                (int)HttpStatusCode.InternalServerError;

            string body = (result != null) ?
                JsonConvert.SerializeObject(result) : string.Empty;

            var response = new APIGatewayProxyResponse
            {
                StatusCode = statusCode,
                Body = body,
                Headers = new Dictionary<string, string>
                {
                    { "Content-Type", "application/json" },
                    { "Access-Control-Allow-Origin", "*" }
                }
            };
            return response;
        }

        /// <summary>
        /// Logs messages to cloud watch
        /// </summary>
        void LogMessage(ILambdaContext ctx, string msg)
        {
            ctx.Logger.LogLine(
                string.Format("{0}:{1} - {2}",
                    ctx.AwsRequestId,
                    ctx.FunctionName,
                    msg));
        }

    }
}

It can be seen that we need to use a specialized APIGatewayProxyRequest/APIGatewayProxyResponse pair

In this example we are exposing the AWS Lambda as a GET only operation. If you wanted to accept POST/DELETE/PUT data you could use the APIGatewayProxyRequest.Body to get the data representing the request.

Ok, so now that we have the code, and lets assume that the Lambda has been published to AWS (see the last article for a detailed explanation of how to do that). For now lets assume we have published the above code to AWS, and we have it available in the AWS console, we would now need to configure the API Gateway part of it.

Which  starts with just telling the ApiGateway trigger which stage to run in. This is shown below

imageOnce we have configured the ApiGateway trigger for the published lambda, we should see it shown something like the screen shot below. We then need to move on to setting up the actual ApiGateway resources themselves and how they relate to the Lambda call.

image

We can do this either by following the link shown within the Api Gateway section of our lambda as shown above, or via the AWS console where we just search for the Api Gateway. Both paths are valid, and should lead you to a screen something like the one below. It is from this screen that we will add new resources.

imageSo we wish (at least for this demo) to create a GET resource that will call our Lambda. We can do this by using the Actions menu, and creating a new GET from the drop down options. We then setup the GET resource to call the lambda (the one for this demo). This is all shown in the screen shot below

imageOnce we have added the resource we should be able to test it out using the Test button (the one shown below with the lightning bolt on it). This will test the Api resource. So for this demo this should call the Lambda and GET a new time returned to the Api Gateway call, and we should see a status code of 200 (Ok)

imageSo if that tests out just fine, we are almost there. All we need to do now ensure that the Api Gateway is deployed. This can be done using the “Deploy API” menu option from the Api Gateway portal, as shown below.

imageWith all that done, we should be able to test our deployed Api Gateway pointing to our Lambda, so lets grab the public endpoint for the Api Gateway, which we can do my examining the Stages menu, then finding our resource (GET in this case) and getting hold of the Invoke Url.

imageSo for me this looks like this

imageCool looks like its working.

See ya later, not goodbye

Ok that’s it for now until the next post