What are we talking about this time?
Last time we talked about the “Serverless Framework” and how it could help us deploy functions with the minimum of fuss. This time we will be looking at Step Functions.
Initial setup
If you did not read the very first part of this series of posts, I urge you to go and read that one now as it shows you how to get started with AWS, and create an IAM user : https://sachabarbs.wordpress.com/2018/08/30/aws-initial-setup/
Where is the code
The code for this post can be found here in GitHub : https://github.com/sachabarber/AWS/tree/master/Compute/SimpleStepFunction
What are Step Functions?
AWS Step Functions lets you coordinate multiple AWS services into serverless workflows so you can build and update apps quickly. Using Step Functions, you can design and run workflows that stitch together services such as AWS Lambda and Amazon ECS into feature-rich applications. Workflows are made up of a series of steps, with the output of one step acting as input into the next. Application development is simpler and more intuitive using Step Functions, because it translates your workflow into a state machine diagram that is easy to understand, easy to explain to others, and easy to change. You can monitor each step of execution as it happens, which means you can identify and fix problems quickly. Step Functions automatically triggers and tracks each step, and retries when there are errors, so your application executes in order and as expected
https://aws.amazon.com/step-functions/ up on date 23/10/18
How do I get started with Step Functions?
Ok the first thing you need to do is do the initial setup stuff at the top of this post, then we can use the AWS .NET Toolkit which we also installed as part of very first post in this series. Lets see what that looks like out of the tin.
We start by choosing the “AWS Serverless Application with Tests”
This will then give us a screen like this
Where we choose the “Step Functions Hello World”, so after doing that we would see something like this shown to us inside Visual Studio.
Lets ignore the Tests project for now, but lets have a look at the actual project and try and understand what you get out of the box.
state-machine.json
So as we stated above Step Functions are workflows that are able to chain functions together. Lambda functions by themselves would not know how to do this, they need some other machinery above orchestrating this. This is the job of “state-machine.json”. If we look at the out of the box example we see this file contents:
{
"Comment": "State Machine",
"StartAt": "Greeting",
"States": {
"Greeting": {
"Type": "Task",
"Resource": "${GreetingTask.Arn}",
"Next": "WaitToActivate"
},
"WaitToActivate": {
"Type": "Wait",
"SecondsPath": "$.WaitInSeconds",
"Next": "Salutations"
},
"Salutations": {
"Type": "Task",
"Resource": "${SalutationsTask.Arn}",
"End": true
}
}
}
There are a couple of take away points here:
- It can be seen that this file represents the possible states, and also expresses how to move from one state to the next
- The other key thing here is the use of the ${…Arn} notations. These are place holders that will be resolved via values in the serverless.template. When the project is deployed the contents of state-machine.json are copied into the serverless.template (Which we will look at very soon). The insertion location is controlled by the –template-substitutions parameter. The project template presets the –template-substitutions parameter in aws-lambda-tools-defaults.json. The format of the value for –template-substitutions is <json-path>=<file-name>.
For example this project template sets the value to be:
–template-substitutions $.Resources.StateMachine.Properties.DefinitionString.Fn::Sub=state-machine.json
State.cs
using System;
using System.Collections.Generic;
using System.Text;
namespace SimpleStepFunction
{
/// <summary>
/// The state passed between the step function executions.
/// </summary>
public class State
{
/// <summary>
/// Input value when starting the execution
/// </summary>
public string Name { get; set; }
/// <summary>
/// The message built through the step function execution.
/// </summary>
public string Message { get; set; }
/// <summary>
/// The number of seconds to wait between calling the Salutations task and Greeting task.
/// </summary>
public int WaitInSeconds { get; set; }
}
}
What is a state machine without state. This is the state for the state machine as defined in the state-machine.json file
StepFunctionTasks.cs
As we saw above we define our state machine flow in the state-machine.json file, but we still need the actual AWS Lambda functions to call. The functions can reside in any file, as long as it’s a valid Lambda Function. For the out of the box example this is the file that goes with the state-machine.json file.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Threading.Tasks;
using Amazon.Lambda.Core;
// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.Json.JsonSerializer))]
namespace SimpleStepFunction
{
public class StepFunctionTasks
{
/// <summary>
/// Default constructor that Lambda will invoke.
/// </summary>
public StepFunctionTasks()
{
}
public State Greeting(State state, ILambdaContext context)
{
state.Message = "Hello";
if(!string.IsNullOrEmpty(state.Name))
{
state.Message += " " + state.Name;
}
// Tell Step Function to wait 5 seconds before calling
state.WaitInSeconds = 5;
return state;
}
public State Salutations(State state, ILambdaContext context)
{
state.Message += ", Goodbye";
if (!string.IsNullOrEmpty(state.Name))
{
state.Message += " " + state.Name;
}
return state;
}
}
}
Couple of points to note there:
- There are 2 Functions here
- Greeting
- Salutations
- There is the State class passed around as state, which may be written to in the functions
aws-lambda-tools-default.json
This file contains the defaults that will be used to do the deployment, it also shows how to integrate the state-machine.json with the serverless.template file
{
"Information" : [
"This file provides default values for the deployment wizard inside Visual Studio and the AWS Lambda commands added to the .NET Core CLI.",
"To learn more about the Lambda commands with the .NET Core CLI execute the following command at the command line in the project root directory.",
"dotnet lambda help",
"All the command line options for the Lambda command can be specified in this file."
],
"profile":"default",
"region" : "eu-west-2",
"configuration" : "Release",
"framework" : "netcoreapp2.1",
"s3-prefix" : "SimpleStepFunction/",
"template" : "serverless.template",
"template-parameters" : "",
"template-substitutions" : "$.Resources.StateMachine.Properties.DefinitionString.Fn::Sub=state-machine.json",
"s3-bucket" : "",
"stack-name" : ""
}
Take a look at the properties.
- profile is the AWS credentials you use to connect to AWS.
- region is the AWS region you are going to use to deploy to
- configuration is what configuration that is being deployed, for example, Release and Debug.
- framework is associated with the .NET framework you wish to use
- s3-prefix is the prefix for the s3 bucket used to store the deployed code artifacts
- template is the name of AWS Cloud Formation template to use
- template-parameters are parameters for deployment.
- template-substitutions is for identifying what to replace within the template. We mentioned this above
- s3-bucket is the AWS S3 bucket that will be used for the deployed artifacts
- stack-name is the name you will see in within AWS Cloud Formation console for the deployment
serverless.template
Is the cloud formation template that describes your infrastructure requirements. For this example that would include
- 2 Lambda functions
- Greeting
- Salutations
- An IAM role for Lambda
- The state machine
Ok so now we understand a bit more about the files lets see how we can deploy this. We can use Visual Studio to do this, but for real life you would use the AWS CLI
Where we can just work through the wizard
Once its published we should be able to go into the AWS console, and have a look at a few things
We can look at the Step Functions console in AWS, and we should see something like this
Which we can drill into, and “Start Execution” to test out
So lets click the button, and see what we get
Ok cool we can see that it worked nicely. So what about all that Cloud Formation stuff, how does that fit into it all. Lets go have a look at Cloud Formation Console in AWS
#
So that’s all looking good.
What about more complex examples
Ok so we have seen the out of the box example, but what else can be done using step functions?
Well if we refer to the documentation on the states, which shows you all the possible state types, you can quickly see we could come up with some pretty cool workflows:
- Pass: A Pass state (“Type”: “Pass”) simply passes its input to its output, performing no work. Pass states are useful when constructing and debugging state machines.
- Task: A Task state (“Type”: “Task”) represents a single unit of work performed by a state machine.
- Choice: A Choice state (“Type”: “Choice”) adds branching logic to a state machine.
- Wait: A Wait state (“Type”: “Wait”) delays the state machine from continuing for a specified time. You can choose either a relative time, specified in seconds from when the state begins, or an absolute end-time, specified as a timestamp.
- Succeed: A Succeed state (“Type”: “Succeed”) stops an execution successfully. The Succeed state is a useful target for Choice state branches that don’t do anything but stop the execution.Because Succeed states are terminal states, they have no Next field, nor do they have need of an End field
- Fail: A Fail state (“Type”: “Fail”) stops the execution of the state machine and marks it as a failure.The Fail state only allows the use of Type and Comment fields from the set of common state fields.
- Parallel: The Parallel state (“Type”: “Parallel”) can be used to create parallel branches of execution in your state machine.
You can read more about these states, and their various parameters here : https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-parallel-state.html
So for now lets expand apon our simple out of the box example, and adjust it to do the following
- Start with an initial state, where we examine the incoming state object, and set the “IsMale” property to 1 if the Name starts with “Mr”
- Enter a pass state (do nothing)
- Enter a choice state, that will either call “PrintMaleInfo” next state, if the incoming state object “IsMale” is set to 1, otherwise if its 0 “PrintFemaleInfo” will be called, if its not 0 or 1, “PrintInfo” will be the next state called
- PrintMaleInfo/PrintFemaleInfo/PrintInfo are all terminals states
Here is what the revised state-machine.json looks like
{
"Comment": "State Machine",
"StartAt": "Initial",
"States": {
"Initial": {
"Type": "Task",
"Resource": "${InitialTask.Arn}",
"Next": "WaitToActivate"
},
"WaitToActivate": {
"Type": "Wait",
"SecondsPath": "$.WaitInSeconds",
"Next": "Pass"
},
"Pass": {
"Type": "Task",
"Resource": "${PassTask.Arn}",
"Next": "ChoiceStateX"
},
"ChoiceStateX": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.IsMale",
"NumericEquals": 1,
"Next": "PrintMaleInfo"
},
{
"Variable": "$.IsMale",
"NumericEquals": 0,
"Next": "PrintFemaleInfo"
}
],
"Default": "PrintInfo"
},
"PrintMaleInfo": {
"Type": "Task",
"Resource": "${PrintMaleInfoTask.Arn}",
"End": true
},
"PrintFemaleInfo": {
"Type": "Task",
"Resource": "${PrintFemaleInfoTask.Arn}",
"End": true
},
"PrintInfo": {
"Type": "Task",
"Resource": "${PrintInfoTask.Arn}",
"End": true
}
}
}
And here is the revised serverless.template file
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Transform" : "AWS::Serverless-2016-10-31",
"Description" : "An AWS Serverless Application.",
"Resources" : {
"InitialTask" : {
"Type" : "AWS::Lambda::Function",
"Properties" : {
"Handler" : "MoreRealWorldStepFunction::MoreRealWorldStepFunction.StepFunctionTasks::Initial",
"Role" : {"Fn::GetAtt" : [ "LambdaRole", "Arn"]},
"Runtime" : "dotnetcore2.1",
"MemorySize" : 256,
"Timeout" : 30,
"Code" : {
"S3Bucket" : "",
"S3Key" : ""
}
}
},
"PassTask" : {
"Type" : "AWS::Lambda::Function",
"Properties" : {
"Handler" : "MoreRealWorldStepFunction::MoreRealWorldStepFunction.StepFunctionTasks::Pass",
"Role" : {"Fn::GetAtt" : [ "LambdaRole", "Arn"]},
"Runtime" : "dotnetcore2.1",
"MemorySize" : 256,
"Timeout" : 30,
"Code" : {
"S3Bucket" : "",
"S3Key" : ""
}
}
},
"PrintInfoTask" : {
"Type" : "AWS::Lambda::Function",
"Properties" : {
"Handler" : "MoreRealWorldStepFunction::MoreRealWorldStepFunction.StepFunctionTasks::PrintInfo",
"Role" : {"Fn::GetAtt" : [ "LambdaRole", "Arn"]},
"Runtime" : "dotnetcore2.1",
"MemorySize" : 256,
"Timeout" : 30,
"Code" : {
"S3Bucket" : "",
"S3Key" : ""
}
}
},
"PrintMaleInfoTask" : {
"Type" : "AWS::Lambda::Function",
"Properties" : {
"Handler" : "MoreRealWorldStepFunction::MoreRealWorldStepFunction.StepFunctionTasks::PrintMaleInfo",
"Role" : {"Fn::GetAtt" : [ "LambdaRole", "Arn"]},
"Runtime" : "dotnetcore2.1",
"MemorySize" : 256,
"Timeout" : 30,
"Code" : {
"S3Bucket" : "",
"S3Key" : ""
}
}
},
"PrintFemaleInfoTask" : {
"Type" : "AWS::Lambda::Function",
"Properties" : {
"Handler" : "MoreRealWorldStepFunction::MoreRealWorldStepFunction.StepFunctionTasks::PrintFemaleInfo",
"Role" : {"Fn::GetAtt" : [ "LambdaRole", "Arn"]},
"Runtime" : "dotnetcore2.1",
"MemorySize" : 256,
"Timeout" : 30,
"Code" : {
"S3Bucket" : "",
"S3Key" : ""
}
}
},
"StateMachine" : {
"Type" : "AWS::StepFunctions::StateMachine",
"Properties": {
"RoleArn": { "Fn::GetAtt": [ "StateMachineRole", "Arn" ] },
"DefinitionString": { "Fn::Sub": "" }
}
},
"LambdaRole" : {
"Type" : "AWS::IAM::Role",
"Properties" : {
"AssumeRolePolicyDocument" : {
"Version" : "2012-10-17",
"Statement" : [
{
"Action" : [
"sts:AssumeRole"
],
"Effect" : "Allow",
"Principal" : {
"Service" : [
"lambda.amazonaws.com"
]
}
}
]
},
"ManagedPolicyArns" : [
"arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
]
}
},
"StateMachineRole" : {
"Type" : "AWS::IAM::Role",
"Properties" : {
"AssumeRolePolicyDocument" : {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": {"Fn::Sub" : "states.${AWS::Region}.amazonaws.com"}
},
"Action": "sts:AssumeRole"
}
]
},
"Policies" : [{
"PolicyName": "StepFunctionLambdaInvoke",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"lambda:InvokeFunction"
],
"Resource": "*"
}
]
}
}]
}
}
},
"Outputs" : {
}
}
And finally here are the revised functions that are used by the state machine
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Threading.Tasks;
using Amazon.Lambda.Core;
// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.Json.JsonSerializer))]
namespace MoreRealWorldStepFunction
{
public class StepFunctionTasks
{
/// <summary>
/// Default constructor that Lambda will invoke.
/// </summary>
public StepFunctionTasks()
{
}
public State Initial(State state, ILambdaContext context)
{
state.Message = $"Hello-{Guid.NewGuid().ToString()}";
LogMessage(context, state.ToString());
state.IsMale = state.Name.StartsWith("Mr") ? 1 : 0;
// Tell Step Function to wait 5 seconds before calling
state.WaitInSeconds = 5;
return state;
}
public State PrintMaleInfo(State state, ILambdaContext context)
{
LogMessage(context, "IS MALE");
return state;
}
public State PrintFemaleInfo(State state, ILambdaContext context)
{
LogMessage(context, "IS FEMALE");
return state;
}
public State Pass(State state, ILambdaContext context)
{
return state;
}
public State PrintInfo(State state, ILambdaContext context)
{
LogMessage(context, state.ToString());
return state;
}
void LogMessage(ILambdaContext ctx, string msg)
{
ctx.Logger.LogLine(
string.Format("{0}:{1} - {2}",
ctx.AwsRequestId,
ctx.FunctionName,
msg));
}
}
}
So with all that in place, lets check it out in the Step Function console, and see what the definition looks like and whether it runs ok. So this is what it looks like from the console
And when we try and execute it in the Step Function console, we can see this
So when we run this, it does indeed go down the “IsMale” choice state of “PrintMaleInfo”
Starting An Execution Using C# Code
So being able to upload a Step Function into AWS and run it via the Step Function AWS console is cool and all, but what would be better is if we are able run it via our own code. This is quite an interesting one, as there are quite a few different ways to do this, I will discuss a few of them
Permissioning an IAM user with the correct privileges
So throughout this series I have been using a single IAM user I created in the very 1st post in this series, now you may not want to do this in real life, but if you want that user to be able to Start a state machine execution request, you could try and add an inline policy something like this
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "states:*",
"Resource": "*"
}
]
}
Of you could just give the IAM user this policy AWSStepFunctionsFullAccess. Either of which should allow you to kick of a step function using code something like this
static void ExecuteStepFunctionUsingDefaultProfileWithIAMStepFunctionsFullAccessInIAMConsole()
{
var options = new AWSOptions()
{
Profile = "default",
Region = RegionEndpoint.EUWest2
};
var amazonStepFunctionsConfig = new AmazonStepFunctionsConfig { RegionEndpoint = RegionEndpoint.EUWest2 };
using (var amazonStepFunctionsClient = new AmazonStepFunctionsClient(amazonStepFunctionsConfig))
{
var state = new State
{
Name = "MyStepFunctions"
};
var jsonData1 = JsonConvert.SerializeObject(state);
var startExecutionRequest = new StartExecutionRequest
{
Input = jsonData1,
Name = $"SchedulingEngine_{Guid.NewGuid().ToString("N")}",
StateMachineArn = "arn:aws:states:eu-west-2:464534050515:stateMachine:StateMachine-z8hrOwmL9CiG"
};
var taskStartExecutionResponse = amazonStepFunctionsClient.StartExecutionAsync(startExecutionRequest).ConfigureAwait(false).GetAwaiter().GetResult();
}
Console.ReadLine();
}
The thing with this is you are having to add the policies to your IAM user, which is cool, but another way may be to use an existing state machine role that was created by a previously deployed Step Function (say from Visual studio deploy).
Assuming Step Function Role
To do this you would need to assume the step function role. This would need code something like this
static void ExecuteStepFunctionUsingAssumedExistingStateMachineRole()
{
var options = new AWSOptions()
{
Profile = "default",
Region = RegionEndpoint.EUWest2
};
var assumedRoleResponse = ManualAssume(options).ConfigureAwait(false).GetAwaiter().GetResult();
var assumedCredentials = assumedRoleResponse.Credentials;
var amazonStepFunctionsConfig = new AmazonStepFunctionsConfig { RegionEndpoint = RegionEndpoint.EUWest2 };
using (var amazonStepFunctionsClient = new AmazonStepFunctionsClient(
assumedCredentials.AccessKeyId,
assumedCredentials.SecretAccessKey, amazonStepFunctionsConfig))
{
var state = new State
{
Name = "MyStepFunctions"
};
var jsonData1 = JsonConvert.SerializeObject(state);
var startExecutionRequest = new StartExecutionRequest
{
Input = jsonData1,
Name = $"SchedulingEngine_{Guid.NewGuid().ToString("N")}",
StateMachineArn = "arn:aws:states:eu-west-2:XXXXX:stateMachine:StateMachine-XXXXX"
};
var taskStartExecutionResponse = amazonStepFunctionsClient
.StartExecutionAsync(startExecutionRequest)
.ConfigureAwait(false)
.GetAwaiter()
.GetResult();
}
Console.ReadLine();
}
public static async Task<AssumeRoleResponse> ManualAssume(AWSOptions options)
{
var stsClient = options.CreateServiceClient<IAmazonSecurityTokenService>();
var assumedRoleResponse = await stsClient.AssumeRoleAsync(new AssumeRoleRequest()
{
RoleArn = "arn:aws:iam::XXXXX:role/SimpleStepFunction-StateMachineRole-XXXXX",
RoleSessionName = "test"
});
return assumedRoleResponse;
}
The important things there are
- The Name of the execution should be unique
- You get the ARN for the state machine from the AWS console
Within A Lambda
You could imagine that you could also use some code like that first example above for the permissioned IAM user, in a Lambda that you expose using the API Gateway (though you would need to provide the key/secret in code for the IAM user used by the AmazonStepFunctionsClient, as the default profile won’t be available in the actual AWS cloud, as profiles are stored locally on your PC).
You will be pleased to know there is already good support for this scenario using Api Gateway directly to execute a step function, you can read more about this here : https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-api-gateway.html (this is also a good read on this topic https://stackoverflow.com/questions/41113666/how-to-invoke-aws-step-function-using-api-gateway)
In fact it doesn’t stop there, the serverless framework that we looked at last time, also has good support for step functions and triggering them via http. You can read more about this approach here : https://serverless.com/blog/how-to-manage-your-aws-step-functions-with-serverless/ (you may need to combine that with the code in my last article as this link assumed a Node.Js based lambda, my article however showed a C# serverless framework example)
See ya later, not goodbye
Ok that’s it for now until the next post
Sacha's Blog 
One of the best article on Step Functions with .NET/C# example. Thanks.
Thanks