Transient Exception Handling

I am in the process of reading this very book : Cloud Design Patterns which discusses a great many patterns for the cloud (geared primarily at Azure), and there is talk of a “Circuit Breaker” and “Retry Pattern”.

The “Circuit Breaker” pattern is a new one to me, but the “Retry Pattern” is  certainly something I have seen many times before. The basic idea is this

You want to run a bit of code, but some Exception could occur, so you have some sort of retry policy in place, say one that retries every 10 seconds, that might work, if the Exception is a transient one caused by the network say.

Now I don’t know about you but I have hand coded this sort of thing a lot and never spent too much time (apart from this attempt I did when using Rx  which does actually work very well: http://stackoverflow.com/questions/20189166/rx-back-off-and-retry

Thing is, this is a recurring issue, so surely there is help out there for this issue. Turns out there is.

I have seen a few library that help out in this area, here are a few that I have found:

Palmer

This project is hosted at GutHub : https://github.com/mitchdenny/palmer/

And here are a few examples (taken from Github)

Retry  For Some Time

Retry for 15 seconds

Retry.On<WebException>().For(TimeSpan.FromSeconds(15)).With(context =>
  {
    // Code that might periodically fail due to connectivity issues.
  });

Retry Forever

Keep Retrying

Retry.On<WebException>().Indefinitely().With(context =>
    {
        // Code that might throw a web exception.
    });

Multiple Exceptions

You can also deal with multiple Exceptions

Retry.On<WebException>().For(5).AndOn<SqlException>().For(5).With(context =>
    {
        // Code that might throw a web exception, or a sql exception.
    });

For more examples check out the Palmer github link posted above

 

Polly

There is another nice library, which you can use which is also hosted at GitHub, it is called “Polly” : https://github.com/michael-wolfenden/Polly

So lets see some example, again this are taken directly from the GitHub readme

Step 1 : Specify the type of exceptions you want the policy to handle

// Single exception type
Policy
  .Handle<DivideByZeroException>()

// Single exception type with condition
Policy
  .Handle<SqlException>(ex => ex.Number == 1205)

// Multiple exception types
Policy
  .Handle<DivideByZeroException>()
  .Or<ArgumentException>()

// Multiple exception types with condition
Policy
  .Handle<SqlException>(ex => ex.Number == 1205)
  .Or<ArgumentException>(ex => x.ParamName == "example")

Step 2 : Specify how the policy should handle those exceptions

Retry
// Retry once
Policy
  .Handle<DivideByZeroException>()
  .Retry()

// Retry multiple times
Policy
  .Handle<DivideByZeroException>()
  .Retry(3)

// Retry multiple times, calling an action on each retry 
// with the current exception and retry count
Policy
    .Handle<DivideByZeroException>()
    .Retry(3, (exception, retryCount) =>
    {
        // do something 
    });

// Retry multiple times, calling an action on each retry 
// with the current exception, retry count and context 
// provided to Execute()
Policy
    .Handle<DivideByZeroException>()
    .Retry(3, (exception, retryCount, context) =>
    {
        // do something 
    });
Retry forever
// Retry forever
Policy
  .Handle<DivideByZeroException>()
  .RetryForever()

// Retry forever, calling an action on each retry with the 
// current exception
Policy
  .Handle<DivideByZeroException>()
  .RetryForever(exception =>
  {
        // do something       
  });

// Retry forever, calling an action on each retry with the
// current exception and context provided to Execute()
Policy
  .Handle<DivideByZeroException>()
  .RetryForever((exception, context) =>
  {
        // do something       
  });
Retry and Wait
// Retry, waiting a specified duration between each retry
Policy
  .Handle<DivideByZeroException>()
  .WaitAndRetry(new[]
  {
    TimeSpan.FromSeconds(1),
    TimeSpan.FromSeconds(2),
    TimeSpan.FromSeconds(3)
  });

// Retry, waiting a specified duration between each retry, 
// calling an action on each retry with the current exception
// and duration
Policy
  .Handle<DivideByZeroException>()
  .WaitAndRetry(new[]
  {
    1.Seconds(),
    2.Seconds(),
    3.Seconds()
  }, (exception, timeSpan) => {
    // do something    
  }); 

// Retry, waiting a specified duration between each retry, 
// calling an action on each retry with the current exception, 
// duration and context provided to Execute()
Policy
  .Handle<DivideByZeroException>()
  .WaitAndRetry(new[]
  {
    1.Seconds(),
    2.Seconds(),
    3.Seconds()
  }, (exception, timeSpan, context) => {
    // do something    
  });

// Retry a specified number of times, using a function to 
// calculate the duration to wait between retries based on 
// the current retry attempt (allows for exponential backoff)
// In this case will wait for
//  1 ^ 2 = 2 seconds then
//  2 ^ 2 = 4 seconds then
//  3 ^ 2 = 8 seconds then
//  4 ^ 2 = 16 seconds then
//  5 ^ 2 = 32 seconds
Policy
  .Handle<DivideByZeroException>()
  .WaitAndRetry(5, retryAttempt => 
    TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)) 
  );

// Retry a specified number of times, using a function to 
// calculate the duration to wait between retries based on 
// the current retry attempt, calling an action on each retry 
// with the current exception, duration and context provided 
// to Execute()
Policy
  .Handle<DivideByZeroException>()
  .WaitAndRetry(
    5, 
    retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)), 
    (exception, timeSpan, context) => {
      // do something
    }
  )

Step 3 : Execute the policy

// Execute an action
var policy = Policy
              .Handle<DivideByZeroException>()
              .Retry();

policy.Execute(() => DoSomething());

// Execute an action passing arbitrary context data
var policy = Policy
    .Handle<DivideByZeroException>()
    .Retry(3, (exception, retryCount, context) =>
    {
        var methodThatRaisedException = context["methodName"];
        Log(exception, methodThatRaisedException);
    });

policy.Execute(
    () => DoSomething(),
    new Dictionary<string, object>() {{ "methodName", "some method" }}
);

// Execute a function returning a result
var policy = Policy
              .Handle<DivideByZeroException>()
              .Retry();

var result = policy.Execute(() => DoSomething());

// Execute a function returning a result passing arbitrary context data
var policy = Policy
    .Handle<DivideByZeroException>()
    .Retry(3, (exception, retryCount, context) =>
    {
        object methodThatRaisedException = context["methodName"];
        Log(exception, methodThatRaisedException)
    });

var result = policy.Execute(
    () => DoSomething(),
    new Dictionary<string, object>() {{ "methodName", "some method" }}
);

// You can of course chain it all together
Policy
  .Handle<SqlException>(ex => ex.Number == 1205)
  .Or<ArgumentException>(ex => ex.ParamName == "example")
  .Retry()
  .Execute(() => DoSomething());

Transient Application Block

There is also the Microsoft Patterns & Practices offering “Transient Application Block”. Now P&P pattern can be, well a bit over the top to be frank. This one is not so bad though, its mainly geared toward working with Azure, but like a lot of stuff in Azure can be used outside of Azure.

There is a nice codeproject article on how to use this application block outside of Azure which you can read right here: Transient Fault Handling Application Block Simplified

Here is small example of how to use this block taken from the article noted here:

static void Main(string[] args)
{
    try
    {
        // Step 1
        var retryStrategy = new Incremental(RETRY_COUNT, TimeSpan.FromSeconds(2), TimeSpan.FromSeconds(2));

        // Step 2 
        var retryPolicy = new RetryPolicy<CustomTransientErrorDetectionStrategy>(retryStrategy);

        // Step 3
        retryPolicy.ExecuteAction(NavigateTo);
    }
    catch (Exception e)
    {
        Console.WriteLine(e.Message);
    }
}

// This is the code that may experience transient errors
static private void NavigateTo()
{
    Console.WriteLine(DateTime.Now);

    WebClient wc = new WebClient();
    wc.DownloadString("c:\\temp.txt");
}

Where the above code makes use of this custom ITransientErrorDetectionStrategy implementation

internal class CustomTransientErrorDetectionStrategy : ITransientErrorDetectionStrategy
{
    public bool IsTransient(Exception ex)
    {
        if (ex is WebException)
            return true;
        return false;
    }
}

Conclusion

Anyway there you go, I know I have done nothing more than paste a few links to other peoples work here, but there may be some people that did not know about these very useful libraries and you may think aha that is the one for me. So happy fault handling.

Until next time

Advertisements

4 thoughts on “Transient Exception Handling

  1. giammin says:

    very interesting! did not know about those library. I always created adhoc code. I’ll give them a try.

    anyway Palmer is simpler but it is also dead… 2 years without a commit…

    Why do you say: “P&P pattern can be, well a bit over the top to be frank”

    • sachabarber says:

      All I mean is that sometimes the P&P team tend to over do things a bit and create these huge frameworks, when you just need something smaller. Like Enterprise Library for example, the validation block in there allows / even encourages you to add validation rules in App.Config, my god nooooooooooooo

      That said I know a few people from that team, and they do also do some truly great frameworks like PRISM for example, and I have nothing but respect for what they are trying to achieve

  2. We’ve also published a Retry / Repeatable framework: https://github.com/endjin/Endjin.Retry and here’s a blog post that covers some of the usage scenarios: http://blogs.endjin.com/2013/05/retrying-tasks-with-tpl-async-and-synchronous-code/

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: