What are we talking about this time?
This time we are going to talk about AWS S3 TransferUtility. This is a fairly simple utility class, and as such this is not going to be a long post.
Initial setup
If you did not read the very first part of this series of posts, I urge you to go and read that one now as it shows you how to get started with AWS, and create an IAM user : https://sachabarbs.wordpress.com/2018/08/30/aws-initial-setup/
Where is the code?
The code for this post can be found here in GitHub : https://github.com/sachabarber/AWS/tree/master/Storage/S3TrasferUtility
So what is the TransferUtility
Provides a high level utility for managing transfers to and from Amazon S3.
TransferUtility provides a simple API for uploading content to and downloading content from Amazon S3. It makes extensive use of Amazon S3 multipart uploads to achieve enhanced throughput, performance, and reliability.
When uploading large files by specifying file paths instead of a stream, TransferUtility uses multiple threads to upload multiple parts of a single upload at once. When dealing with large content sizes and high bandwidth, this can increase throughput significantly.
IAM user privileges needed for S3
You will need to add these permissions to your IAM user to allow them to use S3
- AmazonS3FullAccess
Obviously if you are working in a team you will not want to give out full access, but for this series of posts this is fine.
So what will we try and cover in this post?
So to my mind what I am trying to cover is how to use the TransferUtility class to do the following things
- Write a Stream to S3 bucket from local content
- Read a Stream from s3 bucket
- Download an object from s3 as a Stream to local file
Install the nugets
So lets start. The first thing we need to do is install the Nuget packages, which for this demo are
- AWSSDK.S3
Ok now that we have that in place and we know (thanks to the 1st post about how to use the default Profile which is linked to the IAM user for this demo series), we can just go through the items on the list above one by one
Write a Stream to S3 bucket from local content
When you have large files, you really want to send them as Streams to S3. Even better still would be would be to create different chunks and have different worker upload their own chunk of the overall large file. This is totally possible to do yourself, but luckily for us, this is what the TransferUtility does for us all for free. So all we have to do to upload a potentially large file as a Stream is this sort of thing
async Task WritingAStreamPublicAsync()
{
await CarryOutAWSTask(async () =>
{
//sly inner function
async Task<Stream> GenerateStreamFromStringAsync(string s)
{
var stream = new MemoryStream();
var writer = new StreamWriter(stream);
await writer.WriteAsync(s);
await writer.FlushAsync();
stream.Position = 0;
return stream;
}
await CreateABucketAsync(bucketName);
var fileTransferUtility = new TransferUtility(client);
var contentsToUpload = "some random string contents";
Console.WriteLine("Uploading the following contents using TransferUtility");
Console.WriteLine(contentsToUpload);
using (var streamToUpload = await GenerateStreamFromStringAsync(contentsToUpload))
{
var uploadRequest = new TransferUtilityUploadRequest()
{
InputStream = streamToUpload,
Key = fileName,
BucketName = bucketName,
CannedACL = S3CannedACL.PublicRead
};
//If you are uploading large files, TransferUtility
//will use multipart upload to fulfill the request
await fileTransferUtility.UploadAsync(uploadRequest);
}
Console.WriteLine($"Upload using stream to file '{fileName}' completed");
}, "Writing using a Stream to public file");
}
Read a Stream from s3 bucket
So now that we have uploaded something, it would be reasonable to be able to get a StreamReader to read from the S3 object as Stream, where we do not take the hit of reading the entire object into memory in one go. Obviously as this is just a demo I know what I uploaded to be VERY small, so we just read the entire Stream into memory.
async Task ReadingAnObjectFromS3AsAStream()
{
await CarryOutAWSTask(async () =>
{
var fileTransferUtility = new TransferUtility(client);
using (var fs = await fileTransferUtility.OpenStreamAsync(bucketName, fileName, CancellationToken.None))
{
using (var reader = new StreamReader(fs))
{
var contents = await reader.ReadToEndAsync();
Console.WriteLine($"Content of file {fileName} is");
Console.WriteLine(contents);
}
}
}, "Reading an Object from S3 as a Stream");
}
Download an object from s3 as a Stream to local file
To my mind the last thing you may want to do is to actually download the data as a Stream to a local file. This time we use the TransferUtility.DownloadAsync to download to a file of our choosing. In this contrived example we then read out the saved file contents just to print it out. As before as its my own code, I know the content is tiny, so we can read entire Stream into memory.
async Task DownloadingAnObjectFromS3AsAStream()
{
await CarryOutAWSTask(async () =>
{
var fileTransferUtility = new TransferUtility(client);
string theTempFile = Path.Combine(Path.GetTempPath(), "SavedS3TextFile.txt");
try
{
await fileTransferUtility.DownloadAsync(theTempFile, bucketName, fileName);
using (var fs = new FileStream(theTempFile, FileMode.Open))
{
using (var reader = new StreamReader(fs))
{
var contents = await reader.ReadToEndAsync();
Console.WriteLine($"Content of saved file {theTempFile} is");
Console.WriteLine(contents);
}
}
}
finally
{
File.Delete(theTempFile);
}
}, "Downloading an Object from S3 as a Stream");
}
TransferUtility Request Events
If you want to track the progress of uploads/downloads using the TransferUtility you should use the method overloads that accept the required request. The requests themselves typically expose an event that you can hook into to track progress. Here is one such example
// Summary:
// Contains all the parameters that can be set when making a this request with the
// TransferUtility method.
public class TransferUtilityDownloadRequest : BaseDownloadRequest
{
public TransferUtilityDownloadRequest();
//
// Summary:
// Get or sets the file path location of where the downloaded Amazon S3 object will
// be written to.
public string FilePath { get; set; }
//
// Summary:
// The event for WriteObjectProgressEvent notifications. All subscribers will be
// notified when a new progress event is raised.
// The WriteObjectProgressEvent is fired as data is downloaded from S3. The delegates
// attached to the event will be passed information detailing how much data has
// been downloaded as well as how much will be downloaded.
//
// Remarks:
// Subscribe to this event if you want to receive WriteObjectProgressEvent notifications.
// Here is how: 1. Define a method with a signature similar to this one: private
// void displayProgress(object sender, WriteObjectProgressArgs args) { Console.WriteLine(args);
// } 2. Add this method to the WriteObjectProgressEvent delegate's invocation list
// TransferUtilityDownloadRequest request = new TransferUtilityDownloadRequest();
// request.WriteObjectProgressEvent += displayProgress;
public event EventHandler<WriteObjectProgressArgs> WriteObjectProgressEvent;
}
See ya later, not goodbye
Ok that’s it for now until the next post
Sacha's Blog 