AWS : S3 File System

What are we talking about this time?

This time we are going to talk about AWS S3 fie system. This builds upon what we have covered in the last couple of posts, and will show you how you can leverage your own filesystem and AWS S3 filesystem.

 

Initial setup

If you did not read the very first part of this series of posts, I urge you to go and read that one now as it shows you how to get started with AWS, and create an IAM user : https://sachabarbs.wordpress.com/2018/08/30/aws-initial-setup/

 

Where is the code?

The code for this post can be found here in GitHub : https://github.com/sachabarber/AWS/tree/master/Storage/S3FileSystem

 

Ok so how does S3 File System work?

As we have already seen in the previous post S3 has a concept of buckets, and files. Thing is we can also use S3 to create file system like hierarchies, and place new directories/subdirectories and files in them. There is also a decent set of APIs to make it feel much like working with a regular file system in .NET. Since this is just a bit more about AWS S3 in essence, I will skip what I have already said about buckets etc etc.

 

IAM user privileges needed for S3

You will need to add these permissions to your IAM user to allow them to use S3

 

• AmazonS3FullAccess

 

Obviously if you are working in a team you will not want to give out full access, but for this series of posts this is fine.

 

 

So what will we try and cover in this post?

So to my mind what I am trying to cover is how to use the main classes in the S3 file system namespace, namely S3DirectoryInfo and S3FileInfo where we will see examples of how to do the following things

 

• Create a directory
• Create a file
• Create a sub directory
• Enumerate a directory structure
• Read file contents

 

But before we get onto the rest of the post lets see some of the methods available on these 2 classes

 

S3DirectoryInfo

Here is a breakdown of the methods/properties you can play with on this one

 

Bucket   
The S3DirectoryInfo for the root of the S3 bucket.

CopyFromLocal(String)   
Copies files from the local file system to S3 in this directory. Sub directories are copied as well.

CopyFromLocal(String, DateTime)   
Copies files from the local file system to S3 in this directory. Sub directories are copied as well. Only files that have been modified since the changesSince property will be copied.

CopyTo(String, String)   
Copies the files from this directory to the target directory specified by the bucket and object key.

CopyTo(String, String, DateTime)   
Copies the files from this directory to the target directory specified by the bucket and object key. Only files that have changed since the changeSince date will be copied.

CopyTo(S3DirectoryInfo)   
Copies the files from this directory to the target directory.

CopyTo(S3DirectoryInfo, DateTime)   
Copies the files from this directory to the target directory. Only files that have changed since the changeSince date will be copied.

CopyToLocal(String)   
Copies the files from the S3 directory to the local file system in the location indicated by the path parameter.

CopyToLocal(String, DateTime)   
Copies the files from the S3 directory to the local file system in the location indicated by the path parameter. Only files that have been modified since the changesSince property will be copied.

Create()   
Creates the directory in S3. If no object key was specified when creating the S3DirectoryInfo then the bucket will be created.

CreateSubdirectory(String)   
Creates a sub directory inside the instance of S3DirectoryInfo.

Delete()   
Deletes all the files in this directory as well as this directory.

Delete(Boolean)   
Deletes all the files in this directory as well as this directory. If recursive is set to true then all sub directories will be deleted as well.

EnumerateDirectories()   
Enumerate the sub directories of this directory.

EnumerateDirectories(String)   
Enumerate the sub directories of this directory.

EnumerateDirectories(String, SearchOption)   
Enumerate the sub directories of this directory.

EnumerateFiles()   
Enumerate the files of this directory.

EnumerateFiles(String)   
Enumerate the sub directories of this directory.

EnumerateFiles(String, SearchOption)   
Enumerate the files of this directory.

EnumerateFileSystemInfos()   
Enumerate the files of this directory.

EnumerateFileSystemInfos(String)   
Enumerate the files of this directory.

EnumerateFileSystemInfos(String, SearchOption)   
Enumerate the files of this directory.

Exists   
Checks with S3 to see if the directory exists and if so returns true. Due to Amazon S3’s eventual consistency model this property can return false for newly created buckets.

FullName   
The full path of the directory including bucket name.

GetDirectories()   
Returns an array of S3DirectoryInfos for the directories in this directory.

GetDirectories(String)   
Returns an array of S3DirectoryInfos for the directories in this directory.

GetDirectories(String, SearchOption)   
Returns an array of S3DirectoryInfos for the directories in this directory.

GetDirectory(String)   
Returns the S3DirectoryInfo for the specified sub directory.

GetFile(String)   
Returns the S3FileInfo for the specified file.

GetFiles()   
Returns an array of S3FileInfos for the files in this directory.

GetFiles(String)   
Returns an array of S3FileInfos for the files in this directory.

GetFiles(String, SearchOption)   
Returns an array of S3FileInfos for the files in this directory.

GetFileSystemInfos()   
Returns an array of IS3FileSystemInfos for the files in this directory.

GetFileSystemInfos(String)   
Returns an array of IS3FileSystemInfos for the files in this directory.

GetFileSystemInfos(String, SearchOption)   
Returns an array of IS3FileSystemInfos for the files in this directory.

LastWriteTime   
Returns the last write time of the the latest file written to the directory.

LastWriteTimeUtc   
UTC converted version of LastWriteTime.

MoveFromLocal(String)   
Moves files from the local file system to S3 in this directory. Sub directories are moved as well.

MoveTo(String, String)   
Moves the files from this directory to the target directory specified by the bucket and object key.

MoveTo(S3DirectoryInfo)   
Moves the files from this directory to the target S3 directory.

MoveToLocal(String)   
Moves the files from the S3 directory to the local file system in the location indicated by the path parameter.

Name   
Returns the name of the folder.

Parent   
Return the S3DirectoryInfo of the parent directory.

Root   
Returns the S3DirectroyInfo for the S3 account.

 

S3FileInfo

Here is a breakdown of the methods/properties you can play with on this one

 

CopyFromLocal(String)   
Copies the file from the local file system to S3. If the file already exists in S3 than an ArgumentException is thrown.

CopyFromLocal(String, Boolean)   
Copies the file from the local file system to S3. If the file already exists in S3 and overwrite is set to false than an ArgumentException is thrown.

CopyTo(String, String)   
Copies this file’s content to the file indicated by the S3 bucket and object key. If the file already exists in S3 than an ArgumentException is thrown.

CopyTo(String, String, Boolean)   
Copies this file’s content to the file indicated by the S3 bucket and object key. If the file already exists in S3 and overwrite is set to false than an ArgumentException is thrown.

CopyTo(S3DirectoryInfo)   
Copies this file to the target directory. If the file already exists in S3 than an ArgumentException is thrown.

CopyTo(S3DirectoryInfo, Boolean)   
Copies this file to the target directory. If the file already exists in S3 and overwrite is set to false than an ArgumentException is thrown.

CopyTo(S3FileInfo)   
Copies this file to the location indicated by the passed in S3FileInfo. If the file already exists in S3 than an ArgumentException is thrown.

CopyTo(S3FileInfo, Boolean)   
Copies this file to the location indicated by the passed in S3FileInfo. If the file already exists in S3 and overwrite is set to false than an ArgumentException is thrown.

CopyToLocal(String)   
Copies from S3 to the local file system. If the file already exists on the local file system than an ArgumentException is thrown.

CopyToLocal(String, Boolean)   
Copies from S3 to the local file system. If the file already exists on the local file system and overwrite is set to false than an ArgumentException is thrown.

Create()   
Returns a Stream that can be used to write data to S3. The content is persisted to S3 once the Stream is closed.

CreateText()   
Returns a StreamWriter that can be used to write data to S3. The content is persisted to S3 once the StreamWriter is closed.

Delete()   
Deletes the from S3.

Directory   
Returns the parent S3DirectoryInfo.

DirectoryName   
The full name of parent directory.

Exists   
Checks S3 if the file exists in S3 and return true if it does.

Extension   
Gets the file’s extension.

FullName   
Returns the full path including the bucket.

LastWriteTime   
Returns the last time the file was modified.

LastWriteTimeUtc   
Returns the last time the fule was modified in UTC.

Length   
Returns the content length of the file.

MoveFromLocal(String)   
Moves the file from the local file system to S3 in this directory. If the file already exists in S3 than an ArgumentException is thrown.

MoveFromLocal(String, Boolean)   
Moves the file from the local file system to S3 in this directory. If the file already exists in S3 and overwrite is set to false than an ArgumentException is thrown.

MoveTo(String, String)   
Moves the file to a a new location in S3.

MoveTo(S3DirectoryInfo)   
Moves the file to a a new location in S3.

MoveTo(S3FileInfo)   
Moves the file to a a new location in S3.

MoveToLocal(String)   
Moves the file from S3 to the local file system in the location indicated by the path parameter.

Name   
Returns the file name without its directory name.

OpenRead()   
Returns a Stream for reading the contents of the file.

OpenText()   
Returns a StreamReader for reading the contents of the file.

OpenWrite()   
Returns a Stream for writing to S3. If the file already exists it will be overwritten.

Replace(String, String, String, String)   
Replaces the destination file with the content of this file and then deletes the orignial file. If a backup location is specifed then the content of destination file is backup to it.

Replace(S3DirectoryInfo, S3DirectoryInfo)   
Replaces the destination file with the content of this file and then deletes the orignial file. If a backupDir is specifed then the content of destination file is backup to it.

Replace(S3FileInfo, S3FileInfo)   
Replaces the destination file with the content of this file and then deletes the orignial file. If a backupFile is specifed then the content of destination file is backup to it.

ReplaceLocal(String, String)   
Replaces the content of the destination file on the local file system with the content from this file from S3. If a backup file is specified then the content of the destination file is backup to it.

 

Beware

The reason I went to great pains to list the methods above, was to make you realize NOT ONE of them is Async/Await. So just watch that

 

 

Install the nugets

So lets start. The first thing we need to do is install the Nuget packages, which for this demo are

 

• AWSSDK.S3

 

Ok now that we have that in place and we know (thanks to the 1st post about how to use the default Profile which is linked to the IAM user for this demo series), we can just go through the items on the list above one by one

 

Create a directory

This shows how to create top level directory (bucket really)

S3DirectoryInfo rootDirectory = new S3DirectoryInfo(client, bucketName);
rootDirectory.Create();

 

Create a file

So now we can create a file like this

S3FileInfo readme = rootDirectory.GetFile("README.txt");
using (StreamWriter writer = new StreamWriter(readme.OpenWrite()))
    writer.WriteLine("This is my readme file.");

 

Create a sub directory

Creating a subdirectory is done like this

S3DirectoryInfo codeDir = rootDirectory.CreateSubdirectory("wiki");

 

Enumerate a directory structure

Should we wish to enumerate a directory structure we can do something like this

WriteDirectoryStructure(rootDirectory, 0);
.....
.....
.....
.....
static void WriteDirectoryStructure(S3DirectoryInfo directory, int level)
{
    StringBuilder indentation = new StringBuilder();
    for (int i = 0; i < level; i++)
        indentation.Append("\t");

    Console.WriteLine("{0}{1}", indentation, directory.Name);
    foreach (var file in directory.GetFiles())
        Console.WriteLine("\t{0}{1}", indentation, file.Name);

    foreach (var subDirectory in directory.GetDirectories())
    {
        WriteDirectoryStructure(subDirectory, level + 1);
    }
}

 

 

Read file contents

We can also read file contents (as a Stream too YAY)

foreach (var file in codeDir.GetFiles())
{
    Console.WriteLine("Content of {0}", file.Name);
    Console.WriteLine("------------------------------------");
    using (StreamReader reader = file.OpenText())
    {
        Console.WriteLine(reader.ReadToEnd());
    }
}

 

 

After running the demo code associated with this article we should see something like this (providing you have set the deleteAtEnd variable to false in the code)

 

 

 

See ya later, not goodbye

Ok that’s it for now until the next post