Working With Files and Folders in S3, Using AWS SDK for .NET

Introduction

In this tutorial, I am going to show you how to use AWS SDK for .NET to do some basic file operations on an S3 bucket. AWS provides both low-level and high-level APIs. First, we are going to see how to use low-level APIs and then we will perform the same operations using high-level APIs.

Prerequisite

In order to run the following code, you need to install the AWSSDK.S3 Nuget package.

I have created an S3 bucket named: my-bucket-name-123 and I have created a folder named my-folder inside the bucket:

Low-Level APIs

The low-level APIs are mapped closely to the underlying REST API, here we use a Request object to provide request information and AWS responds with the Response object.

Understanding S3 Path (when using low-level API)

First, we need to understand that there is no concept of folder in S3, everything is an object. If I want to create a folder called sub-folder, I need to append the folder name with a / to let AWS know that what I want is a folder, not a file... so I need to create:

my-s3-bucket-name-123/my-folder/sub-folder/

If I don't include the trailing slash AWS will create an object called sub-folder instead of a folder.

Initializing AmazonS3Client

This is how we initialize S3 clients, which we are going to use for the remaining examples:

string bucketName = "my-bucket-name-123";  
string awsAccessKey = "AKI............";  
string awsSecretKey = "+8Bo..................................";  

IAmazonS3 client = new AmazonS3Client(_awsAccessKey, _awsSecretKey, 
RegionEndpoint.APSoutheast2);

AmazonS3Client is thread safe, you can make it static or use a singletone instance.

Creating a folder

Here we are going to create a folder called sub-folder inside my-folder:

string folderPath = "my-folder/sub-folder/";

PutObjectRequest request = new PutObjectRequest()  
{  
    BucketName = _bucketName,  
    Key = folderPath // <-- in S3 key represents a path  
};  
PutObjectResponse response = client.PutObject(request);  

If you forget the trailing slash in the path (i.e. "my-folder/sub-folder") it would create an object called sub-folder... if you include a slash at the beginning of the path (i.e. "/my-folder/sub-folder/") it will create a folder with name as an empty string and put the remaining folders inside it.

Copying a file into a folder

The following code would copy test.txt inside sub-folder:

FileInfo file = new FileInfo(@"c:\test.txt");  
string path = "my-folder/sub-folder/test.txt";  

PutObjectRequest request = new PutObjectRequest()  
{  
    InputStream = file.OpenRead(),  
    BucketName = _bucketName,  
    Key = path // <-- in S3 key represents a path  
};  
PutObjectResponse response = client.PutObject(request);

Listing content of a folder

The following code would list the contents of sub-folder:

ListObjectsRequest request = new ListObjectsRequest  
{  
    BucketName = _bucketName,  
    Prefix = "my-folder/sub-folder/"  
};  

ListObjectsResponse response = client.ListObjects(request);    
foreach (S3Object obj in response.S3Objects)    
{    
    Console.WriteLine(obj.Key);    
}    

// result:  
// my-folder/sub-folder/  
// my-folder/sub-folder/test.txt  

Deleting file/folder

In the following code first we delete test.txt and then sub-folder:

// delete test.txt file  
string filePath = "my-folder/sub-folder/test.txt";  

var deleteFileRequest = new DeleteObjectRequest  
{  
    BucketName = _bucketName,  
    Key = filePath  
};  
DeleteObjectResponse fileDeleteResponse = client.DeleteObject(deleteFileRequest);  

// delete sub-folder  
string folderPath = "my-folder/sub-folder/";  
var deleteFolderRequest = new DeleteObjectRequest  
{  
    BucketName = _bucketName,  
    Key = folderPath  
};  
DeleteObjectResponse folderDeleteResponse = client.DeleteObject(deleteFolderRequest);

High-level APIs

High-level APIs are designed to mimic the semantic of File I/O operations. They are very similar to working with FileInfo and Directory.

Understanding the S3 path (when using high-level APIs)

When using high-level APIs, we need to use windows' styles paths, so use a backslash (NOT slash) in your path,

"my-folder\sub-folder\test.txt"

Also note that, similar to low-level APIs we need a trailing backslash to indicate a folder, for example, "my-folder\sub-folder\" indicates that sub-folder is a folder whereas "my-folder\sub-folder" indicates that sub-folder is an object inside my-folder.

Initializing AmazonS3Client

Use the same code as low-level APIs (above) to initialize AmazonS3Client.

Creating a folder

Here we are going to create a folder called high-level-folder and create another folder called my-folder inside it:

string path = @"high-level-folder";

S3DirectoryInfo di = new S3DirectoryInfo(client, _bucketName, path);  
if (!di.Exists)  
{  
    di.Create();  
    di.CreateSubdirectory("sub-folder");  
}  

Copying file into folder

The following code would copy test.txt inside sub-folder:

FileInfo localFile = new FileInfo(@"c:\test.txt");  
string path = @"high-level-folder\sub-folder\test.txt";  

S3FileInfo s3File = new S3FileInfo(client, _bucketName, path);  
if (!s3File.Exists)  
{  
    using (var s3Stream = s3File.Create()) // <-- create file in S3  
    {  
        localFile.OpenRead().CopyTo(s3Stream); // <-- copy the content to S3  
    }  
}  

Listing content of a folder

The following code would list the content of a sub-folder:

string path = @"high-level-folder\sub-folder\";  

S3DirectoryInfo di = new S3DirectoryInfo(client, _bucketName, path);  
IS3FileSystemInfo[] files = di.GetFileSystemInfos();  
foreach (S3FileInfo file in files)  
{  
    Console.WriteLine($"{file.Name}");  
}  

// result:  
// test.txt  

Unlike low-level API, here the folder name (sub-folder) is not listed.

Deleting file/folder

In the following code first we delete test.txt and then sub-folder:

// delete test.txt file
string filePath = @"high-level-folder\sub-folder\test.txt";

S3FileInfo s3File = new S3FileInfo(client, _bucketName, filePath);
if (s3File.Exists)
{
    s3File.Delete();
}

// delete sub-folder
string folderPath = @"high-level-folder\sub-folder\";

S3DirectoryInfo directory = new S3DirectoryInfo(client, _bucketName, folderPath);
if (directory.Exists)
{
    directory.Delete();
}

Wrapping up

When designing information systems, we follow a practice called single source of truth (SSOT), it ensures that every data element is edited in only one place. SSOT simplifies the information system and makes working with it a lot easier. Personally I like to extend this practice to every aspect of design... when designing a UI, there is no point in giving users 2 different ways to buy a product, it complicates the UI and confuses the user... in my opinion, the same is true for designing a software library. I think by providing 2 different APIs (low-level and high-level), AWS has unnecessarily complicated the process of communicating with the S3 bucket, especially because these APIs use different paths systems... personally I spent hours on an error because I was using a high-level style path, with low-level APIs.

Now, to make the matter worse, AWS is providing yet another way of interacting with S3 buckets, and that is using AWS TransferUtility which runs on top of low-level API and is the recommended way for reading/writing large objects (larger than a few GB). Have a look at this AWS documentation and see which one is the best option for your needs.