Unit Testing using files from Amazon S3 in C#

Using actual files is not an ideal solution for unit testing. You should have some sort of interface that duplicates what's in a file. Unfortunately, the current solution I'm working on doesn't have an interface (yet!). I thought about storing the files in the project, but was concerned because the files we were testing would probably adjust, and I didn't want the size of the project to expand. In our project, we use NUnit but the concepts are similar across most unit testing tools. One other thing to note is that you will need the AWS SDK for .NET.

First we need a way to store the files in our unit tests and access them across multiple tests while without having to download them multiple times.

private List _s3Files;

[TestFixtureSetUp]
public void Init()
{
    _s3Files = GetFilesFromS3();
}

Now let's take a look at the GetFilesFromS3() method:

private IEnumerable<string> GetFilesFromS3()
{
    List files = new List(5);
    foreach (string fileKey in AmazonS3Utils.GetFilesFromBucket(ACCESS_KEY, SECRET_KEY, BUCKET))
    {
        //place them in a temporary directory
        string fileName = Path.GetTempPath() + fileKey;
        AmazonS3Utils.DownloadFile(ACCESS_KEY, SECRET_KEY, BUCKET, fileKey, fileName);
        files.Add(fileName);
    }
    return files;
}

Notice that you'll need to use your Amazon S3 access key, secret key, and the specific bucket to download the files from. As you can see, I created an Amazon S3 helper class for some of the functions to download the files. Here's the GetFilesFromBucket code:

public static List<string> GetFilesFromBucket(string accessKey, string secretKey, string bucketName)
{
    List<string> files = new List<string>(5);

    using (AmazonS3 client = Amazon.AWSClientFactory.CreateAmazonS3Client(accessKey, secretKey))
    {
        try
        {
            ListObjectsRequest request = new ListObjectsRequest();
            request.BucketName = bucketName;

            do
            {
                ListObjectsResponse response = new ListObjectsResponse();
                try
                {
                    //for some reason, I had to try catch just this method to get it to work
                    response = client.ListObjects(request);
                }
                catch { }

                // Process response.
                files.AddRange(response.S3Objects.Select(entry => entry.Key));

                // If response is truncated, set the marker to get the next
                // set of keys.
                if (response.IsTruncated)
                    request.Marker = response.NextMarker;
                else
                    request = null;
            } while (request != null);
        }
        catch (AmazonS3Exception amazonS3Exception)
        {
            if (amazonS3Exception.ErrorCode != null &&
                (amazonS3Exception.ErrorCode.Equals("InvalidAccessKeyId") ||
                 amazonS3Exception.ErrorCode.Equals("InvalidSecurity")))
            {
                Console.WriteLine("Check the provided AWS Credentials.");
            }
            else
            {
                Console.WriteLine(
                "Error occurred. Message:'{0}' when listing objects",
                amazonS3Exception.Message);
            }
        }
    }
    return files;
}

And now, here's the code to download a file from s3 using an access key, secret key, bucket name, and the output file name:

public static void DownloadFile(string accessKey, string secretKey, string bucketName, string key, string outputFileName)
{
    //only download if file doesn't exist
    if (File.Exists(outputFileName)) 
        return;

    AmazonS3Client client = new AmazonS3Client(accessKey, secretKey);

    GetObjectRequest request = new GetObjectRequest().WithKey(key).WithBucketName(bucketName);
    using (GetObjectResponse response = client.GetObject(request))
    using (FileStream fs = new FileStream(outputFileName, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None))
    using (Stream responseStream = response.ResponseStream)
        //if using .NET 4 or higher, we could just use stream.copy
        //since we are using .NET 3.5 we have a helper method to buffered copy
        responseStream.BufferedCopyTo(fs);
}

So now, we've successfully downloaded a list of files from an Amazon S3 bucket and stored the file names to be used for each of the tests we're running. Now we need to use them in a unit test:

[TestMethod]
public void ClassToTest_TestToRun()
{
    IEnumerable files = GetFilesFromS3();
    foreach (var file in files)
    {
        //do whatever you need to do
    }
}

This might not be an ideal use case for extremely large files because every time the project builds, it has to download the files and run the unit tests. The nice thing is that if the files are already downloaded, the code above doesn't need to download the files from S3.