.NET Core API Integration with AWS S3 and VMWare S3 broker

5 min readSep 23, 2020

This article is about .NET Core API that uses AWS SDK to interact with AWS S3 or VMWare S3 broker for managing the documents in S3. It covers the following topics.

S3 Client-side encryption with the Master key stored in AWS KMS or a Master key maintained outside AWS.
APIs to Upload & Download documents with versions and API to list documents with version information.
S3 storage classes, Server access logging & Object-level logging.

AWS S3

First, create a S3 private bucket, an IAM user, and grant the user with S3 read&write access. Refer doc for details.

S3 Client-side encryption with a Master key maintained outside AWS.

Add AWS S3 “AWSSDK.S3” & “AWSSDK.Core” Nuget packages in the startup project. Configure the AmazonS3EncryptionClient with accessKey, secret access key, and RSA key as follows. You can get the access key and secret access key from the AWS console when user is created. Refer to the file for additional details.

Inject the Client factory in the startup.cs as follows.

Add the following code in the Attachment controller to upload the document into S3 using AWS SDK.

A document can be uploaded to S3 using Postman as follows. This document will be encrypted by using symmetric key generated by AmazonS3 EncryptionClient . The symmetric key is encrypted by using the RSA key before it gets uploaded to S3. This process is called “Envelope Encryption”. The document can only decrypted by the client which has the RSA Key.

S3 bucket view in AWS management console after multiple document are uploaded through the API.

You can view the list of document through the below APIs.

API to list the all the documents in a bucket . http://<<host>>/document/list.

Use this endpoint in Postman as follows.

API to list all versions of a specific document. http://<<host>>/document/listversions/<<DocumentName>>

Use this endpoint in Postman as follows.

API to download the document with version. http://<<host>>/document/<<documentName>>/<<version>>

Use this endpoint in Postman as follows.

S3 Client-side encryption with Master key stored in AWS KMS

In the above approach, we managed the master key/ RSA Key in the App. It’s API owner’s responsibility to secure the key and roate it regularly. Instead of that you can crate the master key (CMK) using KMS APIs as follows. The advantage is that keys are rotated&secured by AWS. Refer this for additional detailts.

First, Add “”AWSSDK.KeyManagementService”” Nuget package and grant KMS permission to the IAM user in KMS (Key management system in AWS console ).

Generate the KMS key id using the KMS SDK as follows. you can refer to this file. That’s it, all the above APIs should work without any issues.

VMware Tanzu S3 broker

If you are using VMWare then you need not to manage IAM user and configuring S3 bucket by yourself. It will be available as tile/broker, you just need to bind it to your APP as shown below. It will be very simple since you don’t need to know AWS concepts.

You can get the private S3 bucket details like bucket name, accessKeyId and secretAccessKey from the VCap service as follows.Command to get the VCap service details of the App is as follows. That’s it, all the above APIs should work without any issues.

cf env <<AppName>>

Limitations: If you want to use KMS, S3 access logs or s3 Life cycle management then you may not find such options in s3 broker.

S3 Server access logging & Object-level logging

CloudTrail logs provide you with detailed API tracking for Amazon S3 bucket-level and object-level operations, while server access logs for Amazon S3 provide you visibility into object-level operations on your data in Amazon S3.

S3 Lifecycle Management:

If S3 bucket has any life cycle management rules to changes the storage class from standard to Glacier (as shown below) then objects can’t be retrieved through the API. First, you need to restore the object to standard storage class and use the API to download the object.

You can download an object through API if it is in any one of the storage classes. 1) Standard-IA 2) Intelligent-Tiering 3)Standard.