DEV Community

Cover image for Five S3 functionalities that are only available through AWS CLI/SDK
Sri
Sri

Posted on • Updated on

Five S3 functionalities that are only available through AWS CLI/SDK

#aws #cloud

During my learning, I discovered five S3 functionalities that are only available through the AWS CLI:

Table of Contents

  1. Configuring S3 MFA delete
  2. S3 Pre-signed URLs
  3. Upload files to S3 Glacier Vault
  4. S3 Multi Part Upload
  5. S3 Access point through VPC

1. Configuring S3 MFA delete

When working with S3 Versioning in Amazon S3 buckets, you can optionally add another layer of security by configuring a bucket to enable MFA (multi-factor authentication) delete. When you do this, the bucket owner must include two forms of authentication in any request to delete a version or change the versioning state of the bucket.

MFA delete requires additional authentication for either of the following operations:

👉 Changing the versioning state of your bucket

Suspend bucket versioning
👉 Permanently deleting an object version

Delete a versioned object

MFA delete requires two forms of authentication together:

  • Your security credentials

  • The concatenation of a valid serial number, a space, and the six-digit code displayed on an approved authentication device

In order to enable MFA, we need to follow the steps below.

Configure AWS Client

Create an AWS access key and then execute aws configure to set up your credentials.
📓 This is not best practice for security purposes but we will use this for the current exercise only.

Command to enable MFA

📓 213849 is the authentication token

[ec2user@some-ip ~]$ aws s3api put-bucket-versioning --profile default --bucket testbucketmfa --versioning-configuration Status=Enabled,MFADelete=Enabled --mfa "arn:aws:iam::XXXXXXXXXXXX:mfa/account-mfa-device 213849"
Enter fullscreen mode Exit fullscreen mode

Command to disable MFA 

[ec2user@some-ip ~]$ aws s3api put-bucket-versioning --profile default --bucket testbucketmfa --versioning-configuration Status=Enabled,MFADelete=Disabled --mfa "arn:aws:iam::XXXXXXXXXXXX:mfa/account-mfa-device 987543"
Enter fullscreen mode Exit fullscreen mode

2. S3 Pre-signed URLs

All objects in S3 are private by default. Only the object owner has permission to access these objects. However, the object owner can optionally share objects with others by creating a presigned URL, using their own security credentials, to grant time-limited permission to download the objects.

The commands below are using AWS CLI. We can also generate pre-signed URLs using AWS SDK.

aws s3 presign s3://testbucket202119/smiley.jpg --region ap-southeast-2
Enter fullscreen mode Exit fullscreen mode

The above command will generate a URL, for example:

https://testbucket202119.s3.amazonaws.com/smiley.jpg?AWSAccessKeyId=AKIAYYDMCK6YRXWASPX2&Expires=1639901714&Signature=VGcqq8ilnCtkd8OTFJP4aMidqI4%3D
Enter fullscreen mode Exit fullscreen mode

3. Upload files to S3 Glacier Vault

S3 Glacier Vault is a container for storing archives.

Command to create a vault 

aws glacier create-vault --vault-name testvault --account-id [AccountId]
Enter fullscreen mode Exit fullscreen mode

Command to upload an archive to a vault 

aws glacier upload-archive --account-id [AccountId] --vault-name testvault --body HappyFace.jpg
Enter fullscreen mode Exit fullscreen mode

To delete a non-empty vault

👉 If deleting a non-empty vault you must first delete all existing archives before deleting the vault. The commands below are using AWS CLI. We can also do this using Rest API or AWS SDK.

Use the initiate-job command to start an inventory retrieval job.

aws glacier initiate-job --vault-name testvault  --account-id [AccountId] --job-parameters '{"Type": "inventory-retrieval"}'
{
  ...
  "jobId": "Nm1J8J2qnjgiT68k09gCbSSm2wG_IsBFwlxYSQo2JsAbDgwCV0nPy-Sxcc5BeUnQF2y13HWF3zmWvj6wPK5sIlzZOh45"
}
Enter fullscreen mode Exit fullscreen mode

Use the describe-job command to check the status of the previous retrieval job or enable notifications on the vault to get notified.

aws glacier describe-job --vault-name testvault --account-id [AccountId] --job-id "jobId from the above output"
{
    "CompletionDate": "2021-12-20T03:53:59.230Z", 
    "VaultARN": "arn:aws:glacier:ap-southeast-2:[AccountId]:vaults/testvault", 
    "InventoryRetrievalParameters": {
        "Format": "JSON"
    }, 
    "Completed": true, 
    "InventorySizeInBytes": 445, 
    "JobId": "Nm1J8J2qnjgiT68k09gCbSSm2wG_IsBFwlxYSQo2JsAbDgwCV0nPy-Sxcc5BeUnQF2y13HWF3zmWvj6wPK5sIlzZOh45", 
    "Action": "InventoryRetrieval", 
    "CreationDate": "2021-12-20T00:01:20.715Z", 
    "StatusMessage": "Succeeded", 
    "StatusCode": "Succeeded"
}
Enter fullscreen mode Exit fullscreen mode

It took me approximately 3 hours to retrieve the file.
When it's complete, use the get-job-output command to download the retrieval job to the file output.json.

aws glacier get-job-output --vault-name testvault --account-id [AccountId] --job-id "Nm1J8J2qnjgiT68k09gCbSSm2wG_IsBFwlxYSQo2JsAbDgwCV0nPy-Sxcc5BeUnQF2y13HWF3zmWvj6wPK5sIlzZOh45" output.json
Enter fullscreen mode Exit fullscreen mode 
cat output.json 
{
   "VaultARN":"arn:aws:glacier:ap-southeast-2:[AccountId]:vaults/testvault",
   "InventoryDate":"2021-12-19T21:20:06Z",
   "ArchiveList":[
      {
         "ArchiveId":"HW0qIFuG4o6Ov4CGm8RpbzBFgftorVdKUSx5yBXssKg2wo5vqvXJwtyds29T86ALW3LmtOjtsLymoqh073gq2QBHr0Nitc3ot4HCu-LPOlkoHIhCtx6xU_JdvH8v9NFEMvsThpPJfA",
         "ArchiveDescription":"",
         "CreationDate":"2021-12-19T09:00:59Z",
         "Size":131281,
         "SHA256TreeHash":"f2216ef309ad918a2b3286652d5b5be8877f81a8d13181058f11d7d28f12c180"
      }
   ]
}
Enter fullscreen mode Exit fullscreen mode

👉 S3 Glacier prepares an inventory for each vault, about once every 24 hours. So we can only delete the vault after 24 hours as there should not have been any writes since the last inventory.

Use the delete-archive command to delete each archive from a vault until none remain.

aws glacier delete-archive --vault-name testvault --account-id [AccountId] --archive-id  “archiveid from the above output"
Enter fullscreen mode Exit fullscreen mode

You can find more information at Deleting an Archive in Amazon S3 Glacier Using the AWS Command Line Interface

4. S3 Multi Part Upload

Multipart upload allows you to upload a single object as a set of parts. Each part is a contiguous portion of the object's data. You can upload these parts in any order.
In general, when your object size reaches 100 MB, you should consider using multipart uploads instead of uploading the object in a single operation.

We can either use s3 or s3 api. You can find more information at How do I use the AWS CLI to perform a multipart upload of a file to Amazon S3?

5. S3 Access point through VPC

Amazon S3 Access Points, a feature of S3, simplify data access for any AWS service or customer application that stores data in S3. With S3 Access Points, customers can create unique access control policies for each access point to easily control access to shared datasets.

The S3 console doesn't support accessing bucket resources using a virtual private cloud (VPC) access point. To access bucket resources from a VPC access point, use the AWS CLI, AWS SDK, or Amazon S3 REST API.

S3 Access point through VPC

Acknowledgements

Please let me know your thoughts in the comments.

Top comments (0)

Read next

mahinur05 profile image

🚀 Deploying Node.js Application with PM2, NGINX, and SSL Configuration 🚀

Mahinur Rahman -

muash10 profile image

Automate Cleaning of Unused EIP Through Lambda Part1

Muhammed Ashraf -

094459 profile image

AWS open source newsletter, #204

Ricardo Sueiras -

balwanishivam profile image

Setting Up an S3 Bucket in LocalStack

Shivam Balwani -