Building Your Own S3 Cloud Uploader CLI with Bash

Hey everyone! Today, I'm super excited to walk you through building your very own cloud uploader CLI using Bash. We'll cover everything from setting up your AWS S3 bucket to scripting the whole thing in Bash. I had an opportunity to also review a lot of peoples work and made mine a little bit different.

This project was completed and uploaded to this Github repository: Cloud Uploader Cli

Before we dive into the script, let’s talk a bit about Bash. Bash (Bourne Again Shell) is a Unix shell and command language. It’s widely available on various operating systems, especially on Linux. Shell scripting allows you to automate tasks by writing a series of commands in a script file. It’s super powerful for automating system administration tasks, managing file systems, and even developing simple CLI tools.

Getting Started with AWS S3

First things first, let's set up an S3 bucket on AWS. If you don't have an AWS account yet, go ahead and create one here. Once you're all set, follow these steps to create a bucket:

• Log into the AWS Management Console: Head over to the S3 section.

• Create a new bucket: Click on the "Create bucket" button. Give your bucket a unique name and choose a region that’s closest to you.

• Bucket settings: Type the preferred name of the bucket making sure it is unique. You can pretty much go with the default settings for now. Just click through and create the bucket.

Getting Your AWS Access Keys

Now that we have our S3 bucket, we need to get our AWS access keys to allow our Bash script to interact with AWS services. Here’s how:

• Navigate to IAM (Identity and Access Management): From the AWS Management Console, go to the IAM section.

• Create a new user: Click on create user on you top right in orange background. Then after, select a username of your choice.

• Set permissions: Select Attach policy directly and in the search box, search for s3. Select the policy "AmazonS3FullAccess".

• After click on your user created on the Users page. select Security credentials.

• Scroll down to find Access Keys then you create the access keys.
• Download the .csv file containing your access key ID and secret access key. Keep these credentials secure. DON'T SHARE!!!

The Cloud Uploader Script

Alright, let’s get to the fun part – writing our Bash script!
Make sure you have the AWS CLI installed and configured on your system. You can install it on Linux terminal with the following commands:

sudo apt-get update

sudo apt-get install awscli

After installing AWS cli, you have to connect yout terminal to you aws account. You can do that by doing the following:

• Run the AWS configure command

aws configure

• Enter your credentials

AWS Access Key ID [None]: YOUR_ACCESS_KEY_ID
AWS Secret Access Key [None]: YOUR_SECRET_ACCESS_KEY
Default region name [None]: YOUR_DEFAULT_REGION
Default output format [None]: json

Replace YOUR_ACCESS_KEY_ID, YOUR_SECRET_ACCESS_KEY, and YOUR_DEFAULT_REGION with your actual AWS credentials and preferred region.

That’s it. We are now connected to AWS. WE can go on to writing our bash script.
Also install pv as well. (you will understand it's use soon)

sudo apt-get install pv 

First create a bash script file that end in .sh eg: clouduploader.sh

1. Log Function: Logs events and errors to clouduploader.log for troubleshooting.

logfile="clouduploader.log"

log() {
    local message="$1"
    echo "$(date +'%Y-%m-%d %H:%M:%S') - $message" >> "$logfile"
}

1. AWS Configuration Check and Confirmation: Ensures AWS CLI is correctly configured and also has access to your S3 bucket.

# Read AWS credentials and configuration from environment variables or use defaults
AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}"
AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}"
AWS_DEFAULT_REGION="${AWS_DEFAULT_REGION}"
S3_BUCKET="${S3_BUCKET}"

# Function to check S3 configuration
check_s3_configuration() {
    echo "Checking S3 configuration..."
    if aws s3 ls "s3://$S3_BUCKET" &> /dev/null; then
        echo "S3 configuration is set correctly. The bucket $S3_BUCKET is accessible."
    else
        echo "Error: S3 configuration is incorrect or the bucket $S3_BUCKET is not accessible."
        exit 1
    fi
}

1. Argument check: Checks if the arguments passed are exactly 2 ie; the file path and destination path

if [ $# -ne 2 ]; then
    echo "Please provide exactly two parameters: the file path and the destination path."
    exit 1
fi

1. Assign 1st and 2nd arguments File path and destination path repectively and extract filename from file path

# Assign the first argument and second argument to FILEPATH and S3_DESTINATION respectively
FILEPATH="$1"
S3_DESTINATION="$2"

# Extract the filename from FILEPATH
file_name=$(basename "$FILEPATH")
echo "Filename: $file_name"

1. File Check: function to check if the file exists locally using -f

# Function to check if the file exists locally
file_check() {
    if [ -f "$FILEPATH" ]; then
        echo "$file_name exists."
    else
        echo "Error: $file_name does not exist."
        exit 1
    fi
}

1. File Upload: Uses AWS CLI to upload the file to the specified S3 destination with a progress bar. Note: "pv" already installed is a terminal-based (command-line based) tool in Linux that allows us for the monitoring of data being sent through pipe.

# Function to upload file to S3 with progress bar using pv
upload_to_s3() {
    local local_file="$1"
    local s3_destination="$2"

    # Upload the file to S3 using AWS CLI and capture output/error
    if pv "$local_file" | aws s3 cp - "$s3_destination"; then
        log "File upload successful: $local_file -> $s3_destination"
        echo "File upload successful: $local_file -> $s3_destination"
    else
        log "Error: Failed to upload file $local_file to $s3_destination"
        echo "Error: Failed to upload file $local_file to $s3_destination"
        exit 1
    fi
}

1. File Synchronization: Offers options to manage existing files in S3 (overwrite, skip, rename).

file_sync() {
    if aws s3 ls "$S3_DESTINATION$file_name"; then
        read -p "File already exists in S3. Overwrite, skip, or rename? [o/s/r]: " choice
        case $choice in
            o)
                upload_to_s3 "$FILEPATH" "$S3_DESTINATION"
                ;;
            s)
                echo "Skipping upload."
                ;;
            r)
                upload_to_s3 "$FILEPATH" "$S3_DESTINATION-renamed"
                ;;
            *)
                echo "Invalid choice. Skipping upload."
                ;;
        esac
    else
        upload_to_s3 "$FILEPATH" "$S3_DESTINATION"
    fi
}

# Perform file synchronization if needed
file_sync

1. Finally a function to generate a shareable link

presigned_url=$(aws s3 presign "$S3_DESTINATION$file_name" --expires-in 3600)
echo "Shareable link for the uploaded file: $presigned_url"

Running the Script

1. Save the script to to the created file, e.g., clouduploader.sh.

2. Make the script executable:
   

chmod +x clouduploader.sh

1. Run the script with the file path and S3 destination as arguments:

./clouduploader.sh /path/to/your/file s3://your-bucket-name/destination/

"/path/to/your/file" is your filepath and "s3://your-bucket-name/destination/" is your destination path

Advanced Features

• Progress Bar: The script uses pv to show a progress bar during file upload.
• Shareable Link: After a successful upload, the script generates a presigned URL for easy sharing.

Troubleshooting

• Permissions: Ensure your IAM user has the necessary permissions to access and upload to the S3 bucket.
• AWS CLI Configuration: Double-check your AWS CLI configuration and credentials.
• Log File: Check clouduploader.log for detailed logs and error messages.

Wrapping Up!

You've now got your very own cloud uploader CLI that can upload files to your AWS S3 bucket. This is just the tip of the iceberg! There’s plenty more you can do to enhance this script – think about adding more features.By following this process you will be able to upload your files to S3 seamlessly through your Command Line.

Bash is incredibly powerful, and mastering it can really up your automation game. Whether it's for system administration, file management, or building custom tools, Bash scripting opens up a world of possibilities. For those keen to dive deeper, check out all bash resources online.

Happy scripting, and keep exploring!

Enjoy trying it out and I am open for any feedback and ideas.