Introduction to the AWS CLI

This post will walk you through setting up AWS CLI, an IAM administrator account for use with AWS CLI, and creating an AWS EC2 Ubuntu instance plus associated resources from the CLI.

Prerequisites

It is assumed that you:

• Possess a working familiarity with the Linux command line and able to perform troubleshooting tasks
• Have an AWS account
• Know the basics of AWS services. At the very least, you should be able to provision an AWS EC2 instance through the web interface and connect to it
• Are aware that following this hands-on session may incur monetary costs and you are solely responsible for any such costs

Introductory knowledge of cloud computing and the underlying technology (hypervisors, hardware virtualization, OS virtualization) would be beneficial.

With the prerequisites addressed, let's get started!

Preparation

Reference: Prerequisites to use the AWS CLI version 2

Log in to your AWS account. To use the AWS CLI securely, we need to set up an IAM(1) administrator account and associated access key(s) which will be used for authenticating against AWS and performing tasks through the CLI. Technically, you could set up access key(s) for your root account(2) and configure AWS CLI to use them, but AWS strongly recommends against it, for reasons similar to why you shouldn't log in to a Linux system directly as root and perform everyday tasks that way.

(1) Identity Access Management
(2) This is the account with an associated email address and password that you likely used to log in to the AWS web console

Visit the IAM console at https://console.aws.amazon.com/iam/ which should bring you to the dashboard. To the left of the dashboard under "Access management", click "Users":

This brings you to a page where you can view and manage IAM users. Near the top right hand corner is an "Add users" button:

Now you should see a form with two sections, "Set user details" and "Select AWS access type" (in that order). Under the first section, fill in "Administrator" as the username:

In the next section, make sure "Access key - Programmatic access" is checked. Depending on your preferences, you may also wish to enable password access for logging in (as the IAM administrator) through the web console:

Now click "Next: Permissions" near the bottom right hand corner:

This brings you to a page where you can "Set permissions" and "Set permissions boundary". The latter can be used to fine-tune the permissions of the IAM user which we will not cover here. For the former, ensure "Add user to group" is selected:

If you have not dealt with IAM before, you should have no IAM groups. Click "Create group" under "Get started with groups":

Now name your group "Administrators":

And ensure the "AdministratorAccess" policy is checked:

Click "Create group":

Back in "Add user to group", ensure your newly created group is checked:

If you can't see your created group, you may need to click "Refresh" until it appears. Now click "Next: Tags":

The following page allows you to associate tags (key-value pairs) with your IAM user. Since the IAM administrator is a general-purpose account that can perform almost all AWS tasks, there probably isn't much point associating it with tags, though a full discussion of tagging AWS resources and their uses can be found in https://docs.aws.amazon.com/general/latest/gr/aws_tagging.html .

Click "Next: Review":

Confirm that everything looks fine, then click "Create user" to confirm the action:

Now the IAM administrator is created. Notice that you have not yet returned to the dashboard. Instead, you are shown the access key ID and secret access key of the IAM administrator, and this is the only time they will be shown. Therefore, it is imperative that you take note of these values and store them in a safe place, such as by downloading the associated CSV file:

Now click "Close":

Congratulations! You have successfully created an IAM administrator account with access key ID and secret access key which can be used with AWS CLI in a later section.

At this point, if you weren't simply following the instructions blindly, you might be wondering what security benefits using an IAM administrator provides over using the root account directly, if the former has (almost) all of the permissions of the latter anyway. In fact, if an attacker gains access to your IAM administrator account, they can perform almost any action on your account, except for a few privileged actions restricted to the root account as documented in https://docs.aws.amazon.com/accounts/latest/reference/root-user-tasks.html , such as:

• Changing your account settings
• Allowing IAM users to view your billing details
• Closing your AWS account

So it's nearly as bad as having your root account compromised, but not quite. Either way, you should take great care not to leak your IAM administrator credentials, and definitely not your root account credentials! If it happens anyway, you should delete and recreate your IAM administrator account ASAP, with a new set of credentials.

Of course, if you know precisely what permissions are required for accessing a particular AWS service through the AWS CLI, you could (should) create a new IAM user with just those permissions to minimize the potential impact of leaking your IAM credentials to an attacker, but that is beyond the scope of this article.

Installing AWS CLI

Reference: Installing or updating the latest version of the AWS CLI

At the time of writing (December 2021), the latest version of AWS CLI is 2.x.x, though AWS CLI 1.x.x is still supported, with the former containing breaking changes that are not backward compatible with the latter. Here, we will install the newer version 2.x.x, and subsequent mentions of AWS CLI should be taken to imply 2.x.x unless otherwise noted.

AWS CLI is available on Linux, macOS and Windows. We'll assume here that you are installing AWS CLI onto a Linux environment:

• Linux: just follow the instructions directly in the terminal
• macOS: Spin up a Linux VM using your hypervisor of choice, e.g. VirtualBox, VMware Fusion or Parallels, and follow the instructions in the VM
• Windows: use WSL2, or follow a similar approach as on macOS

If you plan to follow this article in another environment, you may have to adapt the instructions accordingly.

On Linux, you can install AWS CLI by following the instructions on the official website, or on most mainstream distributions through the system package manager. However, depending on the nature of your distribution, the version of AWS CLI provided by your system package manager may be outdated, possibly up to a full major version behind. For example, running apt-cache show awscli | grep -i version on Ubuntu 20.04 LTS gives:

Version: 1.18.69-1ubuntu0.20.04.1
Version: 1.17.14-1

Therefore, in order to access the latest features of AWS CLI, it is highly recommended to install directly from the official website.

The installation is a simple 3-step process: fetching the zip archive from upstream, unpacking it, and executing the install script as root:

$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
$ unzip awscliv2.zip
$ sudo ./aws/install

If your machine uses ARM architecture, simply replace x86_64 in the first command with aarch64.

Updating AWS CLI is equally simple: follow the instructions above as with first install, but pass the --update flag to the last command, as follows:

$ sudo ./aws/install --update

Unfortunately, there is no uninstall script so if you wish to uninstall AWS CLI (assuming you did so through the official website) and know exactly what you are doing, you'll have to manually find and remove the aws binary and associated resources. By default, the binary is located under /usr/local/bin and resources under /usr/local/aws-cli.

Check which version of AWS CLI is installed:

$ aws --version
aws-cli/2.4.7 Python/3.8.8 Linux/5.11.0-43-generic exe/x86_64.ubuntu.20 prompt/off

In my case, AWS CLI v2.4.7 is installed.

Print a help message (not very useful, unfortunately):

$ aws --help
usage: aws [-h] [--profile PROFILE] [--debug]

optional arguments:
  -h, --help         show this help message and exit
  --profile PROFILE
  --debug

To learn more about the subcommands provided by AWS CLI, refer to the online documentation: https://awscli.amazonaws.com/v2/documentation/api/latest/index.html . Alternatively, as mentioned in online documentation, the proper way of obtaining useful help information directly from the AWS CLI is aws help or aws <command> help, which opens the top-level man page and an appropriate man page for the AWS command respectively.

Configuring AWS CLI to use our IAM administrator

Reference: Configuration basics - AWS Command Line Interface

The command to configure AWS CLI is (unsurprisingly) configure:

$ aws configure

This prompts you to enter 4 pieces of information:

• AWS Access Key ID: this is the access key ID of your IAM administrator. You should have saved this information in a CSV file when creating your IAM user
• AWS Secret Access Key: this is the secret access key of your IAM administrator. Again, refer to the CSV file you downloaded
• Default region name: recall that AWS resources are divided into regions and availability zones. Enter the name of the region you would like to use by default, if no region is explicitly specified. For example, I live in Hong Kong so I might set it to ap-east-1. You can get a list of region names through the web interface, and later through the aws ec2 describe-regions command
• Default output format: what type of output the AWS CLI should generate by default when most commands are run. For example, setting this to json or yaml causes AWS CLI to generate JSON and YAML output by default respectively which could be piped to another program or redirected to an output file for further processing. In this tutorial we will be using the AWS CLI interactively so we could set it to table for human-friendly output, though you can set it to whatever value you see fit

When the command completes, the information is saved in a default profile that AWS CLI uses for authentication, if no profile is explicitly specified. You can think of a profile as a particular named account, though there is nothing stopping you from creating two different named profiles with the same user credentials.

The information you entered is saved in two separate files, $HOME/.aws/config and $HOME/.aws/credentials, where the former stores non-sensitive values and the latter sensitive ones:

$ cat $HOME/.aws/config
[default]
region = ap-east-1
output = table
$ cat $HOME/.aws/credentials
[default]
aws_access_key_id = XXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

To configure a profile other than default, pass the --profile flag followed by the profile name, as follows:

$ aws configure --profile <profile>

To learn more about the configure command, run:

$ aws configure help

or refer to the online documentation.

Provisioning an AWS EC2 Ubuntu instance with AWS CLI

Reference: Using Amazon EC2 with the AWS CLI

The command for managing EC2-related services is ec2. Skim through the man page:

$ aws ec2 help

For example, we can view a list of available regions and their associated information:

$ aws ec2 describe-regions

View the output in YAML:

$ aws ec2 describe-regions --output yaml

Get the output in JSON and redirect to an output file, for use with another program at a later date:

$ aws ec2 describe-regions --output json > regions.json

View the availability zones in the default region:

$ aws ec2 describe-availability-zones

View the availability zones in the us-west-1 region:

$ aws ec2 describe-availability-zones --region us-west-1

View information on EC2 instances in the default region:

$ aws ec2 describe-instances

View information on security groups in the default region:

$ aws ec2 describe-security-groups

View information on key pairs in the default region:

$ aws ec2 describe-key-pairs

View information on instance types available in the default region:

$ aws ec2 describe-instance-types

Now that we've taken a look at the man page and tried out some commands for viewing EC2-related information, let's move on and provision a new instance with the AWS CLI. But before we continue, I've realized that I actually find table output difficult to parse and would prefer reading JSON output instead. Let's also change our default region to us-west-1 for performing experiments, as I have a production web server running in the ap-east-1 region (actually just my personal website) and want to minimize the chances of messing that up by accident.

To change the settings in the default profile, we just have to run configure again:

$ aws configure

For fields that you do not intend to change, simply leave it blank and press Enter. Here, I skip through access key ID and secret access key since they haven't changed, but enter us-west-1 for the default region name and json for the output format when prompted.

Now let's see what Ubuntu AMIs are available in our new default region. With the web interface, we can just search for the term "Ubuntu" and look at the results, and we can usually expect the first few results to contain what we are looking for. However, with the AWS CLI, if we just list all images without filtering them beforehand, it will take at least a few dozen seconds fetching all of the detailed information on all types of images (not just AMIs) and it would be infeasible to skim through the entire list to find what we are looking for. Therefore, take a look at the associated man page and I encourage you to try out various filtering options on your own:

$ aws ec2 describe-images help

After some experimentation on my own, here's what worked for me:

• The name of the AMI should contain "ubuntu" (with lowercase "u")
• The description should contain "20.04 LTS", so Ubuntu Bionic (18.04 LTS) images are not returned
• The image architecture should be x86_64
• The image type should be machine, so I don't get results for other non-AMI image types such as kernel and ramdisk images

Here's the command I used:

$ aws ec2 describe-images --filters \
  "Name=name,Values=*ubuntu*" \
  "Name=description,Values=*20.04 LTS*" \
  "Name=architecture,Values=x86_64" \
  "Name=image-type,Values=machine"

And here's the first result that I get:

{
    "Architecture": "x86_64",
    "CreationDate": "2021-04-16T06:48:46.000Z",
    "ImageId": "ami-000773770ac55efb2",
    "ImageLocation": "aws-marketplace/ubuntu-minimal/images/hvm-ssd/ubuntu-focal-20.04-amd64-minimal-20210416-d5944ad4-5199-4cf3-ab4c-c2c4598f880b",
    "ImageType": "machine",
    "Public": true,
    "OwnerId": "679593333241",
    "PlatformDetails": "Linux/UNIX",
    "UsageOperation": "RunInstances",
    "ProductCodes": [
        {
            "ProductCodeId": "cn75kij0iclp333uyrx6uqp6z",
            "ProductCodeType": "marketplace"
        }
    ],
    "State": "available",
    "BlockDeviceMappings": [
        {
            "DeviceName": "/dev/sda1",
            "Ebs": {
                "DeleteOnTermination": true,
                "SnapshotId": "snap-01325cd2ee26537f0",
                "VolumeSize": 8,
                "VolumeType": "gp2",
                "Encrypted": false
            }
        },
        {
            "DeviceName": "/dev/sdb",
            "VirtualName": "ephemeral0"
        },
        {
            "DeviceName": "/dev/sdc",
            "VirtualName": "ephemeral1"
        }
    ],
    "Description": "Canonical, Ubuntu Minimal, 20.04 LTS, amd64 focal image build on 2021-04-16",
    "EnaSupport": true,
    "Hypervisor": "xen",
    "ImageOwnerAlias": "aws-marketplace",
    "Name": "ubuntu-minimal/images/hvm-ssd/ubuntu-focal-20.04-amd64-minimal-20210416-d5944ad4-5199-4cf3-ab4c-c2c4598f880b",
    "RootDeviceName": "/dev/sda1",
    "RootDeviceType": "ebs",
    "SriovNetSupport": "simple",
    "VirtualizationType": "hvm"
}

This looks about right. Take a note of the image ID, that is, ami-000773770ac55efb2.

Now let's see what VPCs we have. This will be useful for creating an appropriate security group to be attached to our instance later.

$ aws ec2 describe-vpcs

In my case, I only have a default VPC with ID vpc-6d7aaf0b.

Even though we will create a new security group, let's see what security groups we already have anyway:

$ aws ec2 describe-security-groups

In my case, I only have a default security group, as follows:

{
    "Description": "default VPC security group",
    "GroupName": "default",
    "IpPermissions": [
        {
            "IpProtocol": "-1",
            "IpRanges": [],
            "Ipv6Ranges": [],
            "PrefixListIds": [],
            "UserIdGroupPairs": [
                {
                    "GroupId": "sg-bc3d9df5",
                    "UserId": "477657529190"
                }
            ]
        }
    ],
    "OwnerId": "477657529190",
    "GroupId": "sg-bc3d9df5",
    "IpPermissionsEgress": [
        {
            "IpProtocol": "-1",
            "IpRanges": [
                {
                    "CidrIp": "0.0.0.0/0"
                }
            ],
            "Ipv6Ranges": [],
            "PrefixListIds": [],
            "UserIdGroupPairs": []
        }
    ],
    "VpcId": "vpc-6d7aaf0b"
}

This security group blocks all inbound traffic (IpPermissions), so we won't be able to connect to our instance if we attach this security group to it, hence the need to create a new security group.

Skim through the man page:

$ aws ec2 create-security-group help

Now let's create a new security group "UbuntuSG", with the VPC ID we obtained earlier:

$ aws ec2 create-security-group \
  --group-name UbuntuSG \
  --description "Security group for Ubuntu 20.04" \
  --vpc-id vpc-6d7aaf0b

This prints a group ID on success. Take note of the group ID:

{
    "GroupId": "sg-047de2bb619edd924"
}

If we obtain information on our newly created security group with ec2 describe-security-groups, we notice that no inbound traffic is allowed:

{
    "Description": "Security group for Ubuntu 20.04",
    "GroupName": "UbuntuSG",
    "IpPermissions": [],
    "OwnerId": "477657529190",
    "GroupId": "sg-047de2bb619edd924",
    "IpPermissionsEgress": [
        {
            "IpProtocol": "-1",
            "IpRanges": [
                {
                    "CidrIp": "0.0.0.0/0"
                }
            ],
            "Ipv6Ranges": [],
            "PrefixListIds": [],
            "UserIdGroupPairs": []
        }
    ],
    "VpcId": "vpc-6d7aaf0b"
}

So we need to add inbound rules with ec2 authorize-security-group-ingress. Skim through the man page:

$ aws ec2 authorize-security-group-ingress help

Now add an inbound rule for SSH (22/tcp) from anywhere. For optimal security, you can discover your public IP address and only allow SSH from that IP instead:

$ aws ec2 authorize-security-group-ingress \
  --group-id sg-047de2bb619edd924 \
  --protocol tcp \
  --port 22 \
  --cidr 0.0.0.0/0
{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-04c28eddbcd8b598a",
            "GroupId": "sg-047de2bb619edd924",
            "GroupOwnerId": "477657529190",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 22,
            "ToPort": 22,
            "CidrIpv4": "0.0.0.0/0"
        }
    ]
}

Note that we used the group ID here. You can also specify the group name instead if that's easier.

Now our security group looks like this:

{
    "Description": "Security group for Ubuntu 20.04",
    "GroupName": "UbuntuSG",
    "IpPermissions": [
        {
            "FromPort": 22,
            "IpProtocol": "tcp",
            "IpRanges": [
                {
                    "CidrIp": "0.0.0.0/0"
                }
            ],
            "Ipv6Ranges": [],
            "PrefixListIds": [],
            "ToPort": 22,
            "UserIdGroupPairs": []
        }
    ],
    "OwnerId": "477657529190",
    "GroupId": "sg-047de2bb619edd924",
    "IpPermissionsEgress": [
        {
            "IpProtocol": "-1",
            "IpRanges": [
                {
                    "CidrIp": "0.0.0.0/0"
                }
            ],
            "Ipv6Ranges": [],
            "PrefixListIds": [],
            "UserIdGroupPairs": []
        }
    ],
    "VpcId": "vpc-6d7aaf0b"
}

Another thing we need to connect to an instance is a key pair. Currently, I do not have any key pairs:

$ aws ec2 describe-key-pairs
{
    "KeyPairs": []
}

In my case, I have to create or import a key pair. If you do not already have a key pair, generate one with ssh-keygen. Since I already have a key pair, with my public key at $HOME/.ssh/id_rsa.pub, I can import it into AWS using ec2 import-key-pair. Skim through the man page:

$ aws ec2 import-key-pair help

Now let's actually import the key:

$ aws ec2 import-key-pair \
  --key-name "MacBook Air Ubuntu 20.04" \
  --public-key-material fileb://$HOME/.ssh/id_rsa.pub

Remember the key name.

With a security group and key pair, we can finally provision our AWS EC2 Ubuntu instance and connect to it. The subcommand for creating and starting an instance is ec2 run-instances:

$ aws ec2 run-instances help

Let's use the following information:

• AMI ID: ami-000773770ac55efb2
• Instance type: t2.micro
• Security group ID: sg-047de2bb619edd924
• Key name: MacBook Air Ubuntu 20.04

$ aws ec2 run-instances \
  --image-id ami-000773770ac55efb2 \
  --instance-type t2.micro \
  --security-group-ids sg-047de2bb619edd924 \
  --key-name "MacBook Air Ubuntu 20.04"

Chances are you will receive an OptInRequired error like I did:

An error occurred (OptInRequired) when calling the RunInstances operation: In order to use this AWS Marketplace product you need to accept terms and subscribe. To do so please visit https://aws.amazon.com/marketplace/pp?sku=cn75kij0iclp333uyrx6uqp6z

To the best of my knowledge, it is not possible to subscribe to an AWS Marketplace product programmatically, possibly to force the user to read the terms and conditions properly before accepting it. Follow the link shown in the error message, subscribe to the product and retry the same command.

On success, you should see relevant JSON info printed to the console, such as the instance ID.

Now check that the instance is running:

$ aws ec2 describe-instances

For example, the state of my instance is:

{
    "Code": 16,
    "Name": "running"
}

I also learn that the public IP address of my instance is 52.53.151.210, from the PublicIpAddress field.

Now connect to the instance via SSH (replace 52.53.151.210 with the public IP of your instance):

$ ssh ubuntu@52.53.151.210

Congratulations! You have successfully provisioned an EC2 instance through the AWS CLI and connected to it.

Some additional subcommands for managing EC2 instances:

• ec2 stop-instances for stopping instances
• ec2 start-instances for starting existing instances that have been stopped
• ec2 reboot-instances for requesting a reboot on existing instances

Cleaning up

Unless you plan to continue using the instance, terminate it, replacing the instance ID below as appropriate:

$ aws ec2 terminate-instances --instance-ids i-02eb9bb3b0989c57e

Remember that we also created a security group and imported a key pair. You might want to keep them for use with future instances, but in my case, I will delete them as well:

$ aws ec2 delete-security-group --group-id sg-047de2bb619edd924
$ aws ec2 delete-key-pair --key-name "MacBook Air Ubuntu 20.04"

Conclusion

We learned what the AWS CLI is, how to install it on Linux, how to configure an IAM administrator for use with the AWS CLI (and why it is important to do so), how to get help with the AWS CLI, the ec2 command and its various subcommands, and how we can combine those commands to manage EC2 instances almost entirely from the CLI. But that is just the tip of the iceberg. AWS is so much more than just EC2, and I encourage you to take the time to explore other AWS services and how they can be manipulated through the AWS CLI.

I hope you enjoyed this hands-on session and found it useful. Stay tuned ;-) Until then, happy holidays!

References

• Prerequisites to use the AWS CLI version 2: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-prereqs.html
• Tagging AWS resources: https://docs.aws.amazon.com/general/latest/gr/aws_tagging.html
• Tasks that require root user credentials: https://docs.aws.amazon.com/accounts/latest/reference/root-user-tasks.html
• Installing or updating the latest version of the AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
• AWS CLI Command Reference: https://awscli.amazonaws.com/v2/documentation/api/latest/index.html
• Using Amazon EC2 with the AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/cli-services-ec2.html