Remote Development with AWS Cloud9 IDE using a Docker and Docker-Compose application

In this tutorial, we will be setting up Telescope in AWS Cloud9 IDE. Telescope is a tool for tracking blogs around Seneca's open source develop. The application itself has many microservices that use Docker containers and are composed together using Docker-Compose. You can find the GitHub repo here

Telescope Tech Stack

• Next.js
• Material UI
• Node.js
• Elasticsearch
• Redis
• Traefik
• Firebase
• Jest

Telescope Microservices

• Auth
• Feed-Discovery
• Image
• Parser
• Planet
• Posts
• Search
• Status
• Users

AWS Cloud9 is a cloud-based integrated development environment (IDE) that lets you write, run, and debug your code with just a browser.

Disclaimer: The EC2 instance used in this guide is not within AWS's Free-Tier so please see EC2 Pricing to see if you're comfortable with these costs. Cloud9 has a cost-saving setting to help reduce costs by automatically hibernating after 30 minutes of inactivity. Running Docker in development is CPU intensive so these are the EC2 instances I recommend:

• Minimum: t2.medium (4 GiB RAM + 2 vCPU)
• Recommended: t2.large (8 GiB RAM + 2 vCPU)

Summary of Pricing:

• t2.medium costs \$0.0464 per hour
• t2.large costs \$0.0928 per hour
• 30GB Amazon Elastic Block Storage (EBS) costs \$3 per month

Cost Estimate Per Month:

Prerequisites:

• Create an AWS Account. You can watch this part of the AWS Certified Cloud Practitioner course on creating an account if you need help.
• Sign into your AWS Account

Creating your Cloud9 Environment:

1. In the upper-right hand corner of your AWS Management Console, select US East (Ohio) us-east-2 as your Region
2. In the upper-left hand corner of your AWS Management Console, click on Services. This is bring up a list of AWS Services, search for Cloud9.
3. Click on Create Environment

• Step 1 - Name environment:

Name: Telescope-Dev (whatever you want)

Description (optional): AWS Cloud9 development environment for Telescope

• Step 2 - Configure settings:

Environment type: Create a new EC2 instance for environment (direct access)

Instance type: Other instance type: t2.medium (4 GiB RAM + 2 vCPU)

Platform: Ubuntu Server 18.04 LTS

• Step 3 - Review and click Create Environment

It will take a few minutes for AWS to create your new C9 environment

Opening the ports on our EC2 instance:

1. Firstly, we'll need the MAC address of our EC2 instance

$ curl -s http://169.254.169.254/latest/meta-data/mac

06:c5:aa:63:ee:f4

1. Using your EC2 instance's MAC address, we can get a list of Security Groups

$ curl -s http://169.254.169.254/latest/meta-data/network/interfaces/macs/<your_mac>/security-group-ids

sg-0c63c6f026a2b9288

1. Find out what your IP address is using http://checkip.amazonaws.com/
2. You will need to authorize your IP address access to ports 3000, 8000, and 8443

aws ec2 authorize-security-group-ingress --group-id <sg-id> \
--port 3000 \
--protocol tcp \
--cidr <my-ip>/32

aws ec2 authorize-security-group-ingress --group-id <sg-id> \
--port 8000 \
--protocol tcp \
--cidr <my-ip>/32

aws ec2 authorize-security-group-ingress --group-id <sg-id> \
--port 8443 \
--protocol tcp \
--cidr <my-ip>/32

Resize your Amazon EBS volume

Check first using df -h in the terminal

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      9.7G  9.3G  371M  97% /

When you first create an EC2 instance, it has an EBS Volume of 10GB. To increase it to 20GB, create a new file called resize.sh in ~/environment directory and copy the following script:

#!/bin/bash

# Specify the desired volume size in GiB as a command-line argument. If not specified, default to 20 GiB.
SIZE=${1:-20}

# Get the ID of the environment host Amazon EC2 instance.
INSTANCEID=$(curl http://169.254.169.254/latest/meta-data/instance-id)

# Get the ID of the Amazon EBS volume associated with the instance.
VOLUMEID=$(aws ec2 describe-instances \
  --instance-id $INSTANCEID \
  --query "Reservations[0].Instances[0].BlockDeviceMappings[0].Ebs.VolumeId" \
  --output text)

# Resize the EBS volume.
aws ec2 modify-volume --volume-id $VOLUMEID --size $SIZE

# Wait for the resize to finish.
while [ \
  "$(aws ec2 describe-volumes-modifications \
    --volume-id $VOLUMEID \
    --filters Name=modification-state,Values="optimizing","completed" \
    --query "length(VolumesModifications)"\
    --output text)" != "1" ]; do
sleep 1
done

# Check if we're on an NVMe filesystem
if [ $(readlink -f /dev/xvda) = "/dev/xvda" ]
then
  # Rewrite the partition table so that the partition takes up all the space that it can.
  sudo growpart /dev/xvda 1

  # Expand the size of the file system.
  # Check if we are on AL2
  STR=$(cat /etc/os-release)
  SUB="VERSION_ID=\"2\""
  if [[ "$STR" == *"$SUB"* ]]
  then
    sudo xfs_growfs -d /
  else
    sudo resize2fs /dev/xvda1
  fi

else
  # Rewrite the partition table so that the partition takes up all the space that it can.
  sudo growpart /dev/nvme0n1 1

  # Expand the size of the file system.
  # Check if we're on AL2
  STR=$(cat /etc/os-release)
  SUB="VERSION_ID=\"2\""
  if [[ "$STR" == *"$SUB"* ]]
  then
    sudo xfs_growfs -d /
  else
    sudo resize2fs /dev/nvme0n1p1
  fi
fi

Disclaimer: AWS Free-Tier includes 30GB of Storage, 2 million I/Os, and 1GB of snapshot storage with Amazon Elastic Block Store (EBS) free for 12 months.

In the terminal, execute the script by running

sh resize.sh

Verify size change with df -h again

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       20G  9.3G   11G  48% /

Install Docker-Compose

By default, Docker is installed on AWS EC2's Ubuntu but Docker-Compose is not, so we have to install it ourselves.

1. Run to download the current stable version of Docker-Compose:

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

1. Apply executable permissions to the downloaded file:

sudo chmod +x /usr/local/bin/docker-compose

1. Check installation using:

$ docker-compose --version

docker-compose version 1.29.2, build 5becea4c

Setting up the Telescope repository in Cloud9:

1. In the terminal, clone the Telescope repository and name the remote upstream by entering

git clone -o upstream https://github.com/Seneca-CDOT/telescope.git

1. Change to the telescope directory

cd telescope

1. Set all the necessary environment variables in your env.remote file to contain your EC2 instance's public IPv4 address by executing the aws-ip.sh script

sh ./tools/aws-ip.sh

If you did everything correctly, you've completed the environment setup using AWS Cloud9! Yay!

Now to get started with development...

1. Install all depenencies

npm install

1. Start all Telescope services. This will take some time to complete

docker-compose --env-file .env up -d

1. Start the Telescope development server on Port 3000

PORT=3000 npm start

1. Find your EC2 instance's public IPv4

$ curl -s http://169.254.169.254/latest/meta-data/public-ipv4

35.174.16.133

1. Open <public-ip>:8000 browser tab to see Telescope running on a AWS Cloud9 environment!

2. Open <public-ip>:3000/feeds in another browser tab to see all the feeds in the backend

3. Open <public-ip>:8443/v1/<microservice-port> in another browser tab to see the microservices. For example 35.174.16.133:8443/v1/posts

Frequently Asked Questions (FAQ)

What's in aws-ip.sh?

ipv4=$(curl http://169.254.169.254/latest/meta-data/public-ipv4)

sed -r \
  -e  "s@(.+=)http://localhost:8000(/[^ ]*)*@\1http://$ipv4:3000\2@g" \
  -e  "s@(.+=)http://localhost:3000(/[^ ]*)*@\1http://$ipv4:8000\2@g" \
  -e  "s@(.+=)http://localhost([^:]*)@\1http://$ipv4:8443\2@g" \
  -e  "s@development\.yml@gitpod\.yml@" \
  config/env.development > .env

How do I stop my docker containers?

npm run services:stop

How do I delete my docker containers?

docker system prune -af --volumes

I think I have multiple services using the same port, how I can check?

To see a list of all running services and what ports they're binding to

sudo lsof -i -P -n | grep LIST

I can't open :8000 running, what could I be doing wrong?

1. If you have a VPN on, turn it off and get your IP address by visiting http://checkip.amazonaws.com/ then allow your IP address to access the ports 3000 and 8000.

2. AWS may change your EC2 instance IP address when you stop or restart your EC2 instance. One solution is to purchase an Elastic IP address to reserve the particular public IP address. However, you can just clean out the env.remote and .env files and run the ./tools/aws-ip.sh script again to set your new EC2 IP address in the appropriate environment variables. Just remember to use the new EC2 IP address in the browser as well.

Resources

AWS Certified Developer - Associate 2020