DEV Community

Allen D. Ball
Allen D. Ball

Posted on • Edited on • Originally published at blog.hcf.dev

AWS EC2 User Data Shell Script

#aws #ebs #ec2 #ansible

Creators of Amazon Elastic Compute Cloud (EC2) instances may stuff a script into the "user data" which will be executed on the instance's initial boot. This script may be useful to mount Elastic Block Store (EBS) volumes as file systems since those volumes cannot be attached to the instance until after the instance has been created and started.

This article presents a script which leverages the functions provided in the aws.rc script described in a previous article.

The "user data" script described herein provides the following services to Redhat and CentOS instances:

  1. Update OS software:

    a. yum update

    b. Install/Update Python

    c. Install AWS CLI

  2. Create users, install their SSH authorized keys, and configure sudo

  3. Attach volumes, create file systems (if neccessary), mount, and update /etc/fstab

AWS Configuration

The scripts require specific configuration in AWS. These requirements are described in the next subsections.

Users

The script will configure all users specified in the instance's http://169.254.169.254/latest/meta-data/public-keys/. These key pairs must be imported either through AWS Management Console or the AWS CLI. The name of the key pair must be the same as the user name.

A primary use case is to create and configure ec2-user on CentOS images to be consistent with Amazon's Linux images.

EBS File System Volumes

Any EBS volume that will be mounted as a file system must be configured with the following tags:

  • host
  • fstype
  • mntpt

Where host is the http://169.254.169.254/latest/meta-data/local-ipv4 address of the newly created instance, fstype is a file system type compatible with the mkfs(8) and mount(8) commands'-t argument, and mntpt is the local directory on which the file system will be mounted.

Once the EBS volume is formatted with a valid file system, the volume's uuid tag should be updated with the file system's UUID. The user-data.bash script will not format the volume if the uuid tag is present. The user-data.bash script will update the volume's uuid tag if it successfully creates a file system on the volume.

Finally, the script will configure /etc/fstab to mount the EBS volume on the mntpt directory.

Theory of Operation

The script:

  1. Update the operating system software
  2. Configures the users whose keys are specified in http://169.254.169.254/latest/meta-data/public-keys/
  3. Attaches, formats, and mounts any EBS volumes tagged with this instance's http://169.254.169.254/latest/meta-data/local-ipv4 address

The corresponding parts of the user-data.bash script are described in detail in the following subsections.

Software Update

The software update consists of:

  1. Updating all packages managed by yum(8)
  2. Install and update python(1)
  3. Install the AWS Command Line Interface 
export LANG=en_US.UTF-8
export LC_ALL=${LANG}

yum -y update
yum -y install python
easy_install --prefix /usr pip
pip install --prefix /usr --upgrade pip
pip install --prefix /usr --upgrade awscli
Enter fullscreen mode Exit fullscreen mode

Users Configuration

For each public key specified in http://169.254.169.254/latest/meta-data/public-keys/:

  1. If they do not exist, create the user specified by the key name and create that user's home directory
  2. Add the openssh-key value to the user's ~/.ssh/authorized_keys
  3. Configure the user in sudoers(5) 
for key in $(metadata public-keys/); do
    username=${key#*=}

    useradd -G wheel -m -s /bin/bash -U ${username}
    userhome=$(eval echo ~${username})

    mkdir -p ${userhome}/.ssh
    echo "$(metadata public-keys/${key%%=*}/openssh-key)" \
         >> ${userhome}/.ssh/authorized_keys
    chown -R ${username}:${username} ${userhome}/.ssh
    chmod -R go-rwx ${userhome}/.ssh

    file=/etc/sudoers.d/user-data-${username}
    echo "${username} ALL=(ALL) NOPASSWD:ALL" > /Users/ball/hcf-dev/blog/2018-08-22-aws-user-data-script/pom.xml
    chmod a-wx,o-r ${file}
done
Enter fullscreen mode Exit fullscreen mode

Attach and Mount EBS Volumes

For each EBS volume tagged with host equalling the value at http://169.254.169.254/latest/meta-data/local-ipv4, a non-nil fstype, and non-nil mntpt:

  1. Attach the EBS volume to this instance
  2. Test if the volume contains data known to file(1) and, if it does not, create a file system of type specified by fstype
  3. Determine the UUID of the file system and add an entry to /etc/fstab 
export HOST=$(metadata local-ipv4)
export VOLUMES=$(ec2 describe-volumes \
                       --filters Name=tag:host,Values=${HOST} \
                       --output text --query 'Volumes[*].VolumeId')

if [ -n "${VOLUMES}" ]; then
    for volume in ${VOLUMES}; do
        fstype=$(ec2-get-tag-value ${volume} fstype)

        if [ "${fstype}" != "" ]; then
            device=$(next-unattached-block-device)

            ec2-attach-volume ${volume} ${device}

            if [ "$(file -b -s ${device})" == "data" ]; then
                volume-mkfs ${volume} ${device} ${fstype}
            fi

            uuid=$(ec2-get-tag-value ${volume} uuid)
            mntpt=$(ec2-get-tag-value ${volume} mntpt)

            if [ "${uuid}" != "" -a "${mntpt}" != "" ]; then
                mkdir -p ${mntpt}
                echo "UUID=${uuid} ${mntpt} ${fstype} defaults 0 2" >> /etc/fstab
            fi
        fi
    done
fi

mount -a

exit 0
Enter fullscreen mode Exit fullscreen mode

The file systems are then mounted with mount -a.1

systems were then connected on start-up.

user-data.bash

For reference, the complete user-data.bash Ansible template is included below. The aws.rc script is included through a relative path. If another tool than Ansible is used, the aws.rc script must be included to provide the functions through that tool's appropriate mechanism.

#!/bin/bash
# ----------------------------------------------------------------------------
# user-data.bash
# ----------------------------------------------------------------------------
export LANG=en_US.UTF-8
export LC_ALL=${LANG}

if [ -e /usr/bin/apt ]; then
    apt -y update
    apt -y install awscli
else
    yum -y update

    if [ ! -e /usr/libexec/platform-python ]; then
        yum -y install python
    fi

    if [ ! -e /usr/bin/pip -a -x /usr/bin/easy_install ]; then
        /usr/bin/easy_install --prefix /usr pip
    fi

    /usr/bin/pip install --prefix /usr --upgrade pip
    /usr/bin/pip install --prefix /usr --upgrade awscli
fi
# ----------------------------------------------------------------------------
# Functions
# ----------------------------------------------------------------------------
{{ lookup('template', '../../aws-rc/templates/etc/aws.rc') }}
# ----------------------------------------------------------------------------
# Create users and install respective .ssh/authorized_keys for public-keys'
# metadata
# ----------------------------------------------------------------------------
getent group sudo >> /dev/null 2>&1

if [ $? -eq 0 ]; then
    wheel=sudo
else
    wheel=wheel
fi

for key in $(metadata public-keys/); do
    username=${key#*=}

    useradd -G ${wheel} -m -s /bin/bash -U ${username}
    userhome=$(eval echo ~${username})

    mkdir -p ${userhome}/.ssh
    echo "$(metadata public-keys/${key%%=*}/openssh-key)" >> ${userhome}/.ssh/authorized_keys
    chown -R ${username}:${username} ${userhome}/.ssh
    chmod -R go-rwx ${userhome}/.ssh

    file=/etc/sudoers.d/user-data-${username}
    echo "${username} ALL=(ALL) NOPASSWD:ALL" > ${file}
    chmod a-wx,o-r ${file}
done
export HOST=$(metadata local-ipv4)
export VOLUMES=$(ec2 describe-volumes \
                       --filters Name=tag:host,Values=${HOST} \
                       --output text --query 'Volumes[*].VolumeId')

if [ -n "${VOLUMES}" ]; then
    for volume in ${VOLUMES}; do
        fstype=$(ec2-get-tag-value ${volume} fstype)

        if [ "${fstype}" != "" ]; then
            device=$(next-unattached-block-device)

            ec2-attach-volume ${volume} ${device}

            if [ "$(file -b -s ${device})" == "data" ]; then
                volume-mkfs ${volume} ${device} ${fstype}
            fi

            uuid=$(ec2-get-tag-value ${volume} uuid)
            mntpt=$(ec2-get-tag-value ${volume} mntpt)

            if [ "${uuid}" != "" -a "${mntpt}" != "" ]; then
                mkdir -p ${mntpt}
                echo "UUID=${uuid} ${mntpt} ${fstype} defaults 0 2" >> /etc/fstab
            fi
        fi
    done
fi

mount -a

exit 0
Enter fullscreen mode Exit fullscreen mode

Within Ansible, the user-data.bash can be expanded from the template
into a "fact:"

- name: user-data script
  set_fact:
    user_data: >
      {{ lookup('template', 'user-data.bash') }}
Enter fullscreen mode Exit fullscreen mode

An example fragment for creating the EC2 instance with the user-data.bash
script is given below.

- name: 172.31.0.4
  community.aws.ec2_instance:
    ...
    instance_profile_name: ec2-user
    key_name: ec2-user
    user_data: >
      {{ user_data }}
    ...
Enter fullscreen mode Exit fullscreen mode

References

  1. Previous versions of this script rebooted the instance and the file ↩

Top comments (0)

Read next

justin_luzano_6b2348d7dc6 profile image

Technical Documentation Blog For AWS Services: AWS Lambda

Justin Luzano -

mohit0303 profile image

Your ML/AI Success Begins Here: Data Ingestion & Storage on AWS

mohit mishra -

zirkonium88 profile image

AWS Architectural Diagrams on a Commit Base: Using AWS PDK Diagram Plugin with Python

Dr. Malte Polley -

mayuri2103 profile image

AWS SIMULEARN - 1

Mayuri G -