Installing AWS CloudWatch Agent on On-Premises Servers Using SSM

Introduction

In this guide, we'll walk you through the process of installing the AWS CloudWatch Agent on on-premises servers using AWS Systems Manager (SSM). This is particularly useful for those managing hybrid environments where both on-premises servers and cloud-based resources are monitored using AWS CloudWatch.

To make this guide practical, we'll simulate an on-premises server using an EC2 instance created in another AWS account. We'll cover everything from registering the on-premises server with SSM, installing the CloudWatch Agent, configuring it, and then using it to collect and send logs to CloudWatch.

What You'll Learn

• How to register an on-premises Debian server with AWS Systems Manager using a Hybrid Activation.
• How to install and configure the CloudWatch Agent on the server.
• How to send logs from your server to AWS CloudWatch

Prerequisites

• An AWS Account with necessary permissions.
• A Debian-based Server (simulated using an EC2 instance created in a different AWS Account for this guide).

Step 1: Create an SSM Hybrid Activation

Before registering your on-premises server with AWS Systems Manager, you need to create a Hybrid Activation. This step will provide you with an Activation Code and Activation ID, which are required to register your server.

1. Navigate to the Systems Manager Console

Log in to the AWS Management Console and open the Systems Manager console.

2. Create a New Hybrid Activation

• In the Systems Manager navigation pane, choose Hybrid Activations under Node Management.
• Click on Create Activation.

• Fill in the following details:

  • Activation Description: Provide a meaningful description, like "On-Premises Server Registration".
  • Instance Limit: Set the number of on-premises servers you want to register.
  • IAM Role: Choose or create an IAM role that has the necessary permissions for Systems Manager.
  • Registration Expiration Date: Set the expiration date for this activation, after which it can no longer be used.

• Click Create Activation.

3. Save the Activation Code and Activation ID

After creating the activation, you'll receive an Activation Code and Activation ID. Make sure to note these down, as you'll need them later to register your on-premises server.

Step 2: Onboard a Debian Server to AWS Systems Manager (SSM)

With your Hybrid Activation in hand, you can now register your Debian server with AWS Systems Manager.

1. Update Your Package List

Start by updating your server's package list:

sudo apt-get update

2. Install the SSM Agent

Next, download and install the SSM Agent:

mkdir /tmp/ssm
cd /tmp/ssm
wget https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/debian_amd64/amazon-ssm-agent.deb
sudo dpkg -i amazon-ssm-agent.deb

3. Register the Server with SSM

Register your server using the Activation Code, Activation ID, and your AWS region:

sudo amazon-ssm-agent -register -code "<your-activation-code>" -id "<your-activation-id>" -region "<your-region>"

For example:

sudo amazon-ssm-agent -register -code "h7FfWBbOrDCeXexxxxxx" -id "914e2266-e1c1-4c3a-b638-2azzzzzzzzzz" -region "eu-central-1"

4. Start the SSM Agent

Once registered, start the SSM Agent:

sudo systemctl start amazon-ssm-agent

5. Enable the SSM Agent to Start on Boot

Ensure the agent starts automatically on boot:

sudo systemctl enable amazon-ssm-agent

6. Verify the SSM Agent Status

Finally, confirm that the agent is running:

sudo systemctl status amazon-ssm-agent

Your Debian server should now be successfully registered with AWS Systems Manager, making it manageable through the AWS Management Console.

Step 3: Install and Configure Nginx (For Log Collection)

To generate some logs for the CloudWatch Agent, let’s install Nginx on the Debian server.

Install Nginx

sudo apt update
sudo apt install nginx
sudo systemctl status nginx

The Nginx log files that we’ll be sending to Cloudwatch are:

/var/log/nginx/error.log
/var/log/nginx/access.log


Step 4: Install CloudWatch Agent Using SSM

Now, let’s use the SSM Agent to install the CloudWatch Agent on our server.

1. Access the Systems Manager Console

Open the Systems Manager console at AWS Systems Manager Console.

2. Run the Command to Install CloudWatch Agent

• Navigate to Run Command.
• Select AWS-ConfigureAWSPackage from the list of Command documents.
• Choose the on-premises server as the target.
• Set Action to Install.
• Enter AmazonCloudWatchAgent in the Name box.
• Leave the Version field blank to install the latest version.
• Choose Run.

The CloudWatch Agent will now be installed on your server.

Step 5: Configure CloudWatch Agent

To enable the CloudWatch Agent to send logs from your on-premises server to AWS CloudWatch, you need to set up an IAM user with the necessary permissions, configure your server to use this IAM user's credentials, and ensure that the CloudWatch Agent is properly configured to use these credentials.

1. Create an IAM User with the Necessary Permissions

First, you'll need to create an IAM user that has permissions to send logs to CloudWatch.

Steps to Create the IAM User :

1. Log in to the AWS Management Console and open the IAM console.

2. Create a New User:

• Navigate to Users and click on Add user.
• Enter a user name (e.g., CloudWatchAgentUser).
• Under Access type, select Programmatic access to generate an access key ID and secret access key for this user.

3. Assign Permissions:

• Click on Attach policies directly.

• Attach the following managed policies to the user:

  CloudWatchAgentServerPolicy
CloudWatchAgentAdminPolicy
AmazonSSMManagedInstanceCore

• These policies grant the necessary permissions to send logs to CloudWatch, access SSM, and interact with the CloudWatch Agent.

4. Complete the User Creation:

• Proceed to review and create the user.
• On the final page, make sure to download the .csv file containing the Access Key ID and Secret Access Key, or copy them to a secure location. You’ll need these credentials in the next step.

2. Configure the Server with IAM User Credentials

Now that you have the Access Key ID and Secret Access Key, you need to configure your server to use these credentials by creating an AWS CLI profile named AmazonCloudWatchAgent.

Configure AWS CLI with the IAM User Credentials:

1. On your on-premises server, run the following command to configure the AWS CLI with the IAM user credentials:

sudo aws configure --profile AmazonCloudWatchAgent

1. When prompted, enter the following details:

• AWS Access Key ID: Enter the Access Key ID you obtained earlier.
• AWS Secret Access Key: Enter the Secret Access Key.
• Default region name: Enter the region where you want the logs to be sent (e.g., eu-central-1).
• Default output format: Leave this field blank or enter json.

This creates a profile named AmazonCloudWatchAgent on your server that the CloudWatch Agent will use to send logs to AWS CloudWatch.

3. Update the CloudWatch Agent Configuration

If you're simulating an on-premises environment using an EC2 instance, you might need to update the CloudWatch Agent configuration file (common-config.toml) to use the newly created profile.

Update the Configuration File:

1. Open the common-config.toml file:

sudo nano /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

1. Uncomment and update the following section to include the profile name and credentials file:

[credentials]
shared_credential_profile = "AmazonCloudWatchAgent"
shared_credential_file = "/root/.aws/credentials"

1. Save and exit the file.

Step 6: Start CloudWatch Agent with a Pre-Created Config File Using SSM

In this step, we’ll use a configuration file created in advance and saved in the AWS Systems Manager Parameter Store to start the CloudWatch Agent.

1. Create and Save the Configuration File in SSM Parameter Store

First, create a CloudWatch Agent configuration file on your local machine. Here’s an example configuration:

{
  "agent": {
    "metrics_collection_interval": 60,
    "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log"
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/nginx/access.log",
        "log_group_class": "INFREQUENT_ACCESS",
            "log_group_name": "{instance_id}-nginx-access.log",
            "log_stream_name": "{instance_id}",
            "retention_in_days": 7
          },
          {
            "file_path": "/var/log/nginx/error.log",
            "log_group_class": "INFREQUENT_ACCESS",
            "log_group_name": "{instance_id}-nginx-error.log",
            "log_stream_name": "{instance_id}",
            "retention_in_days": 7
          }
        ]
      }
    }
  }
}

Once your configuration file is ready, save it to the AWS Systems Manager Parameter Store:

aws ssm put-parameter --name "CloudWatchAgentConfig" --type "String" --value file://configuration_file_pathname

Replace configuration_file_pathname with the actual path to your configuration file.

For more information about how to create cloudwatch agent configuration file, please visit Link

2. Access the Systems Manager Console

Return to the Systems Manager console.

3. Run the Command to Start CloudWatch Agent

• Navigate to Run Command.
• Select AmazonCloudWatch-ManageAgent from the Command documents.
• Select the on-premises server as the target.
• Set Action to configure.
• Set Mode to onPremise.
• In the Optional Configuration Location box, enter the name of the configuration file stored in the Parameter Store (e.g., CloudWatchAgentConfig).
• Choose Run.

The CloudWatch Agent will now start with the specified configuration.

Verify CloudWatch Agent Logs
You can monitor the CloudWatch Agent's logs by running:

sudo tail -f /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

Note on KMS Encryption for Log Groups

At the moment, the CloudWatch Agent does not support KMS encryption for log groups during their creation. The workaround is to allow the agent to create the log groups first and then manually associate them with a KMS key using the following command:

aws logs associate-kms-key --log-group-name LOG_GROUP_NAME --kms-key-id KEY_ARN

There is an ongoing feature request regarding this issue, which you can track here.

Conclusion

I created this blog post as a practical reference for anyone who needs to register on-premises servers with AWS Systems Manager (SSM) and install the CloudWatch Agent using SSM. Instead of having to sift through a whole bunch of AWS documentation each time you need to perform these tasks, you can use this guide to streamline the process. By following the steps outlined here, you can quickly and efficiently onboard your servers to SSM and configure the CloudWatch Agent to monitor and log your system's activity, ensuring you maintain visibility and control over your infrastructure, whether it's on-premises or in the cloud.

Additional References

• SSM Agent PreInstalled AMI: Link
• SSM Supported OS and Machine Types: Link
• Install SSM Agent on Debian: Link
• Check SSM Agent Status : Link