Monitoring AWS Managed Services using Elastic

Over the course of years the number of cloud native services provided by the cloud platforms have been ever increasing. The platforms do come in with their inbuilt monitoring services but from an administrative standpoint of view it is not that helpful as it does not come with a consolidated view for the end users. To overcome this and to provide a common platform for various service logs and its corresponding visualisation many third party providers have come into the play such as Elastic, Datadog, Splunk etc.

In this article we will be seeing the use of one such service provider: Elastic. Elastic provides out of the box dashboards to visualise AWS managed services such as Loadbalancers, VPC, S3, CloudTrail and Billing. We will be using a sample Web application hosted on AWS cloud to provide how consolidated dashboards can be viewed in Kibana.

Monitoring Dashboards for AWS Services

Once we complete the exercise, below are the set of Dashboards that would be available for the Infrastructure Monitoring team to visualise how AWS services are functioning, identify and troubleshoot problems pertaining to any of the services.

Deployment Architecture

Below is a typical web application deployed on AWS cloud that uses a combination of cloud native services and web application server to serve APIs. We also have a static site hosted on Cloudfront to serve static html/css content.

Metricbeat

Metricbeat is a lightweight shipper that you can install on your servers to periodically collect metrics (cpu usage, Disk I/O, network bytes in/out ) from the operating system and from services running on the server. Metricbeat takes the metrics and statistics that it collects and ships them to the output that you specify, such as Elasticsearch or Logstash. Metricbeat by default supports various pre-built modules but in this article we will be focusing on the AWS module.

Metricbeat collects two broad categories of metrics i.e. host metrics and managed services metrics. We will be dedicating one EC2 machine to run metricbeat for AWS managed services metrics collection and another metricbeat on EC2 machine to demonstrate metrics collected from the host server.

NOTE: We will be focusing on metricbeat agent from 7.4 and later

AWS Module of Metricbeat

The AWS module for metricbeat currently supports out of the box collection of metrics for the following AWS Services:

• EC2
• Elastic Loadbalancer
• Lambda functions
• NAT Gateway
• RDS
• S3 Storage
• SNS
• SQS
• Transit Gateway
• VPN
• Billing

Before we dive into installing and configuring metricbeat, let's understand how metricbeat collects, stores and sends metrics to Elastic.

1. Enable Cloudwatch metrics for each of the AWS Managed services. This will start sending AWS services to CloudWatch, from where Metricbeat will be reading. This documentation gives step by step instructions on how to enable Cloudwatch for a specific AWS service.
2. Metricbeat will be running on a EC2 machines and configured to collect AWS managed services metrics. This is as simple as enabling the AWS module in metricbeat config file.
3. Hands-off, Metricbeat periodically queries AWS Cloudwatch to read metrics and sends it to Elasticsearch server, where it is indexed and visualised in Kibana.

Prerequisites:

Running a Metricbeat requires the following:

1. AWS account with credentials. Below section covers the Roles required for Metricbeat to read metrics.
2. Running Elastic Stack (use you self hosted Elastic or create a free 14-day trial on Elastic Cloud). This article doesn’t work with AWS managed elasticsearch.
3. EC2 machine to run Metricbeat agent

IAM policy:

Metricbeat requires certain IAM Policy permissions for it to fetch data from the required resources. An IAM policy is an entity that defines permissions to an object within your AWS environment. Create a customized IAM policy for Metricbeat with specific permissions is needed. Please see Creating IAM Policies for more details. After Metricbeat IAM policy is created, you need to add this policy to the IAM user which provided the credentials in the previous step.

The following table shows the IAM Policies that needs to be added to each Metricbeat

Consolidated IAM Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeRegions",
                "ec2:DescribeInstances"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:GetMetricData",
                "cloudwatch:ListMetrics"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "sts:GetCallerIdentity"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:ListAccountAliases"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "tag:getResources"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "rds:DescribeDBInstances",
                "rds:ListTagsForResource"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "sns:ListTopics"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "sqs:ListQueues"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ce:GetCostAndUsage"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Metricbeat Agent:

Installing the Metricbeat Agent on EC2 machine:

Launch a EC2 machine that runs Ubuntu and login to that machine and run below command to download and install metricbeat.

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.11.1-amd64.deb
sudo dpkg -i metricbeat-7.11.1-amd64.deb

Edit the configuration:

Elastic Cloud:

Modify /etc/metricbeat/metricbeat.yml to set the connection information for Elastic Cloud:

cloud.id: <Get your cloud_id from the Elastic Cloud>
cloud.auth: "elastic:<password>"

Enable the AWS module:

In the out-of-box configuration of Metricbeat, only the system module is enabled by default, so you will need to explicitly enable the AWS module. The following command enables the AWS configuration in the modules.d directory on MacOS and Linux systems:

metricbeat modules enable aws

Set AWS credentials in the config file:

To configure AWS credentials, users can put the credentials into the Metricbeat module configuration or use environment variables to pass them. The ability to load AWS credentials from a shared credentials file is added into aws module. Create a shared credential by following these steps. File will be created under the below location.

For Windows:
  C:\Users\<yourusername>\.aws\credentials
For  Linux, MacOS, or Unix:
  ~/.aws/credentials

(Optional) Users can specify the profile name using parameter credential_profile_name in aws module config. For more details on AWS credentials types and supported formats, please see AWS credentials configuration for more detail.

Sample AWS module Configuration:

metricbeat.modules:
- module: aws
  period: 300s
  metricsets:
    - ec2
- module: aws
  period: 300s
  metricsets:
    - sqs
  regions:
    - us-west-1
- module: aws
  period: 86400s
  metricsets:
    - s3_request
    - s3_daily_storage
- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/EC2
      name: ["CPUUtilization"]
      statistic: ["Average"]    
    - namespace: AWS/EBS
    - namespace: AWS/ELB
      resource_type: elasticloadbalancing
- module: aws
  period: 60s
  metricsets:
    - elb
    - natgateway
    - rds
    - transitgateway
    - usage
    - vpn

• namespace: A namespace in AWS CloudWatch is a container for metrics from a specific application or service. Each service has its own namespace, for example Amazon EC2 uses AWS/EC2 namespace and Amazon Elastic Block Storage uses AWS/EBS namespace. Please see the full list of services and namespaces that publish CloudWatch metrics for more details. name: Users can specify what are the specific CloudWatch metrics that need to be collected per namespace.
• dimensions: Dimensions are used to refine metrics returned for each instance. For example, InstanceId, ImageId and InstanceType all can be used as dimensions to filter data requested from Amazon EC2.
• statistic: Users can specify one or more statistic methods for each CloudWatch metric setting. By default, average, count, maximum, minimum and sum will all be collected for each metric.
• tags: resource_type_filter: Tags for resources will not be collected unless this parameter is set. Each resource in AWS has a specific resource type and the common format is service[:resourceType]. Please see resource type filters for more details.
• credential_profile_name: If the aws credentials config is done it will be automatically picked up, else one can enter the details manually.

Start Metricbeat:

The setup command loads the Kibana dashboards. If the dashboards are already set up, omit this command.

sudo metricbeat setup
sudo service metricbeat start

Module Status:

Metricbeat comes with pre-built Kibana dashboards and UIs for visualizing log data. The dashboards would have been loaded earlier when the setup command was run.

In the Kibana side navigation:

1. Click Discover, to see Metricbeat data. Also make sure the predefined metricbeat-* index pattern is selected.
2. Click Dashboard, then search for "AWS metric*", select the dashboard that you want to open.

Filebeat

Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that you specify, collects log events, and forwards them either to Elasticsearch or Logstash for indexing. Filbeat by default supports various pre-built modules but in this article we will be focusing on the AWS module.

AWS Module of Filebeat:

The AWS module for filebeat currently supports the following:

• CloudTrail
• CloudWatch
• EC2
• Elastic Loadbalancer
• S3 Access
• VPC Flow logs

Prerequisites:

Running a filebeat requires the following:

1. Configure AWS Services to send logs to S3. All files can be placed on the same bucket.
   • CloudTrail logs to S3
   • VPC flow logs to S3
   • Elastic Loadbalancer Access logs to S3
   • Enable S3 Access Logs
2. Setting up the SQS service in the AWS Account. (SQS Service). This SQS service will be used to notify filebeat when new file is placed in S3 bucket configured in step 1. AWS account with credentials.
3. Running Elastic Stack. (This can either be a cluster in 4. Elasticsearch Service on Elastic Cloud)
4. Filebeat agent

Filebeat Agent:

Installing the Filebeat Agent

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.11.1-amd64.deb
sudo dpkg -i filebeat-7.11.1-amd64.deb

Configure

1. Modify /etc/filebeat/filebeat.yml to set the connection information for Elastic Cloud:

cloud.id: <Get your cloud_id from the Elastic Cloud>
cloud.auth: "elastic:<password>"

1. Enable S3 to send a notification to SQS when a new file is placed in bucket.

Steps to Enable S3 Event notification to SQS are documented here.

Enable the AWS module

From the configuration of Filebeat, one will need to explicitly enable the AWS module. The following command enables the AWS configuration in the modules.d directory on MacOS and Linux systems:

sudo filebeat modules enable aws

Set AWS credentials in the config file

(Skip, if this has already been done during metricbeat setup)
To configure AWS credentials, users can put the credentials into the Filebeat module configuration or use environment variables to pass them. Create a shared credential by following these steps. File will be created under the below location.

For Windows:
C:\Users\<yourusername>\.aws\credentials

For  Linux, MacOS, or Unix:
~/.aws/credentials

(Optional) Users can specify the profile name using parameter credential_profile_name in aws module config. For more details on AWS credentials types and supported formats, please see AWS credentials configuration for more detail.

Sample AWS Configuration for filebeat:

- module: aws
  cloudtrail:
    enabled: true
    var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
  cloudwatch:
    enabled: true
    var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
  ec2:
    enabled: true
    var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
  elb:
    enabled: true
    var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
  s3access:
    enabled: true
    var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
  vpcflow:
    enabled: true
    var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue

• var.queue_url: (Required) AWS SQS queue url.

Start Filebeat:

The setup command loads the Kibana dashboards. If the dashboards are already set up, omit this command.

sudo filebeat setup
sudo service filebeat start

Module Status:

Filebeat comes with pre-built Kibana dashboards and UIs for visualizing log data. The dashboards would have been loaded earlier when the setup command was run.

In the Kibana side navigation:

1. Click Discover, to see Filebeat data. Also make sure the predefined filebeat-* index pattern is selected.
2. Click Dashboard, then search for "AWS Filebeat*", select the dashboard that you want to open.