Prometheus - Quick Setup on Amazon EC2 (Ubuntu) - Part 2

#prometheus #cloudnative #observability #opensource

In this article, we will look at how we can quickly setup Prometheus and Node Exporter on a Ubuntu instance on Amazon EC2

Prerequisites

Setup an EC2 instance of type t2.micro
Ubuntu 22.04 LTS as AMI
10 GB of hard disk space
Open ports 22 for SSH, 9090 for Prometheus and 9100 for Node Exporter

Installation

Prometheus

$ ssh -i <key_name>.pem ubuntu@<ip_address>

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:        22.04
Codename:       jammy

Download and extract the latest release from their GitHub page

$ wget https://github.com/prometheus/prometheus/releases/download/v2.43.0/prometheus-2.43.0.linux-amd64.tar.gz

$ tar -xzvf prometheus-2.43.0.linux-amd64.tar.gz
prometheus-2.43.0.linux-amd64/
prometheus-2.43.0.linux-amd64/LICENSE
prometheus-2.43.0.linux-amd64/consoles/
prometheus-2.43.0.linux-amd64/consoles/prometheus.html
prometheus-2.43.0.linux-amd64/consoles/node-disk.html
prometheus-2.43.0.linux-amd64/consoles/node-overview.html
prometheus-2.43.0.linux-amd64/consoles/prometheus-overview.html
prometheus-2.43.0.linux-amd64/consoles/index.html.example
prometheus-2.43.0.linux-amd64/consoles/node-cpu.html
prometheus-2.43.0.linux-amd64/consoles/node.html
prometheus-2.43.0.linux-amd64/prometheus
prometheus-2.43.0.linux-amd64/promtool
prometheus-2.43.0.linux-amd64/NOTICE
prometheus-2.43.0.linux-amd64/console_libraries/
prometheus-2.43.0.linux-amd64/console_libraries/prom.lib
prometheus-2.43.0.linux-amd64/console_libraries/menu.lib
prometheus-2.43.0.linux-amd64/prometheus.yml

Change the directory and run the Prometheus binary

$ cd prometheus-2.43.0.linux-amd64/

$ ./prometheus
ts=2023-04-28T14:52:07.254Z caller=main.go:520 level=info msg="No time or size retention was set so using the default time retention" duration=15d
ts=2023-04-28T14:52:07.255Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)"
ts=2023-04-28T14:52:07.255Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)"
ts=2023-04-28T14:52:07.255Z caller=main.go:570 level=info host_details="(Linux 5.15.0-1031-aws #35-Ubuntu SMP Fri Feb 10 02:07:18 UTC 2023 x86_64 ip-172-31-84-196 (none))"
ts=2023-04-28T14:52:07.255Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2023-04-28T14:52:07.255Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2023-04-28T14:52:07.257Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2023-04-28T14:52:07.258Z caller=main.go:1005 level=info msg="Starting TSDB ..."
ts=2023-04-28T14:52:07.261Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2023-04-28T14:52:07.261Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.434µs
ts=2023-04-28T14:52:07.261Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2023-04-28T14:52:07.264Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9090
ts=2023-04-28T14:52:07.264Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9090
ts=2023-04-28T14:52:07.265Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
ts=2023-04-28T14:52:07.265Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=36.679µs wal_replay_duration=3.153164ms wbl_replay_duration=721ns total_replay_duration=3.306296ms
ts=2023-04-28T14:52:07.267Z caller=main.go:1026 level=info fs_type=EXT4_SUPER_MAGIC
ts=2023-04-28T14:52:07.267Z caller=main.go:1029 level=info msg="TSDB started"
ts=2023-04-28T14:52:07.267Z caller=main.go:1209 level=info msg="Loading configuration file" filename=prometheus.yml
ts=2023-04-28T14:52:07.274Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=prometheus.yml totalDuration=6.593992ms db_storage=1.674µs remote_storage=2.431µs web_handler=939ns query_engine=1.285µs scrape=6.143405ms scrape_sd=30.548µs notify=32.577µs notify_sd=12.129µs rules=1.905µs tracing=7.357µs
ts=2023-04-28T14:52:07.274Z caller=main.go:990 level=info msg="Server is ready to receive web requests."
ts=2023-04-28T14:52:07.274Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..."

Open http://public_ip:9090 in the browser and you can see the Prometheus expression browser

Navigate to Status → Targets to view all targets configured in the configuration file

Navigate to Status → Configuration to view the configuration file contents

Navigate to Status → TSDB Status to view time-series database details

Open http://public_ip:9090/metrics in the browser to view metrics

Press ctrl+c to stop running the Prometheus server

Node Exporter

A popular exporter that collects system-level metrics from Linux and Unix-based systems
Provides a wide range of metrics that can be used to monitor system health
Different metrics are CPU usage, memory usage, disk usage, network statistics etc
Download and extract the latest release from their GitHub page

$ wget https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz

$ tar -xzvf node_exporter-1.5.0.linux-amd64.tar.gz
node_exporter-1.5.0.linux-amd64/
node_exporter-1.5.0.linux-amd64/LICENSE
node_exporter-1.5.0.linux-amd64/NOTICE
node_exporter-1.5.0.linux-amd64/node_exporter

Change the directory and run the Node Exporter binary

$ cd node_exporter-1.5.0.linux-amd64/

$ ./node_exporter
ts=2023-04-28T15:27:11.226Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)"
ts=2023-04-28T15:27:11.226Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)"
ts=2023-04-28T15:27:11.227Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/)
ts=2023-04-28T15:27:11.227Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
ts=2023-04-28T15:27:11.227Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$
ts=2023-04-28T15:27:11.228Z caller=node_exporter.go:110 level=info msg="Enabled collectors"
ts=2023-04-28T15:27:11.228Z caller=node_exporter.go:117 level=info collector=arp
ts=2023-04-28T15:27:11.228Z caller=node_exporter.go:117 level=info collector=bcache
ts=2023-04-28T15:27:11.228Z caller=node_exporter.go:117 level=info collector=bonding
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=btrfs
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=conntrack
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=cpu
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=cpufreq
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=diskstats
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=dmi
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=edac
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=entropy
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=fibrechannel
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=filefd
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=filesystem
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=hwmon
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=infiniband
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=ipvs
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=loadavg
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=mdadm
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=meminfo
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=netclass
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=netdev
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=netstat
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=nfs
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=nfsd
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=nvme
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=os
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=powersupplyclass
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=pressure
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=rapl
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=schedstat
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=selinux
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=sockstat
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=softnet
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=stat
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=tapestats
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=textfile
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=thermal_zone
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=time
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=timex
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=udp_queues
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=uname
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=vmstat
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=xfs
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=zfs
ts=2023-04-28T15:27:11.231Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100
ts=2023-04-28T15:27:11.232Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100