The realm of open source log aggregators has seen significant growth over the years, with key players like ElasticSearch (2010) and Graylog (2016) dominating the field. More recently, Grafana Loki (2019) has emerged as a compelling addition to this ecosystem.
This article aims to facilitate a rapid onboarding process for log shipping using Promtail, log aggregation with Loki, and data visualization through Grafana. Collectively, this powerful trio of tools is commonly referred to as the PLG (Promtail-Loki-Grafana) stack.
The Server System
These instructions will work on Debian or Ubuntu system.
Using Vagrant (Virtualbox)
If you have Intel system, you can use Vagrant with Virtualbox to quickly bring up virtual servers on your workstation. Follow the instructions from respect sites to install these tools.
When ready, create a file called Vagrantfile
with the following contents:
Vagrant.configure("2") do |config|
config.vm.box = "generic/ubuntu2204"
config.vm.network "forwarded_port", guest: 3000, host: 3000, host_ip: "127.0.0.1"
config.vm.network "forwarded_port", guest: 3100, host: 3100, host_ip: "127.0.0.1"
end
NOTE: For this setup, the configuration will utilize 3000
for Grafana and 3100
for Loki on the localhost. If any services are currently running on these ports, you'll have to either halt those services or modify the host: key in the provided above configuration to use an alternative port.
When ready, you can bring up the system and log into the system with the following:
vagrant up
vagrant ssh
Installing Loki
On your desired system, such as the Vagrant managed virtual machine above, you can run the following to install PLG:
sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
sudo wget -q -O /usr/share/keyrings/grafana.key \
https://apt.grafana.com/gpg.key
# add a repository for stable releases
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" \
| sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y promtail loki grafana
Verify Services are running
sudo service loki status
sudo service grafana-server status
If any of these services are stopped, you can run:
sudo service loki start
sudo service grafana-server start
The Client Workstation
Visualization
You can access the login page by opening a web browser on your system and navigating to http://127.0.0.1:3000. Simply use the default credentials, admin for both the username and password. Upon login, you will be prompted to set a new password for the admin account.
After logging in, you can click on Explore, Loki, and then select on of the logs by clicking on Label.
Ship More Logs with Promtail
When setting up a new system, there may not be many interesting logs initially. However, your host workstation system is likely to have accumulated a wealth of interesting logs over time.
If you are running the server as a virtual machine using the previously mentioned Vagrantfile
configuration, you can easily ship logs from the laptop to localhost:3100
, where Loki is up and running. This allows you to leverage the abundant logs from your host workstation for analysis and visualization.
You can install Promtail locally as well. If the host system is running Debian or Ubuntu, you can run this:
# add a repository for stable releases
sudo wget -q -O /usr/share/keyrings/grafana.key \
https://apt.grafana.com/gpg.key
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" \
| sudo tee -a /etc/apt/sources.list.d/grafana.list
# install promtail
sudo apt-get update && sudo apt-get install -y promtail
By default, the configuration file /etc/promtail/config.yml
will set up Promtail to ship logs to the URL: http://localhost:3100/loki/api/v1/push
. This configuration will function correctly if you are using Vagrant with Virtualbox to run Loki. However, if you are not running the Loki server locally, you will need to modify the address (currently set to localhost
) in the configuration accordingly.
As an option you can modify the configuration with the following:
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
host: my_workstation
agent: promtail
__path__: /var/log/*log
And later restart the service:
sudo systemctl restart promtail
Top comments (0)