A lot of observability is done using cloud provider tools that are one subscription away, yet replicating them for local development can be challenging. At Woovi, we prioritize aligning our development environment with production. That's why we run locally all services available in production using Docker Compose. Observability is not an exception. In this article, we'll delve into the detailed setup of ELK observability tools in our local development environment.
ELK Tools
This are the tools we use for observability in our stack.
APM
Application performance monitoring.Elastic Search
Search Engine.Kibana
Visualization PlatformLogstash
Log ingestionMetricBeat
Server data shipperFilebeat
Log harvester and aggregatorFleet
Agents integration
Elastic Integrations offers an unified way to ingrate features of ELK stack, managed by fleet servers you can integrate over Kibana all integrations available using elastic agents, and manage them using policies trough an unified interface. In order to use this features we need security enable in kibana and elastic search and we need to provide certificate authentication for each service. In order to simplify we will be using the same certificate for all services, but this is only suited for local development and should not be used in production.
First we will need configuration files for some services, the file structure will be:
├── .env
├── docker-compose.yml
├──|observability |── filebeat.yml
├── |observability |── logstash.conf
└── |observability |── metricbeat.yml
Docker Compose
The docker compose will be responsible for setting up certificates and copying to the volume of each service, that way we can setup SSL between services. You can find the .env file on the github repo
version: "3.8"
services:
setup:
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes:
- certs:/usr/share/elasticsearch/config/certs
user: "0"
command: >
bash -c '
if [ x${ELASTIC_PASSWORD} == x ]; then
echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
exit 1;
elif [ x${KIBANA_PASSWORD} == x ]; then
echo "Set the KIBANA_PASSWORD environment variable in the .env file";
exit 1;
fi;
if [ ! -f config/certs/ca.zip ]; then
echo "Creating CA";
bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
unzip config/certs/ca.zip -d config/certs;
fi;
if [ ! -f config/certs/certs.zip ]; then
echo "Creating certs";
echo -ne \
"instances:\n"\
" - name: es01\n"\
" dns:\n"\
" - es01\n"\
" - localhost\n"\
" ip:\n"\
" - 127.0.0.1\n"\
" - name: kibana\n"\
" dns:\n"\
" - kibana\n"\
" - localhost\n"\
" ip:\n"\
" - 127.0.0.1\n"\
" - name: fleet-server\n"\
" dns:\n"\
" - fleet-server\n"\
" - localhost\n"\
" ip:\n"\
" - 127.0.0.1\n"\
> config/certs/instances.yml;
bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
unzip config/certs/certs.zip -d config/certs;
fi;
echo "Setting file permissions"
chown -R root:root config/certs;
find . -type d -exec chmod 750 \{\} \;;
find . -type f -exec chmod 640 \{\} \;;
echo "Waiting for Elasticsearch availability";
until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
echo "Setting kibana_system password";
until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
echo "All done!";
'
healthcheck:
test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
interval: 1s
timeout: 5s
retries: 120
es01:
depends_on:
setup:
condition: service_healthy
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
labels:
co.elastic.logs/module: elasticsearch
volumes:
- certs:/usr/share/elasticsearch/config/certs
- esdata01:/usr/share/elasticsearch/data
ports:
- ${ES_PORT}:9200
environment:
- node.name=es01
- cluster.name=${CLUSTER_NAME}
- discovery.type=single-node
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=certs/es01/es01.key
- xpack.security.http.ssl.certificate=certs/es01/es01.crt
- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=certs/es01/es01.key
- xpack.security.transport.ssl.certificate=certs/es01/es01.crt
- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.license.self_generated.type=${LICENSE}
mem_limit: ${ES_MEM_LIMIT}
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
test:
[
"CMD-SHELL",
"curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
]
interval: 10s
timeout: 10s
retries: 120
kibana:
depends_on:
es01:
condition: service_healthy
image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
labels:
co.elastic.logs/module: kibana
volumes:
- certs:/usr/share/kibana/config/certs
- kibanadata:/usr/share/kibana/data
- ./observability/kibana.yml:/usr/share/kibana/config/kibana.yml:ro
ports:
- ${KIBANA_PORT}:5601
environment:
- SERVERNAME=kibana
- ELASTICSEARCH_HOSTS=https://es01:9200
- ELASTICSEARCH_USERNAME=kibana_system
- ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
- ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
- XPACK_SECURITY_ENCRYPTIONKEY=${ENCRYPTION_KEY}
- XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${ENCRYPTION_KEY}
- XPACK_REPORTING_ENCRYPTIONKEY=${ENCRYPTION_KEY}
- XPACK_REPORTING_KIBANASERVER_HOSTNAME=localhost
- SERVER_SSL_ENABLED=true
- SERVER_SSL_CERTIFICATE=config/certs/kibana/kibana.crt
- SERVER_SSL_KEY=config/certs/kibana/kibana.key
- SERVER_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
- ELASTIC_APM_SECRET_TOKEN=${ELASTIC_APM_SECRET_TOKEN}
mem_limit: ${KB_MEM_LIMIT}
healthcheck:
test:
[
"CMD-SHELL",
"curl -I -s --cacert config/certs/ca/ca.crt https://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
]
interval: 10s
timeout: 10s
retries: 120
metricbeat01:
depends_on:
es01:
condition: service_healthy
kibana:
condition: service_healthy
image: docker.elastic.co/beats/metricbeat:${STACK_VERSION}
user: root
volumes:
- certs:/usr/share/metricbeat/certs
- metricbeatdata01:/usr/share/metricbeat/data
- "./observability/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro"
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro"
- "/proc:/hostfs/proc:ro"
- "/:/hostfs:ro"
environment:
- ELASTIC_USER=elastic
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- ELASTIC_HOSTS=https://es01:9200
- KIBANA_HOSTS=https://kibana:5601
- LOGSTASH_HOSTS=http://logstash01:9600
- CA_CERT=certs/ca/ca.crt
- ES_CERT=certs/es01/es01.crt
- ES_KEY=certs/es01/es01.key
- KB_CERT=certs/kibana/kibana.crt
- KB_KEY=certs/kibana/kibana.key
command: -strict.perms=false
filebeat01:
depends_on:
es01:
condition: service_healthy
image: docker.elastic.co/beats/filebeat:${STACK_VERSION}
user: root
volumes:
- certs:/usr/share/filebeat/certs
- filebeatdata01:/usr/share/filebeat/data
- "./filebeat_ingest_data/:/usr/share/filebeat/ingest_data/"
- "./observability/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro"
- "/var/lib/docker/containers:/var/lib/docker/containers:ro"
- "/var/run/docker.sock:/var/run/docker.sock:ro"
environment:
- ELASTIC_USER=elastic
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- ELASTIC_HOSTS=https://es01:9200
- KIBANA_HOSTS=https://kibana:5601
- LOGSTASH_HOSTS=http://logstash01:9600
- CA_CERT=certs/ca/ca.crt
command: -strict.perms=false
logstash01:
depends_on:
es01:
condition: service_healthy
kibana:
condition: service_healthy
image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
labels:
co.elastic.logs/module: logstash
user: root
volumes:
- certs:/usr/share/logstash/certs
- logstashdata01:/usr/share/logstash/data
- "./logstash_ingest_data/:/usr/share/logstash/ingest_data/"
- "./observability/logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro"
environment:
- xpack.monitoring.enabled=false
- ELASTIC_USER=elastic
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- ELASTIC_HOSTS=https://es01:9200
fleet-server:
depends_on:
kibana:
condition: service_healthy
es01:
condition: service_healthy
image: docker.elastic.co/beats/elastic-agent:${STACK_VERSION}
volumes:
- certs:/certs
- fleetserverdata:/usr/share/elastic-agent
- "/var/lib/docker/containers:/var/lib/docker/containers:ro"
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro"
- "/proc:/hostfs/proc:ro"
- "/:/hostfs:ro"
ports:
- ${FLEET_PORT}:8220
- ${APMSERVER_PORT}:8200
user: root
environment:
- SSL_CERTIFICATE_AUTHORITIES=/certs/ca/ca.crt
- CERTIFICATE_AUTHORITIES=/certs/ca/ca.crt
- FLEET_CA=/certs/ca/ca.crt
- FLEET_ENROLL=1
- FLEET_INSECURE=true
- FLEET_SERVER_ELASTICSEARCH_CA=/certs/ca/ca.crt
- FLEET_SERVER_ELASTICSEARCH_HOST=https://es01:9200
- FLEET_SERVER_ELASTICSEARCH_INSECURE=true
- FLEET_SERVER_ENABLE=1
- FLEET_SERVER_CERT=/certs/fleet-server/fleet-server.crt
- FLEET_SERVER_CERT_KEY=/certs/fleet-server/fleet-server.key
- FLEET_SERVER_INSECURE_HTTP=true
- FLEET_SERVER_POLICY_ID=fleet-server-policy
- FLEET_URL=https://fleet-server:8220
- KIBANA_FLEET_CA=/certs/ca/ca.crt
- KIBANA_FLEET_SETUP=1
- KIBANA_FLEET_USERNAME=elastic
- KIBANA_FLEET_PASSWORD=${ELASTIC_PASSWORD}
- KIBANA_HOST=https://kibana:5601
volumes:
elastic_data: {}
certs:
driver: local
esdata01:
driver: local
kibanadata:
driver: local
metricbeatdata01:
driver: local
filebeatdata01:
driver: local
logstashdata01:
driver: local
fleetserverdata:
driver: local
cache:
driver: local
networks:
default:
name: elastic
external: false
Its not as complicated as it looks, some of the things the compose file does by service are:
- Setup
Create an instance of an Elastic Search service to use on validation of the certificates. Then, via bash script we create the certificates using elasticsearch-certutil
, here we specify what are our cluster nodes and servers instances creating CA cert and node certs giving the correct file permissions and distributing the cert and keys. We test the authentication worked using elastic search service and then the service stops.
- ES01
Main Elastic Search service node, xpack.security options need to be enabled and pointing to the correct folder with the node certificate and certificate authority. A basic health check is passed using auth validation as main test parameter.
- Kibana
Kibana service, security config is the same as ES, xpack need to be enabled, extra configuration will be passed in its own yml file. To Access this file we pass it as bind read only volume.
./observability/kibana.yml:/usr/share/kibana/config/kibana.yml:ro
We add internal apm traces so once the initial setup go up we can verify kibana for itself.
- filebeat01
For filebeat we bind the volumes /var/lib/docker/containers
and /var/run/docker.sock
allowing filebeat to get all containers internal logs, the config in observability/filebeat.yml
will set autodiscover for that purpose, using docker as a provider, will also config kibana and ES. Ingest data input and ES outupt.
- metricbeat01
For Metricbeat the binding mounts are to allow getting host system metrics and information and sending to elastic search.
- logstash01
For logstash the bind mount config file will setup ingest plugins and output to elasti search. We are using basic starter configuration but in the config file is where you will setup formating. The ingest data path you can send the log file to be read using the strategy described in the config file.
- fleet-server
Fleet server will manage agents and its integrations.
Running Stack
1 - Run:
docker compose up -d
Wait untill is finished:
2 - Open kibana
Enter username and password setup in .env file
3 - Go to Fleet/Management
4 - Go to Settings
5 - Edit the output agent
You will need 3 items to configure the certificate.
Url: https://es01:9200
Elastic CA trusted Fingerprint:
To get this value you need to sign the certificate from ES container
Firtst copy the certificate to a temp folder
docker cp woovi-es01-1:/usr/share/elasticsearch/config/certs/ca/ca.crt /tmp/.
Then sign the certificate getting the Fingerprint
openssl x509 -fingerprint -sha256 -noout -in /tmp/ca.crt | awk -F"=" {' print $2 '} | sed s/://g
The output will be something like this:
5A7464CEABC54FA60CAD3BDF16395E69243B827898F5CCC93E5A38B8F78D5E7
Use the value on the field Elastic CA trusted Fingerprint
Now in Advanced YAML configuration you will need the certificate.
Print the certificate you copied from the container
cat /tmp/ca.crt
And edit to match the format on this example file
Its important not to input the wrong indentation, yml really sucks and this step is usually to blame when something goes wrong.
Save and apply settings, wait a few seconds and you should be able to see system metrics like memory and cpu usage on the agent page here
That indicates your fleet server is configured the correct way.
Now in your local development you only need to use apm config, go over on the agent configuration for APM in your desired language and check out, most are like the following:
{
serviceName: 'my-service-name',
secretToken: 'supersecrettoken',
serverUrl: 'http://localhost:8200',
environment: 'my-environment'
}
Check out for aditional information in this agent integration page
We're constantly seeking ways to automate and streamline processes. If you have any ideas on how to enhance this workflow, please don't hesitate to share them with us!
Woovi
Woovi is a Startup that enables shoppers to pay as they like. To make this possible, Woovi provides instant payment solutions for merchants to accept orders.
If you want to work with us, we are hiring!
Top comments (0)