- In this tutorial you’ll see how to set up easily an ELK (Elastic, Logstash, Kibana) stack to have a centralized logging solution for your Docker swarm cluster. Install the stack Below you’ll find the full stack to have a working ELK stack on your docker swarm. Version: '3' services: elasticsearch: image: elasticsearch:7.9.3 environment.
- We are going to ru n the ELK stack in docker containers. This is so cool because containers are lightweight (again), increase isolation therefore security and can literally run anywhere. I will assume you’re familiar with basic docker concepts and you have docker-compose installed. So let’s get started!
The Elastic Stack (also known as the ELK Stack) is used across a variety of use cases — from observability to security, from enterprise search to business analytics. ELK is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. On this blog post, we will go through necessary steps to run elk using docker.
This post is about a log parser I quickly put together, with the help of the Elastic Search – Logstash – Kibana (ELK) stack and Docker.
The ELK Stack, in undoubtedly a phenomenal solution for analyzing centralized logging. Logstash beautifully breaks down every detail in the log and takes care of pushing them into ElasticSearch, which is then displayed by Kibana as neat charts.
In this case, we configure Logstash to additionally generate geographic data, based on an IP address we have. We also have an Nginx instance, which then forwards incoming requests to Kibana.
Setup Elk Stack Docker 2
Like I said, we use Docker for the entire setup, and so, the only thing installed on the host machine is Docker. Every other environment is spun up on demand, using configurations that can be persisted on the disk. Persisted on the disk is particularly important, because with Docker and modern day applications, an important benefit is that we can persist environments as configurations. Gone are the days when we have had to create environments and template them as VMWare / VBox images. Docker allows you to script every single aspect of an environment into files that can be checked-in into a source code repository such as GIT.
We start off by having Docker and Docker-Compose installed on the host machine. My go-to website to find server how-to’s has got to be DigitalOcean. The documentation they have is just unparalleled in my opinion. Here are two how-to’s on installing docker and docker-compose.
We start by creating a docker-compose.yml file that has the necessary information to bring up our environment.
To have a brief walk-through of the file, we have four services listed – nginx, elasticsearch, logstash and kibana.
The nginx service uses the nginx:latest image from Docker Hub. The service also creates a volume (mount) for a configuration file which ensures Nginx acts as a proxy to our kibana service. The Docker service also exposes port 80 to port 80 on the host machine.
elasticsearch is our next service. The service has a few labels that define its purpose. The image is defined to be elasticsearch:6.6.1 on the Docker Hub. Like nginx, and our other services, the image would be pulled from Docker Hub at on first bootup. The service defines a named volume called esdata. This is to ensure that the data we have in ElasticSearch is persisted on the host machine. k
Setup Elk Stack Docker Free
kibana our next service, in addition to the properties we have already seen, defines an environment variable that points to the elasticsearch service we already defined. A reference can be maintained across services, with the service name duplicating as a domain name within a virtual network created by Docker. Since requests are not expected to be directly consumed by Kibana, we expose no ports. The service finally has a depends_on declaration which tells Docker to bring up the dependency service, elasticsearch in this case, before spinning up Kibana.
logstash, our final service, has a couple of volumes mounted. One is a directory in which our .log files exist – within a sub-directory, logs. A configuration file is also mounted. The service also has an instruction that needs to be executed once the container is up. For the sake of completeness, a container can be said to be the executable representation of an image. Here is the configuration file mounted,
Let’s look at our logstash.conf in a little more detail. The type of logs being analyzed is a little more clear here. We are looking a download logs for various files. The configuration file primarily defines three sections – input, filter and output. The input section defines the source, which is the container local directory we have volume mounted the log files into – /logstash_dir. The output section on the other hand defines the destination to post our logstash parsed data – in this case, its our elasticsearch service.
The filter section within logstash.conf is a little more interesting. It has five sub-sections. The first, dissect, defines how each log entry must be broken down into variables that are eventually pushed to elasticsearch. This is done by defining variables inline, while maintaining the pattern of the log entry. The second section, date, ensures ElasticSearch considers the actual file download date in all queries, as opposed to the data insertion (from Logstash to ElasticSearch) time. The third, geoip, decodes the IP Address listed in the log file, down to the level of a Country / City name. It also gives us lat/lng coordinates among other things. The fourth, ruby, defines a simple ruby script to further break down a variable into multiple other variables. In this case, we are breaking down the targetDownload variable, which evidently contains the full path to the download URI, to two more variables that contain the file name and file type. The final section, fingerprint, defines a algorithm that could be used to hash your data against a source, so that duplicate entries are avoided during insertion. You could modify these filter sub-sections to match the data on your log files.
We now have all the scripts required for booting up our environment, and one command to do that, must import all our logs on bringing up the enviroment – remember the command that our Logstash container executes on startup?
Install Elk Stack Ubuntu Docker
Let’s execute. Run,
What Is Elk Stack
If everything was setup right, after a few minutes, you must have Kibana come up on port 80 of the host machine – with the data in your logs!
To collect new logs, you would just need to restart the Logstash service. New data would be seen in Kibana after a few minutes.
Time to get going with custom dashboards / charts on Kibana!
Viel Glück! 🙂