... | ... | @@ -10,27 +10,5 @@ The purpose of this project is to test various solutions for monitoring of Kuber |
|
|
- [Prometheus](Prometheus)
|
|
|
- [Kubernetes Dashboard](Kubernetes-Dashboard)
|
|
|
- [Jaeger](Jaeger)
|
|
|
- [Elastic Stack](Elastic-stack)
|
|
|
|
|
|
|
|
|
|
|
|
## Elastic Stack
|
|
|
|
|
|
Elastic stack (ELK stack) is a group of products from [Elastic](https://www.elastic.co) used to analyze, search and visualize data of any source. It consists of 4 components:
|
|
|
- **Elasticsearch** is a full text search engine written in Java
|
|
|
- **Logstash** is a log aggregator that collects, transforms and then ships logs to other services
|
|
|
- **Kibana** is a data visualization tool that works on top of Elasticsearch
|
|
|
- **Beats** are agents that collects various metrics and send them to other destinations such as Logstash. Some of the most commonly used Beats are Filebeat, Metricbeat or Packetbeat.
|
|
|
|
|
|
#### Configuration
|
|
|
|
|
|
[Elastic Cloud on Kubernetes(ECK)](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-overview.html) is the recommended way of deploying Elastic stack on k8s. There are individual Helm charts for each component of the stack but these will be deprecated soon. We first create all the necessary CRDs(Custom Resource Definition) such as Beat, Kibana etc. and then install the elastic operator. By default this will be installed in the `elastic-system` namespace.
|
|
|
We should start by installing `Kibana` and `Elasticsearch` resources as these require little to none configuration, we only need to set the `elasticsearchRef` attribute correctly in the Kibana deployment.
|
|
|
Next we will deploy Filebeat, which collects log from logfiles. We configure it to look at files at `/var/lib/containers/` directory which is where k8s stores logs from its containers and then we send the output to Logstash service on port 5044. Since Filebeat needs to read files on all pods across all nodes on our cluster, we need it give it elevated privileges using `ClusterRole` and `ClusterRoleBinding`.
|
|
|
Lastly, we need to configure Logstash. Unfortunately, Logstash is not part of the ECK platform, which complicates its deployment slightly, but there is an official [recipe](https://github.com/elastic/cloud-on-k8s/tree/main/config/recipes/logstash) to include it in ECK. Logstash pipeline is used to transform logs using [grok](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html) filter.
|
|
|
|
|
|
#### Preview
|
|
|
|
|
|
![image](uploads/3a49a894b3dd223234fe0dabc5d959a1/image.png)
|
|
|
*Kibana dashboard*
|
|
|
![image](uploads/7386e0f5d4bfa90ff8d9222a62505d2a/image.png)
|
|
|
*Raw contents of logfile (Message) and attributes parsed from it* |
|
|
\ No newline at end of file |