... | ... | @@ -3,39 +3,13 @@ |
|
|
The purpose of this project is to test various solutions for monitoring of Kubernetes(k8s) environment. Inside a k8s cluster we can observe various metrics, for example: interactions between pods inside a cluster, whether pods are up or down, utilization of system resources inside of pods and many more. It is useful to track these metrics to get a sense of overall health of the cluster and prevent failures in advance. We will use `minikube` tool, which creates a local k8s cluster with single node. It is recommended to run this project in VM(virtual machine).
|
|
|
|
|
|
## Web app
|
|
|
[Web app](Web-app)
|
|
|
## Prometheus
|
|
|
|
|
|
#### Overview
|
|
|
|
|
|
[Prometheus](https://prometheus.io) is an open-source system monitoring tool with support for alerting. It is specifically targeted for highly dynamic container environments, such as `kubernetes`. It offers monitoring of many aspects of environment and automatic alerting. Prometheus works by collecting metrics from monitored targets by scraping the `/metrics` HTTP endpoint and stores these metrics to storage. We can use PromQL language to then query the stored metrics. While Prometheus allows same basic data visualization, we will use Grafana to visualize the data and Prometheus for scraping.
|
|
|
|
|
|
#### Setup
|
|
|
|
|
|
When installing Prometheus, we have two options: manually or using [helm](https://helm.sh/) package manager. Since Prometheus is stateful application(it saves the data from current session to storage), it is not recommended to set it up manually(unless we need some customizable feature), as it would be difficult. Instead, we will use `helm`, the package manager for k8s, for the initial setup. In general, using `helm` with official charts(packaging format used by `helm`) is the preferred way of installing kubernetes components.
|
|
|
|
|
|
#### Config
|
|
|
[Web app](Web-app)
|
|
|
|
|
|
Before we begin, we need to enable the `/metrics` endpoint on our custom pods so Prometheus can scrape the metrics. This can be done using exporters, a library which helps exporting metrics from third-party services to Prometheus. There are [a lot](https://prometheus.io/docs/instrumenting/exporters/) of officially supported exporters. Since our backend pod is using Flask, we will utilize [prometheus-flask-exporter](https://github.com/rycus86/prometheus_flask_exporter). Enabling some basic metrics is as easy as writing two lines of code:
|
|
|
## Monitoring solutions
|
|
|
- [Prometheus](Prometheus)
|
|
|
|
|
|
```python
|
|
|
from flask import Flask
|
|
|
from prometheus_flask_exporter import PrometheusMetrics
|
|
|
|
|
|
app = Flask(__name__)
|
|
|
PrometheusMetrics(app)
|
|
|
```
|
|
|
Now we need to make Prometheus aware of this new target. For this we will use ServiceMonitor object, that will get deployed into the same namespace as our webapp and will monitor a service given by the `selector` property. By default, Prometheus only monitors objects in the same namespace, we need to add `serviceMonitorNamespaceSelector` that will match common labels on all namespaces we want to monitor.
|
|
|
To export metrics from `frontend` we will use [nginx-prometheus-exporter](https://github.com/nginxinc/nginx-prometheus-exporter). Since nginx exporter is a docker container, it is easiest to add it to our frontend pod as a [sidecar](https://medium.com/bb-tutorials-and-thoughts/kubernetes-learn-sidecar-container-pattern-6d8c21f873d). Next we edit our `nginx.conf` to return [stub-stats](http://nginx.org/en/docs/http/ngx_http_stub_status_module.html) at specified port, and that should be it.
|
|
|
|
|
|
#### Preview
|
|
|
Now we should see our two new targets in Prometheus UI:
|
|
|
![image](uploads/d29f564530a493039ae3b15c2ccf7447/image.png)
|
|
|
*Prometheus targets, our 2 custom targets at the bottom*
|
|
|
![image](uploads/b4b73977c7c83d8bc9b15b5d358c4c27/image.png)
|
|
|
*Grafana showing resource utilization on node*
|
|
|
![image](uploads/1e9b7bf9465720c7e84c06101b2df820/image.png)
|
|
|
*Custom charts*
|
|
|
|
|
|
## Kubernetes Dashboard
|
|
|
[Dashboard](https://github.com/kubernetes/dashboard) is a web based UI for displaying as well as managing k8s clusters.It can be used to get an overview of what apps are running on our cluster, but also for creating and configuring existing k8s resources(Deployment, Services)
|
... | ... | |