gregjeanmart

6 min read - Posted 01 Apr 20

(8/8) Deploy Prometheus and Grafana to monitor a Kubernetes cluster


This article is part of the series Build your very own self-hosting platform with Raspberry Pi and Kubernetes
  1. Introduction
  2. Install Raspbian Operating-System and prepare the system for Kubernetes
  3. Install and configure a Kubernetes cluster with k3s to self-host applications
  4. Deploy NextCloud on Kuberbetes: The self-hosted Dropbox
  5. Self-host your Media Center On Kubernetes with Plex, Sonarr, Radarr, Transmission and Jackett
  6. Self-host Pi-Hole on Kubernetes and block ads and trackers at the network level
  7. Self-host your password manager with Bitwarden
  8. Deploy Prometheus and Grafana to monitor a Kubernetes cluster


Introduction

Monitoring is an important part of the maintenance of a Kubernetes cluster to gain visibility on the infrastructure and the running applications and consequently detect anomalies and undesirables behaviours (service downtime, errors, slow responses).

Prometheus and Grafana is a common combination of tools to build up a monitoring system where Prometheus acts as a data collector pulling periodically metrics from different systems and Grafana as a dashboard solution to visualise the data.

article image

Specifically for Kubernetes, a great Open-Source project called cluster-monitoring offers a scripts to automate the installation and configuration of Prometheus + Grafana and provides a lot of dashboards for Kubernetes out of the box.



Prerequisite

In order to run entirely the tutorial, we will need:

  • A running Kubernetes cluster (see previous articles if you haven't set this up yet)
  • Install Golang on the machine that runs kubectl
$ sudo apt  install golang-go
$ export PATH=$PATH:$(go env GOPATH)/bin
$ export GOPATH=$(go env GOPATH)


Configuration

1. Clone the repository

Clone the repository cluster-monitoring with the following command (change ~/workspace/cluster-monitoring by the target folder of your choice):

$ git clone https://github.com/carlosedp/cluster-monitoring.git ~/workspace/cluster-monitoring

2. Configuration

Then open the file ~/workspace/cluster-monitoring/vars.jsonnet and modify the following sections:

Enable k3s and put the IP of our master node kube-master.

  k3s: {
    enabled: true,
    master_ip: ['192.168.0.22'],
  },

The suffix domain is used to deploy an ingress to access Prometheus prometheus.<suffixDomain> and Grafana grafana.<suffixDomain>. You can manually configure a DNS entry to point to 192.168.0.240 (Load Balancer IP of the Nginx) or used nip.io to automatically resolve a domain to an IP (basically it resolves <anything>.<ip>.nip.io by <ip> without requiring any other configuration).

    suffixDomain: '192.168.0.240.nip.io',

Enable the persistence to store the metrics (Prometheus) and dashboard settings (Grafana).

  enablePersistence: {
    prometheus: true,
    grafana: true,
  },


Installation

Once we've done the configuration part, we can now proceed to the installation.

Navigate into the folder cluster-monitoring.

$ cd ~/workspace/cluster-monitoring/

First run make vendor to download all the necessary packages.

$ make vendor

rm -rf vendor
/home/gjeanmart/go/bin/jb install
GET https://github.com/coreos/kube-prometheus/archive/285624d8fbef01923f7b9772fe2da21c5698a666.tar.gz 200
GET https://github.com/brancz/kubernetes-grafana/archive/57b4365eacda291b82e0d55ba7eec573a8198dda.tar.gz 200
(...)
GET https://github.com/metalmatze/slo-libsonnet/archive/5ddd7ffc39e7a54c9aca997c2c389a8046fab0ff.tar.gz 200

Finally deploy the monitoring stack with the command make deploy.

$ make deploy

rm -rf manifests
./scripts/build.sh main.jsonnet /home/gjeanmart/go/bin/jsonnet
using jsonnet from arg
+ set -o pipefail
+ rm -rf manifests
+ mkdir manifests
+ /home/gjeanmart/go/bin/jsonnet -J vendor -m manifests main.jsonnet
+ xargs '-I{}' sh -c 'cat {} | gojsontoyaml > {}.yaml; rm -f {}' -- '{}'
kubectl apply -f ./manifests/
namespace/monitoring created
(...)
servicemonitor.monitoring.coreos.com/kubelet created

Repeat this command again if you see some errors as a result.


Once deployed, check that the namespace monitoring has all the components up and running.

$ kubectl get pods -n monitoring -o wide

NAME                                   READY   STATUS    RESTARTS   AGE     IP             NODE          NOMINATED NODE   READINESS GATES
prometheus-operator-56d66c5dc6-8ljsb   1/1     Running   0          7m54s   10.42.0.154    kube-master   <none>           <none>
alertmanager-main-0                    2/2     Running   0          7m43s   10.42.0.155    kube-master   <none>           <none>
kube-state-metrics-5c6fb7544b-6p956    3/3     Running   0          7m36s   10.42.0.157    kube-master   <none>           <none>
node-exporter-h48d6                    2/2     Running   0          7m34s   192.168.0.22   kube-master   <none>           <none>
prometheus-adapter-679db899b7-x89l6    1/1     Running   0          7m22s   10.42.0.158    kube-master   <none>           <none>
grafana-6fb45f9c8-j8qbv                1/1     Running   0          7m41s   10.42.0.159    kube-master   <none>           <none>
prometheus-k8s-0                       3/3     Running   1          7m15s   10.42.0.161    kube-master   <none>           <none>


Dashboard

After configuring and deploying our Kubernetes Monitoring stack, you can now access the different components:

Click on the Grafana link and login with the default login/password admin/admin (you will be asked to choose a new password)

article image

You can now finally enjoy a lot of pre-configured dashboards for your Kubernetes cluster.

article image

Created with Sketch.Content is"CC-BY-SA 4.0" licensed
Article Author

Grégoire Jeanmart

Kauri Software Engineer

27

3

7

3 Comments
Related Articles
(6/8) Self-host Pi-Hole on Kubernetes and block ads and trackers at the network level

This article is part of the series Build your very own self-hosting platform with Raspberry Pi and Kubernetes Introduction Install Raspbian Operating-System and

(1/5) Getting started with Elastic Stack for monitoring on Kubernetes

Introduction In this article, we will learn how to set up a monitoring stack for your Kubernetes environment (k8s in short). This kind of solution allows your team to gain visibility on your infrastructure and each application with a minimal impact on the existing. The goal of observability is to provide tools to operators responsible of running the production to detect undesirables behaviours (service downtime, errors, slow responses) and have actionable information to find the root cause of an