6 min read - Posted 01 Apr 20

(8/8) Deploy Prometheus and Grafana to monitor a Kubernetes cluster

This article is part of the series Build your very own self-hosting platform with Raspberry Pi and Kubernetes
  1. Introduction
  2. Install Raspbian Operating-System and prepare the system for Kubernetes
  3. Install and configure a Kubernetes cluster with k3s to self-host applications
  4. Deploy NextCloud on Kuberbetes: The self-hosted Dropbox
  5. Self-host your Media Center On Kubernetes with Plex, Sonarr, Radarr, Transmission and Jackett
  6. Self-host Pi-Hole on Kubernetes and block ads and trackers at the network level
  7. Self-host your password manager with Bitwarden
  8. Deploy Prometheus and Grafana to monitor a Kubernetes cluster


Monitoring is an important part of the maintenance of a Kubernetes cluster to gain visibility on the infrastructure and the running applications and consequently detect anomalies and undesirables behaviours (service downtime, errors, slow responses).

Prometheus and Grafana is a common combination of tools to build up a monitoring system where Prometheus acts as a data collector pulling periodically metrics from different systems and Grafana as a dashboard solution to visualise the data.

article image

Specifically for Kubernetes, a great Open-Source project called cluster-monitoring offers a scripts to automate the installation and configuration of Prometheus + Grafana and provides a lot of dashboards for Kubernetes out of the box.


In order to run entirely the tutorial, we will need:

  • A running Kubernetes cluster (see previous articles if you haven't set this up yet)
  • Install Golang on the machine that runs kubectl
$ sudo apt  install golang-go
$ export PATH=$PATH:$(go env GOPATH)/bin
$ export GOPATH=$(go env GOPATH)


1. Clone the repository

Clone the repository cluster-monitoring with the following command (change ~/workspace/cluster-monitoring by the target folder of your choice):

$ git clone ~/workspace/cluster-monitoring

2. Configuration

Then open the file ~/workspace/cluster-monitoring/vars.jsonnet and modify the following sections:

Enable k3s and put the IP of our master node kube-master.

  k3s: {
    enabled: true,
    master_ip: [''],

The suffix domain is used to deploy an ingress to access Prometheus prometheus.<suffixDomain> and Grafana grafana.<suffixDomain>. You can manually configure a DNS entry to point to (Load Balancer IP of the Nginx) or used to automatically resolve a domain to an IP (basically it resolves <anything>.<ip> by <ip> without requiring any other configuration).

    suffixDomain: '',

Enable the persistence to store the metrics (Prometheus) and dashboard settings (Grafana).

  enablePersistence: {
    prometheus: true,
    grafana: true,


Once we've done the configuration part, we can now proceed to the installation.

Navigate into the folder cluster-monitoring.

$ cd ~/workspace/cluster-monitoring/

First run make vendor to download all the necessary packages.

$ make vendor

rm -rf vendor
/home/gjeanmart/go/bin/jb install
GET 200
GET 200
GET 200

Finally deploy the monitoring stack with the command make deploy.

$ make deploy

rm -rf manifests
./scripts/ main.jsonnet /home/gjeanmart/go/bin/jsonnet
using jsonnet from arg
+ set -o pipefail
+ rm -rf manifests
+ mkdir manifests
+ /home/gjeanmart/go/bin/jsonnet -J vendor -m manifests main.jsonnet
+ xargs '-I{}' sh -c 'cat {} | gojsontoyaml > {}.yaml; rm -f {}' -- '{}'
kubectl apply -f ./manifests/
namespace/monitoring created
(...) created

Repeat this command again if you see some errors as a result.

Once deployed, check that the namespace monitoring has all the components up and running.

$ kubectl get pods -n monitoring -o wide

NAME                                   READY   STATUS    RESTARTS   AGE     IP             NODE          NOMINATED NODE   READINESS GATES
prometheus-operator-56d66c5dc6-8ljsb   1/1     Running   0          7m54s    kube-master   <none>           <none>
alertmanager-main-0                    2/2     Running   0          7m43s    kube-master   <none>           <none>
kube-state-metrics-5c6fb7544b-6p956    3/3     Running   0          7m36s    kube-master   <none>           <none>
node-exporter-h48d6                    2/2     Running   0          7m34s   kube-master   <none>           <none>
prometheus-adapter-679db899b7-x89l6    1/1     Running   0          7m22s    kube-master   <none>           <none>
grafana-6fb45f9c8-j8qbv                1/1     Running   0          7m41s    kube-master   <none>           <none>
prometheus-k8s-0                       3/3     Running   1          7m15s    kube-master   <none>           <none>


After configuring and deploying our Kubernetes Monitoring stack, you can now access the different components:

Click on the Grafana link and login with the default login/password admin/admin (you will be asked to choose a new password)

article image

You can now finally enjoy a lot of pre-configured dashboards for your Kubernetes cluster.

article image

Created with Sketch.Content is"CC-BY-SA 4.0" licensed
Article Author

Grégoire Jeanmart

Kauri Software Engineer




Related Articles
(1/5) Getting started with Elastic Stack for monitoring on Kubernetes

Introduction In this article, we will learn how to set up a monitoring stack for your Kubernetes environment (k8s in short). This kind of solution allows your team to gain visibility on your infrastructure and each application with a minimal impact on the existing. The goal of observability is to provide tools to operators responsible of running the production to detect undesirables behaviours (service downtime, errors, slow responses) and have actionable information to find the root cause of an

(2/5) Install ElasticSearch and Kibana to store and visualize monitoring data

ElasticSearch cluster As explained in the introduction of this article, to setup a monitoring stack with the Elastic technologies, we first need to deploy ElasticSearch that will act as a Database to store all the data (metrics, logs and traces). The database will be composed of three scalable nodes connected together into a Cluster as recommended for production. Moreover, we will enable the authentication to make the stack more secure to potential attackers. 1. Setup the ElasticSearch master no