The Grafana Stack
The four steps from Monitoring in General are collect, store, show and alert. In this module each step is done by one open-source tool. Together they are called the Grafana stack.
Table of Contents
Why
These tools are free, run in Docker on a laptop and are common in real teams. Learning them here carries over to most companies.
You do not need to know how they work inside. You need to know what each one is for, so that when a graph is empty you know which piece to check.
How: four pieces
| Piece | Job | Patient monitor equivalent |
|---|---|---|
| Alloy | Collects: reads the numbers and log lines and sends them on | The sensors on the finger and chest |
| Prometheus | Stores metrics: numbers over time | The monitor's memory of the last hours |
| Loki | Stores logs: written lines about events | The nurse's notes |
| Grafana | Shows and alerts: graphs, dashboards and alarms | The screen and the alarm |
How: how data moves
- The playground containers run as usual. They are not changed.
- Alloy reads the Docker socket. From it, it gets CPU and memory per container (network metrics are missing on OrbStack), and every container's log lines.
- Alloy also asks the Postgres database for its own stats: connections, locks and transactions.
- Alloy pushes the metrics to Prometheus and the log lines to Loki.
- k6 pushes its test results to Prometheus too. Watch a Load Test uses this.
- Grafana asks Prometheus and Loki for data when you open a dashboard. You open Grafana in the browser on port 3001.
Pull and push, in short. Prometheus normally pulls: it visits each target on a schedule and scrapes the numbers. Here, Alloy collects and pushes to Prometheus by remote write, and k6 does the same. Prometheus itself only scrapes itself and Alloy's health. Loki always receives pushes.
How: what we can see without changing the app
| What | Can we see it? | Where it comes from |
|---|---|---|
| Container CPU, memory | Yes | Alloy, from the Docker socket |
| Container logs | Yes | Alloy, from the Docker socket |
| Postgres connections, locks, transactions | Yes | Alloy, asking the database |
| JVM details (heap, garbage collection, threads) | No | Needs a change in the app. See What Needs the Dev Team |
Tips
- Old guides say "Grafana Agent". Alloy replaced it. The idea is the same, so the guides still help.
- One Alloy per server is enough. It collects for every container on that machine.
- InfluxDB appears in JMeter guides, through the Backend Listener. That is a different path to a different store. See Backend Listener.
- If a graph is empty, follow the arrows in the diagram. Is the data reaching Prometheus or Loki at all, or is the problem only in Grafana?
Next: Set Up Prometheus starts the first piece.