Monitoring During a Performance Test
The load tool tells you that response time went up. The server tells you why. During a test you need both views, side by side.
Think of the patient monitor again. The load test is the treadmill. The load tool shows how fast the patient runs. The monitor shows the heart rate, which is what explains it.
Table of Contents
Why
A load tool such as k6 or JMeter measures from the outside: response time, throughput, errors. It cannot see inside the server.
Say response time doubles at 200 users. Is the CPU full? Is the database out of connections? Is memory running out? Only monitoring on the server side can answer. Without it, you report "it got slow" and the team has nothing to fix.
For where this step sits in a whole test, see Performance Testing Process.
How: what to watch
Start with these five. Each one has a typical shape when something is wrong.
| Resource | What to look at | What "bad" looks like |
|---|---|---|
| CPU | CPU % per container | One container sits flat at 100% |
| Memory | Memory per container | Keeps climbing and never comes down |
| DB connections | Connections in use, against the maximum | In use equals the maximum, so requests queue |
| DB locks | Number of locks held | Keeps rising as load rises |
| Error logs | ERROR lines per minute | Appear, or jump, when load rises |
Look per container, not per machine. "The server is at 40%" can hide one container at 100%.
How: baseline first
Before the test, record 5 to 10 minutes of the system idle. Keep the same dashboard open.
That gives you "normal": the CPU at rest, the usual memory, the usual connections. During the test you compare against it. Without a baseline you cannot tell whether 30% CPU is high or just how this system always looks.
How: line up the two timelines
The load tool and Grafana each draw their own graph. To read them together:
- Use the same clock on the load tool and on Grafana. Same machine, or check the time zones.
- Write down the start and end time of the test.
- In Grafana, zoom the time range to that window.
- Find the moment the load tool's response time changed. Read what the server graphs did at that same moment.
Watch a Load Test puts both on one dashboard, so you do not need to compare two screens.
Tips
- Monitor the load generator too. A maxed-out laptop makes the server look fast, because the requests never arrive in time.
- Change one thing between runs. If you change two, you cannot say which one helped.
- Keep a screenshot of the dashboard for every run. It goes straight into the test report.
- Run the baseline on the same dashboard you use for the test, so the graphs are comparable.
Next: The Grafana Stack shows the tools that collect, store and show these numbers.