As a backend developer, on top of building and designing new systems, the ability to spot problems is one of the must-have skills. This is especially true in today’s environment where AI is all the rage — countless people have already handed their problems over to AI, hoping to let it take charge of everything. I have neither the standing nor the ability to pass judgment on whether that’s right or wrong, but far more important than “right or wrong” is the fact that we’re handing all our decision-making power to AI while neglecting to cultivate our own abilities — and that is undeniably fatal. It’s fatal not because it does any immediate harm. Let the bullet fly for a while: the consequences of outsourcing your core competency often don’t show up in the moment. Just like how the United States, 40 years on, is only now realizing that offshoring its factories in the name of globalization has hollowed out its domestic productivity, and is therefore aggressively pushing to bring American companies back home. AI is the same idea — the seven giants keep one-upping each other on compute, but in the end they still hit a wall at hardware resources and have to answer to the whims of the chip and semiconductor industry.
I’ve said too much. In any case, the point I want to stress is that cultivating your personal ability still cannot be ignored. This time around, I’m going to use AI paired with real-world testing to learn and verify what AI knows.
First, on the practical side of performance, before we rush headlong into writing optimization code, we first need to talk about: what is poor performance? By current thinking, the metrics for poor performance usually include:
- CPU usage
- Memory usage
- GPU usage
- I/O Wait (disk read/write latency)
- Network Latency
- Thread / Connection Pool utilization
- GC (Garbage Collection) frequency and duration
- Throughput
- Response Time (including P50 / P95 / P99)
CPU Usage
Common causes of CPU problems include: large amounts of compute-intensive work (e.g. encryption/decryption, Context Switches, and so on), frequent JVM GC, serialization overhead, improper thread declaration, infinite loops, and the like.
Memory Usage
Now that CPU is out of the way, it’s Memory’s turn to take the stage. A lot of the time, GC and Memory are tied together — if you don’t manage Memory well, GC gets triggered frequently, and the price of frequent triggering is that Stop The World (STW) shows up constantly: the service halts while users keep on using it. In a scenario where Threads keep getting created, different Threads compete for CPU resources, driving CPU usage up, and each Thread also consumes Memory resources, so in the end both CPU and Memory get dragged into the same muddy water. That’s why high Memory usage is usually more worth worrying about than high CPU usage — because it tends to trigger a chain reaction.
Other common causes include: Java Heap configuration problems, Memory Leaks, reading an entire file directly via IO, connection pools with no TTL set, and so on.
GPU Usage
Compared with CPU and Memory, GPU usage is a slightly more niche concern for traditional backend services — unless your service involves AI inference, image/video processing, or machine-learning-related Batch Jobs, you normally wouldn’t go out of your way to monitor this metric. But now that AI services are everywhere, needs like model inference, Embedding computation, and image recognition are showing up more and more in backend architectures, so GPU usage can no longer be ignored.
Common causes include: improperly set Batch Size — you can think of Batch Size as the amount of data an AI Model can process at once; set it too small and you fail to leverage the GPU’s parallel-computing advantage, set it too large and you easily blow out your VRAM — the model itself not being optimized (e.g. no Quantization, not using inference-acceleration frameworks like TensorRT), multiple Processes fighting over the same GPU (poor resource isolation or scheduling), and so on.
You can just skim this part — I’m honestly not all that familiar with it, and I won’t be covering it later either.
I/O Wait (Disk Read/Write Latency)
I/O Wait refers to the idle time produced while the CPU waits for the disk (or another I/O device) to finish reading or writing data. Let me give a preemptive heads-up here in case anyone doesn’t know: fundamentally, how fast data can be read depends on where it’s stored, and the speed from highest to lowest goes registers, cache memory (L1, L2), main memory, disk — of which disk read/write speed is the slowest.
The concept of I/O Wait is like an external resource: it doesn’t directly burden the CPU or Memory, but the CPU/Memory both have to wait on the I/O Wait, so it still causes the system’s response speed to drop and indirectly keeps the CPU and Memory from releasing their resources. So if you see low CPU usage but the system responding very slowly, I/O Wait is usually the first suspect to look at.
Common causes include: disk read/write speed being too slow (like a traditional HDD), large amounts of synchronous read/write operations, Log writes being too frequent with no Buffer mechanism, and database queries lacking indexes leading to a Full Table Scan, among others.
Worth mentioning is that I/O Wait is often intertwined with the Memory and GC problems mentioned earlier — insufficient Memory leads to Swap, Swap in turn generates a large amount of I/O, and the I/O latency then drags down overall response time, forming a vicious cycle. This is exactly why, when observing a system, you can’t just look at a single metric; you have to put these metrics side by side and cross-reference them to find the true root cause.
Network Latency
Network Latency refers to the time that elapses from when a request is sent to when its response is received. Where it’s similar to I/O Wait is that it, too, is a kind of “waiting on external resources” latency — it’s just that this time what you’re waiting on isn’t the disk but the other end of the network: possibly a database, a cache server, a third-party API, or another Service in a microservice architecture.
Common causes include: slow DNS resolution; cross-region and cross-country network requests; the extra overhead of the TCP Three-way Handshake and the TLS Handshake; the downstream service itself responding slowly (passing its latency on to you); an overly long call chain under a microservice architecture (a single Request has to pass through several layers of Service before getting a result, with every layer piling on more latency); and an insufficient Connection Pool causing Requests to queue up waiting for an available connection.
Thread / Connection Pool Utilization
The concept behind a Thread Pool and a Connection Pool is “prepare a batch of reusable resources in advance, to avoid creating them anew every time.” The difference is that a Thread Pool manages threads, while a Connection Pool manages connections (database connections, HTTP connections, etc.). These two metrics are symptoms — when CPU/Memory usage spikes and you need to find the bottleneck, these are directions you can investigate.
When you see Thread Pool utilization sitting up against its ceiling for a long time, the reasons behind it might be: some Blocking call has seized a Thread and won’t let go (for example, a synchronous call to an external API with no Timeout set), a database query being too slow so the Thread takes forever to be released, or simply the Thread Pool Size being set too conservatively — plenty of hardware resources available, yet not enough threads opened up.
Connection Pools, on the other hand, commonly come up with database connections or HTTP Clients. If Connection Pool utilization stays high for a long time, or you even frequently see “couldn’t get an available connection” Timeout errors, common causes include: connections being used but never returned (especially when an exception occurs and the connection isn’t properly closed/released via try-with-resources or finally), the Pool Size being set too small, and a single Request holding a connection for too long (for example, a Transaction that wraps in too much unnecessary logic and takes forever to Commit and release the connection).
Worth mentioning is that these two metrics are well-suited for “early warning” — compared with CPU and Memory, which usually only spike once the problem has already happened, Thread Pool and Connection Pool utilization tend to show anomalies earlier, making them dashboards worth watching first, before things actually blow up.
GC (Garbage Collection) Frequency and Duration
The concept of GC is that it’s responsible for reclaiming objects nobody is using anymore and freeing up Memory space, so the Heap doesn’t expand without bound. But GC isn’t free, and its frequency and duration are two things in particular that, once out of control, cause a double whammy of CPU/Memory problems. Generally, GC metrics should be read together with CPU/Memory.
If GC gets triggered every so often, it means objects are being produced too fast. This usually implies that the code creates a large number of “use once and toss” temporary objects (commonly seen in the serialization, string concatenation, and Stream operations mentioned earlier, which produce intermediate objects), or that the Heap is configured too small to keep up with the rate of object production under normal traffic.
If the duration of a single GC run stretches out, or a Full GC (a complete reclamation targeting the Old Generation) occurs, the service will show noticeable latency spikes, and may even cause the Health Check to misjudge the service as being down. Common reasons for prolonged Full GC times include: too many long-lived objects stuffed into the Old Generation (possibly a genuine Memory Leak, or possibly a Cache with no eviction mechanism set), the Heap being set too large so a single scan covers too wide a range, and an unsuitable choice of GC algorithm (for example, using an algorithm not good at handling large Heaps in a high-traffic, low-latency scenario).
What’s special about the GC metric is that it’s almost the intersection of CPU and Memory problems — frequent GC usually comes with rising CPU usage, while excessive GC duration often corresponds to problems with Memory configuration or object lifecycle management. So observing GC is, to some degree, observing the health of both CPU and Memory at the same time.
Throughput
Throughput refers to the number of requests a system can handle per unit of time (common units include TPS and QPS — Transactions/Queries Per Second). Usually, when you want to assess a system’s overall performance you’ll use Throughput, and the typical use case is observing how Throughput changes under stress testing or high-traffic scenarios.
How to improve Throughput is often a whole-system problem; in reality it involves all the metrics mentioned above, making it something of a culminating criterion for judgment.
Response Time (including P50 / P95 / P99)
Response Time refers to the total time spent from sending a request to receiving a response. Like Throughput, it’s also a composite metric, but the two look at things from different angles — Throughput answers “how many requests the system as a whole can hold up under,” while Response Time answers “the wait time users actually feel.”
Here I especially want to bring up the terms P50, P95, and P99, because only looking at the Average is easily misleading, so we must also look at:
- P50 (median): means half of all requests have a response time within this number, reflecting the experience of the “average user”
- P95: means 95% of requests are within this number, with the remaining 5% being slower; this is usually used to catch the group that is “still acceptable, but already on the slow side”
- P99: means 99% of requests are within this number, with the remaining 1% being the slowest Long Tail; this is often the one that truly deserves attention, because it means some users are having an awful experience that the average completely fails to reveal
In real scenarios, just staring at the average isn’t enough; P99 (or even P99.9) is the key to catching a system’s long-tail problems. Any hiccup in any of the earlier stages will ultimately be reflected in Response Time going up.
Now you know how we evaluate the metrics for poor performance. There are many metrics, and the various problems are frequently chain reactions; sometimes even after making an improvement it by no means takes effect right away. So before making changes, what’s more important is the means to identify the problem — sufficient observation and evaluation are what let you avoid wasted effort. So in the next article, we’ll go on to talk about how we evaluate and monitor these numbers.