Monitoring GPU Utilization in Production ML Inference

GPU monitoring is not the same as CPU monitoring. You can have a CPU at 30% and a GPU at 100% — or a GPU at 20% and requests queuing because the bottleneck is somewhere else. Here's what to track and how.

The Key Metrics

GPU Utilization (%)

NVIDIA reports this via nvidia-smi. It measures what fraction of the GPU's compute units are active over a sample period.

• < 20%: GPU is mostly idle. Your model might be CPU-bound, waiting on I/O, or you're overprovisioned.
• 40-80%: Healthy zone for inference. Enough headroom for spikes.
• > 90%: At capacity. Requests are queuing. Either add another replica or optimize the model (batching, quantization, TensorRT).

But this number alone is misleading. A GPU at 30% utilization can still be a bottleneck if the 30% represents sequential kernel launches that take 50ms each.

GPU Memory Usage

More important than utilization for inference. If memory exceeds 90%, you're close to OOM.

Monitor:

• Allocated memory: What your model and tensors use
• Reserved memory: What PyTorch's CUDA allocator has claimed
• Free memory: What's left for KV cache, batch expansion, new allocations

import torch

def gpu_memory_report():
    if not torch.cuda.is_available():
        return {"cuda": False}
    return {
        "allocated_gb": round(torch.cuda.memory_allocated() / 1e9, 2),
        "reserved_gb": round(torch.cuda.memory_reserved() / 1e9, 2),
        "free_gb": round(torch.cuda.mem_get_info()[0] / 1e9, 2),
        "total_gb": round(torch.cuda.mem_get_info()[1] / 1e9, 2),
    }

Request Latency

Track p50, p95, and p99 latency. The p50 tells you typical experience. The p95 and p99 tell you about worst-case.

Metric	What it means
p50 latency	Half of requests are faster than this
p95 latency	5% of requests are slower than this
p99 latency	1% of requests are slower than this

If p95 latency is 200ms and p99 is 2,000ms, something is causing tail latency. Common causes: garbage collection, CUDA context switching, batch size boundary effects.

Throughput

Requests per second or tokens per second. Track throughput alongside latency. If throughput drops without a corresponding increase in latency, the issue is in the data pipeline, not the GPU.

Queue Depth

Number of requests waiting to be processed. If this grows, you're underprovisioned or something is blocking. Queue depth should average near 0 for a well-provisioned endpoint.

Tools

nvidia-smi (Command Line)

nvidia-smi --query-gpu=timestamp,utilization.gpu,memory.used,memory.total,temperature.gpu --format=csv -l 1

Prints GPU stats every second. Good for debugging, not for production monitoring.

nvtop

Interactive terminal tool. Shows GPU utilization, memory, per-process breakdown. Install: apt install nvtop or pip install nvitop.

Prometheus + DCGM Exporter

NVIDIA's Data Center GPU Manager (DCGM) exports GPU metrics to Prometheus. This is the standard approach for production monitoring:

docker run -d --gpus all --name dcgm \
  nvidia/dcgm-exporter:latest

Exposes metrics at :9400/metrics. Prometheus scrapes this. Grafana visualizes.

Key DCGM metrics:

• DCGM_FI_DEV_GPU_UTIL — GPU utilization
• DCGM_FI_DEV_FB_USED — framebuffer memory used
• DCGM_FI_DEV_FB_FREE — framebuffer memory free
• DCGM_FI_DEV_GPU_TEMP — GPU temperature

Application-Level Export

Don't rely only on nvidia-smi. Export metrics from your application:

from prometheus_client import Gauge, Histogram, generate_latest

request_latency = Histogram("inference_latency_seconds", "Per-request latency")
gpu_memory_used = Gauge("gpu_memory_used_bytes", "GPU memory allocated")

@app.middleware("http")
async def metrics_middleware(request, call_next):
    start = time.time()
    response = await call_next(request)
    request_latency.observe(time.time() - start)
    return response

@app.get("/metrics")
def metrics():
    gpu_memory_used.set(torch.cuda.memory_allocated())
    return Response(generate_latest(), media_type="text/plain")

This gives you per-request latency alongside GPU metrics in one dashboard.

Alert Thresholds

Alert	Threshold	Action
GPU memory > 90%	5 min sustained	Add replica or reduce batch size
GPU utilization > 95%	10 min sustained	Add replica or optimize model
p95 latency > 2× baseline	5 min	Investigate model/GPU bottleneck
Queue depth > 50	immediate	Scale up immediately
GPU temp > 85°C	5 min	Check cooling, reduce load

Common Issues Found Through Monitoring

GPU utilization at 15%, latency high. The model is CPU-bound. Data preprocessing, tokenization, or network I/O is the bottleneck. Move preprocessing to a separate thread/process, or pre-compute embeddings.

GPU memory slowly growing over days. Memory leak. Check for tensors accumulating without being freed. Common cause: not calling torch.cuda.empty_cache() after deleting large tensors, or storing hidden states across requests.

Periodic latency spikes every few minutes. Garbage collection or CUDA context synchronization. Profile with torch.cuda.synchronize() calls in your code to find the blocking point.

Roptal includes GPU monitoring in every deployment. Memory, utilization, temperature, and request metrics are tracked from the moment the endpoint goes live.