Model Deployment Monitoring Checklist: Latency, Errors, GPU Memory, and Drift

Model monitoring is not one dashboard. It is a set of signals that answer four questions:

1. Is the endpoint available?
2. Is it fast enough?
3. Is the infrastructure healthy?
4. Is the model still behaving as expected?

This checklist covers the minimum production baseline.

Availability

Track endpoint health separately from process health.

Signal	Target	Alert threshold
HTTP success rate	> 99.9%	< 99.5% for 5 minutes
Health check success	100%	2 consecutive failures
Ready replicas	Expected minimum	Below minimum for 2 minutes

A process can be running while the model is not loaded. Your /health endpoint should verify model readiness, not just return a static response.

Latency

Record p50, p95, and p99 latency. Average latency hides tail problems.

p50: typical request
p95: slowest 5% of requests
p99: slowest 1% of requests

Example alert rules:

• p95 exceeds 500ms for 10 minutes
• p99 exceeds 2 seconds for 5 minutes
• latency rises 50% compared with the prior 24-hour baseline

Break latency down by stage when possible: request parsing, tokenization, model inference, serialization, network response. Otherwise you will not know whether the GPU is the bottleneck.

GPU Health

Signal	Why it matters	Alert threshold
GPU utilization	Capacity planning	> 90% sustained
VRAM used	OOM prevention	> 85% sustained
GPU temperature	Hardware stability	> 85°C
Power draw	Detect throttling	Unexpected drop under load
CUDA errors	Failing runtime	Any occurrence

High GPU utilization is not always bad. A stable 85% utilization with acceptable p95 latency is efficient. The problem starts when queue depth grows or p95 latency climbs with it.

Request Quality Signals

For classifiers, track label distribution. If a sentiment endpoint normally returns 70% positive and suddenly returns 98% negative, investigate before calling it model drift.

For generative models, track:

• input token count
• output token count
• tokens per second
• timeout rate
• refusal/error rate

Do not log raw customer prompts by default. Log length, hashes, aggregate statistics, and redacted error categories.

Drift Monitoring

Use Population Stability Index (PSI) to compare current input or output distributions with a baseline.

PSI value	Meaning	Action
< 0.1	Stable	Continue monitoring
0.1-0.2	Moderate change	Investigate data source changes
> 0.2	Significant change	Review model performance and retraining data
> 0.3	Severe shift	Consider rollback or traffic reduction

PSI is a signal, not proof of a bad model. A marketing campaign, seasonal trend, or upstream product change can legitimately shift inputs.

Cost Alerts

Set a budget before you need one:

• Daily GPU spend above expected range
• Projected monthly spend above budget
• GPU instance running with zero requests for 30 minutes
• Egress above expected baseline

The last alert catches an expensive and common failure: a permanent GPU endpoint serving no traffic after a DNS or application integration failure.

Minimum Dashboard

One dashboard should show:

Availability:     success rate, health checks, replicas
Performance:      p50/p95/p99, queue depth, throughput
Infrastructure:   GPU utilization, VRAM, temperature, CUDA errors
Model behavior:   output distribution, drift score
Cost:             spend today, forecast, idle time

Without these signals, an ML deployment is a black box. Roptal brings deployment logs, health checks, cost visibility, and drift signals into the same operating surface so teams do not need to assemble it from separate cloud consoles.