Cold Starts in ML Inference: Why Your First Request Takes 30 Seconds

You deploy a model. The endpoint shows "healthy." Your first request takes 30 seconds. Every request after that takes 200ms.

This is a cold start. It happens because the GPU needs time to initialize CUDA, load model weights into VRAM, and warm up kernels. Here's exactly what happens and how to minimize it.

What Happens During a Cold Start

Phase 1: Container Start (1-3 seconds)

The container runtime pulls the image (if not cached), creates the container, starts the process. If you followed the Docker optimization guide, image pull is 2-5 seconds on a fast connection. If your image is 6GB, add 20-60 seconds.

Phase 2: Python Import (2-5 seconds)

Your FastAPI/Uvicorn server starts. Python imports torch, transformers, and your application modules. PyTorch's initial import is surprisingly slow because it probes CUDA devices.

Phase 3: CUDA Context Initialization (1-3 seconds)

PyTorch creates a CUDA context. This allocates memory on the GPU, initializes the CUDA runtime. First call to torch.cuda.is_available() or model.to("cuda") triggers this.

Phase 4: Model Loading (5-20 seconds)

Model weights are loaded from disk to RAM, then transferred to GPU VRAM. For a 7B parameter model in FP16, that's 14GB to transfer. PCIe 3.0 x16 transfers at ~12 GB/s — theoretically 1.2 seconds. In practice, 5-10 seconds because of overhead and disk I/O.

Phase 5: Kernel Warmup (1-2 seconds)

The first inference triggers CUDA kernel compilation (JIT). Subsequent requests reuse compiled kernels. This is why the first prediction is slower even if the model is already loaded.

Total cold start: 10-33 seconds for a typical inference server.

How to Reduce It

1. Keep Images Small (< 2GB)

Smaller image = faster pull. If you're using spot instances or autoscaling, the image is pulled every time a new instance starts. Every 100MB you trim saves ~0.5 seconds on pull time.

2. Pre-Compile CUDA Kernels

PyTorch JIT-compiles CUDA kernels on first use. You can pre-compile at build time:

# warmup.py — run during Docker build or container startup
import torch

model = torch.load("model.pt").to("cuda")
dummy = torch.randn(1, 512).to("cuda")

# Run inference once to trigger JIT compilation
with torch.no_grad():
    for _ in range(3):
        _ = model(dummy)

torch.cuda.synchronize()
print("Kernels warmed up")

Call this script before starting the server. It forces PyTorch to compile all CUDA kernels, so the first real request doesn't.

3. Eager Model Loading on Startup

Don't lazy-load the model on the first request. Load it in the FastAPI startup event:

@app.on_event("startup")
async def load_model():
    app.state.model = AutoModel.from_pretrained("model-path").to("cuda")
    app.state.tokenizer = AutoTokenizer.from_pretrained("model-path")

Your health check should verify the model is loaded before marking the container healthy:

@app.get("/health")
async def health():
    if not hasattr(app.state, 'model'):
        return JSONResponse({"status": "not ready"}, status_code=503)
    return {"status": "healthy", "model_loaded": True}

4. Keep a Warm Pool

If your traffic is bursty, consider keeping at least 1 instance always warm. Serverless platforms like RunPod keep workers warm for 5 minutes after the last request. If your traffic has gaps longer than that, you'll hit cold starts on the next request.

For always-on deployments, this is automatic. For serverless, you can send a "ping" request every 4 minutes to keep the worker warm:

# Cron job
*/4 * * * * curl -s https://your-endpoint/health > /dev/null

This costs essentially nothing (one request per 4 minutes) but eliminates cold starts entirely.

5. Use a Fast Model Server

Standard FastAPI + transformers adds overhead. Alternative serving frameworks:

• vLLM: 2-5× faster than transformers for LLMs. Continuous batching, PagedAttention, pre-compiled kernels.
• TensorRT-LLM: NVIDIA's optimized inference. Fastest option but requires model conversion.
• Triton Inference Server: NVIDIA's production server. Supports multiple backends, dynamic batching.

The tradeoff is complexity. FastAPI + transformers is the easiest to set up. vLLM is the sweet spot for LLMs — much faster, not much more complex.

Cold Start by Platform

Platform	Typical Cold Start	Why
RunPod Serverless	15-25 sec	Worker spin-up + image pull + model load
GCP Cloud Run GPU	20-35 sec	Container cold start + GPU attachment
AWS SageMaker Serverless	30-60 sec	Endpoint provisioning + model load
AWS SageMaker (always warm)	0 sec	Instance always running
Self-hosted (always on)	0 sec	Instance always running

The Cold Start Tradeoff

Cold starts are the price you pay for not running a 24/7 GPU. For low-traffic endpoints (under 100 requests/day), accepting 20-second cold starts saves $200+/month vs an always-on instance.

For anything customer-facing or with latency SLOs, cold starts are unacceptable. Run at least 1 always-warm instance.

Roptal handles startup optimization automatically — pre-warms models, verifies health checks, and keeps standby instances warm during deployments.