ML Model Deployment Checklist: 12 Things You're Probably Missing

Shipping an ML model to production is harder than training it. After helping dozens of teams deploy models, here's the checklist of things almost everyone misses on the first deploy.

1. Health Check Endpoint

Every deployment needs a /health endpoint that returns 200 OK. But a real health check should verify more than "the server is running."

@app.get("/health")
async def health():
    return {
        "status": "healthy",
        "model_loaded": True,
        "gpu_available": torch.cuda.is_available(),
        "gpu_memory_free_mb": torch.cuda.mem_get_info()[0] / 1e6 if torch.cuda.is_available() else 0,
        "uptime_seconds": time.time() - start_time,
    }

Use this in your Dockerfile healthcheck. If the model fails to load, the container should report unhealthy — not crash 30 seconds into the first request.

2. Non-Root Container User

Running as root in production is security malpractice. Add this to your Dockerfile:

RUN adduser --disabled-password --gecos '' appuser
USER appuser

Use UID 1000. Most cloud platforms expect it. Without this, an exploited vulnerability gives an attacker root access to your host.

3. Graceful Shutdown

Kubernetes and cloud orchestrators send SIGTERM before SIGKILL. Your app should:

1. Stop accepting new requests
2. Complete in-flight requests (up to a timeout)
3. Unload the model from GPU memory
4. Exit cleanly

import signal

@app.on_event("shutdown")
def shutdown():
    del model
    torch.cuda.empty_cache()

Without this, your GPU memory leaks on restart, and eventually you OOM.

4. Request Timeout and Rate Limiting

LLMs don't have fixed response times. A single slow request can block all others. Set a timeout:

@app.middleware("http")
async def timeout_middleware(request, call_next):
    try:
        return await asyncio.wait_for(call_next(request), timeout=30.0)
    except asyncio.TimeoutError:
        return JSONResponse({"error": "timeout"}, status_code=504)

And rate-limiting. Slowloris-style attacks on inference endpoints are real.

5. Input Validation

Never trust client input. Validate:

• Input length (prevent token bombs — 1M token prompts = GPU OOM)
• Input type (string, not arbitrary JSON)
• Encoding (UTF-8, reject binary garbage)

Pydantic makes this trivial with FastAPI:

from pydantic import BaseModel, Field, validator

class PredictRequest(BaseModel):
    text: str = Field(..., min_length=1, max_length=4096)
    
    @validator("text")
    def not_empty(cls, v):
        if not v.strip():
            raise ValueError("text cannot be empty")
        return v

6. Model Warm-up

First request after deployment is always slow (cold GPU, CUDA context initialization). Warm up your model on startup:

@app.on_event("startup")
async def warmup():
    dummy_input = "warmup"
    tokenizer(dummy_input)
    model.generate(**tokenizer(dummy_input, return_tensors="pt"))

Your first real user won't wait 10 seconds.

7. Dependency Pinning

# Bad: torch>=2.1.0
# Good: torch==2.1.2

Reproducibility matters. Use pip freeze or poetry.lock. A patch version change in torch can silently change model output.

8. Layer Caching in Dockerfiles

Docker builds from scratch every time unless you order layers correctly:

# Install dependencies first (rarely changes)
COPY requirements.txt .
RUN pip install -r requirements.txt

# Then copy app code (changes frequently)
COPY ./app /app

This way, rebuilding after a code change reuses the pip install layer.

9. Environment Variables for Configuration

MODEL_NAME = os.getenv("MODEL_NAME", "distilbert-base-uncased")
BATCH_SIZE = int(os.getenv("BATCH_SIZE", "32"))

Never hardcode paths, model IDs, or credentials. Use env vars. Cloud platforms inject them at deploy time.

10. Structured Logging

Print statements aren't observable. Use structured logging:

import structlog
logger = structlog.get_logger()
logger.info("prediction", latency_ms=142, input_length=128, gpu_util=0.82)

This lets you query logs: "show me all predictions with latency > 500ms." JSON-format logs are ingestible by any monitoring system.

11. Version Tagging

Every deployment should have a version. Tag Docker images with git commit SHA:

docker build -t my-model:$(git rev-parse --short HEAD) .

When something breaks, you know exactly which code is running.

12. Drift Monitoring

Models don't fail loudly. They degrade silently as input data changes. Monitor for drift:

• Data drift: Compare input distributions week-over-week (PSI score)
• Prediction drift: Track output distribution changes
• Accuracy decay: If you have ground truth labels, track them

Set up alerts: PSI > 0.2 = investigate. PSI > 0.3 = rollback to previous version.

The Meta-Checklist

If you're using Roptal, we handle items 1, 2, 3, 4, 7, 8, 9, 10, 11, and 12 by default. The generated Dockerfiles include health checks, non-root users, timeout middleware, structured logging, and version tags. Monitoring and drift detection are built into the platform.

Focus on your model. We'll handle the rest.

Join the waitlist →