Canary Deployments for ML Models: A Practical Guide

You trained a new version of your model. It scores higher on your eval set. Time to ship it, right?

Wrong. Eval metrics don't guarantee production performance. The only way to know if your new model is better is to test it on real traffic. That's where canary deployments come in.

What Is a Canary Deployment?

Named after the canaries miners used to detect toxic gas (the bird dies first, the miners escape), a canary deployment sends a small percentage of production traffic to the new model version and monitors it before rolling out to everyone.

Traffic
  ↙    ↘
v1.0   v1.1
90%    10% (canary)

If v1.1 shows:

• Higher error rate → Rollback
• Higher latency → Rollback
• Different output distribution → Investigate
• All clear → Promote to 100%

Why ML Models Need Canary Deployments

Software canary deployments check for crashes. ML canary deployments check for subtle behavioral changes:

1. Model quality regression: New version predicts "neutral" where old said "positive" for the same inputs
2. Latency spikes: Bigger model, different batch size, unexpected CUDA behavior
3. Throughput collapse: New model is 2x slower per request, saturating your GPU
4. Output drift: Distribution shift in predictions — not necessarily wrong, but worth investigating

Traditional canaries (check error rate only) miss ML-specific failures.

ML Canary Metrics to Track

Metric	What to Monitor	Threshold
Error rate	5xx responses	> baseline + 2% = rollback
p95 latency	95th percentile response time	> baseline * 1.5 = rollback
Prediction distribution	Mean/median prediction value	PSI > 0.2 = investigate
GPU memory	Peak memory usage	> 90% capacity = rollback
Token throughput	Tokens/second	< baseline * 0.8 = investigate

Implementing Canary Deployments

Approach 1: Load Balancer Level

If you're using a cloud load balancer (AWS ALB, GCP LB), configure weighted routing:

Target Group A (v1.0): weight 90
Target Group B (v1.1): weight 10

Pros: Language-agnostic, works with any app. Cons: Requires infra setup. Hard to do blue-green (swap all at once).

Approach 2: Application-Level Routing

Your FastAPI app has a router that sends 10% of requests to a different model instance:

import random

@app.post("/predict")
async def predict(request: PredictRequest):
    if random.random() < 0.10:
        return await canary_model.predict(request)
    return await stable_model.predict(request)

Pros: Zero infra. Simple to implement. Cons: Both models share the same GPU (memory pressure). Not truly isolated.

Approach 3: Roptal's Built-in Canary

Roptal has canary deployments built into the deployment pipeline:

Roptal Dashboard → Deploy new version → Canary (10% traffic)

Roptal:

1. Deploys v1.1 to a separate endpoint
2. Routes 10% of traffic to the new endpoint
3. Monitors error rate, latency, prediction distribution
4. Surfaces a comparison dashboard (v1.0 vs v1.1)
5. One-click promotion to 100% or rollback to 0%

The canary runs independently — separate container, separate GPU, separate monitoring. If it fails, v1.0 is unaffected.

Canary to Blue-Green: The Full Release Cycle

Canary (10%) → Monitor 24h → 
  ↓ Good
Canary (50%) → Monitor 6h →
  ↓ Good
Blue-Green (both active, switch) → 
  ↓ Good
Promote v1.1 to 100% → Decommission v1.0

This conservative approach catches:

• 0-1 hour: Crash bugs, config errors, missing files
• 1-24 hours: Latency issues, memory leaks, GPU-specific bugs
• 24-72 hours: Subtle model quality issues, prediction drift

Rollback Strategy

Always keep the previous version warm for instant rollback:

Active endpoints:
  v1.1 (current) — 100% traffic
  v1.0 (standby) — 0% traffic, but running and health-checking

Rollback should be one click (or one API call/CLI command). If v1.1 fails, traffic goes back to v1.0 in < 30 seconds.

With Roptal, this is built in. The previous deployment stays warm until you explicitly decommission it.

When NOT to Canary

Canaries add complexity. Skip them if:

• You're deploying an internal-only endpoint with < 10 QPS
• The model change is purely cosmetic (dependency upgrade, no code change)
• You have a separate staging environment with production-like traffic

But for any customer-facing inference endpoint? Always canary. The 10% overhead is worth catching the 1% of deploys that silently degrade.

TL;DR

1. Deploy new model version to 10% traffic
2. Monitor error rate, latency, prediction distribution for 24h
3. Increase to 50%, monitor 6h more
4. Promote to 100% or rollback
5. Keep previous version warm for instant rollback

Roptal automates steps 1-5. Focus on your model — we handle the release safety.

Join the waitlist →