Multi-Cloud ML Deployment: When One Provider Isn't Enough

Running all your inference on one cloud provider is a bet. A bet that AWS won't have an outage in us-east-1. A bet that GCP's T4 pricing won't spike during conference season. A bet that RunPod's A100 capacity won't dry up when a new open-source model drops.

Multi-cloud removes that bet.

Why Single Cloud Fails

Regional Outages

AWS us-east-1 had a major outage in December 2021 that took down large parts of the internet for hours, followed by another significant incident in June 2023. Every major cloud has them. If your inference endpoint is in that region, it's down.

Multi-cloud means you have a hot standby in another provider's region. Traffic shifts from AWS us-east-1 to GCP us-central1 in under a minute.

GPU Price Spikes

NVIDIA GPU availability fluctuates. When a new open-source LLM drops (like Llama 3 or Mistral), demand for inference GPUs spikes across all platforms. Spot prices on AWS can double. GCP might have better availability. RunPod's community cloud might be fully booked.

With multi-cloud, you route traffic to whichever provider has the best price-performance at that moment.

Regional Cost Differences

A T4 instance costs $0.35/hr on AWS us-east-1 but $0.39/hr on AWS eu-west-1. GCP charges $0.28/hr for T4 in us-central1 but $0.33/hr in europe-west4. These differences are small per-hour but compound:

Scenario	Single Cloud (AWS us-east-1)	Multi-Cloud
4x T4, 24/7	$1,008/month	$806/month (GCP)
1 year	$12,096	$9,672

$2,424 saved per year just by picking the cheapest provider for your region.

Compliance and Data Residency

GDPR requires certain data to stay in the EU. If you're on AWS and your primary region is us-east-1, you need a separate deployment in eu-west-1. Multi-cloud means you can deploy to GCP's europe-west4 instead — whichever is cheaper and compliant.

The Architecture

              Route 53 / Cloudflare DNS
                     |
           ┌─────────┴─────────┐
           |                   |
     AWS Load Balancer    GCP Load Balancer
           |                   |
    ┌──────┴──────┐    ┌──────┴──────┐
    |      |      |    |      |      |
  T4-1  T4-2  T4-3  T4-1  T4-2  T4-3
  (us-east-1)        (us-central1)

Traffic is split 50/50 between AWS and GCP under normal conditions. Health checks run against both. If one provider goes unhealthy, DNS shifts 100% to the healthy provider.

Health Checks Are the Critical Path

Multi-cloud fails without good health checks. Every endpoint needs:

@app.get("/health")
async def health():
    return {
        "status": "healthy",
        "cloud": os.getenv("CLOUD_PROVIDER"),
        "region": os.getenv("CLOUD_REGION"),
        "model_loaded": True,
        "gpu_available": torch.cuda.is_available(),
        "latency_p95_ms": get_p95_latency(),
    }

The DNS health check calls /health every 30 seconds from multiple geographic locations. If the endpoint returns non-200, fails to respond, or reports gpu_available: false, it's marked unhealthy and traffic is withdrawn.

Failover: Active-Active vs Active-Passive

Active-Active: Both providers serve traffic simultaneously. Both are always warm. Failover is instant — the remaining provider just picks up 100% of traffic. Costs 2x but has zero downtime.

Active-Passive: One provider serves traffic. The other runs at minimum capacity (single instance, warm but idle). On failure, the passive side scales up and takes over. Cheaper but 30-60 seconds of reduced capacity during failover.

For production inference, active-active. The cost is worth the reliability. For non-critical workloads, active-passive saves 40-50%.

Data Consistency

If your model writes predictions to a database, you need a single source of truth. Don't write to an AWS database and a GCP database — you'll have split brain.

Options:

• Managed database with multi-region replication: AWS Aurora Global Database, GCP Cloud Spanner
• Database hosted neutrally: PlanetScale or Supabase, accessible from both clouds
• Don't write at all: Many inference endpoints are read-only. No consistency issues.

When Multi-Cloud Is Overkill

If you have:

• A single model with < 100 QPS
• Acceptable downtime of < 1 hour
• A budget under $500/month for infrastructure
• No compliance requirements for data residency

Single cloud is fine. The complexity of multi-cloud isn't worth it at this scale.

When to Go Multi-Cloud

You should consider multi-cloud when:

• Downtime costs more than the extra infrastructure (typically > $1,000/hour)
• Your users are geographically distributed across continents
• You have compliance requirements in multiple jurisdictions
• You're spending $2,000+/month on inference and can save 20%+ by switching providers

Roptal handles multi-cloud deployment from a single control plane. Deploy once, run everywhere. Traffic splitting, health checks, and failover are built in — no per-cloud configuration needed.