GPU Instance Spot Pricing: How to Run ML Inference at 70% Less Cost

Spot instances are the cheapest way to run GPU workloads. AWS, GCP, and Azure sell unused capacity at steep discounts. The tradeoff: your instance can be terminated with 30 seconds to 2 minutes of notice.

For inference, this is both a risk and an opportunity. If you architect for it, spot instances can cut your GPU bill by 60-80%.

Current Spot Pricing (August 2026)

GPU Instance	On-Demand/hr	Spot/hr	Savings
AWS g4dn.xlarge (T4)	$0.35	$0.09-0.14	60-74%
AWS g5.xlarge (A10G)	$0.75	$0.18-0.28	63-76%
GCP T4 (preemptible)	$0.28	$0.08-0.11	61-71%
GCP A100 (preemptible)	$1.55	$0.42-0.58	63-73%
Azure NC4as_T4_v3 (spot)	$0.39	$0.10-0.15	62-74%

Prices fluctuate based on region, time of day, and overall demand. us-east-1 tends to have the most stable spot pricing for AWS. GCP preemptible instances are fixed-price (no bidding), which makes budgeting predictable.

The Architecture: Surviving Instance Reclamation

You can't prevent spot termination. You can survive it.

1. Always Run at Least 2 Instances

One gets terminated, the other handles traffic. Minimum fleet size of 2, distributed across availability zones.

                     Load Balancer
                    /              \
         Instance A (spot)    Instance B (spot)
              us-east-1a        us-east-1b

When A gets reclaimed (2-minute warning), the load balancer routes all traffic to B. A replacement instance is launched and joins the pool. Total downtime: 0 seconds for users (assuming B can handle the full load).

2. Health-Check Aggressively

Don't wait for the 2-minute spot termination notice. Some terminations happen with less notice, or the notice doesn't reach your application correctly.

Check health every 10 seconds. If an instance fails 2 consecutive health checks, drain it from the load balancer. This covers spot terminations, GPU failures, and application crashes.

3. Pre-Warm Replacement Instances

When a termination notice arrives, don't just wait. Immediately launch a replacement. The 2-minute window is usually enough to:

• Pull the container image (if cached in the same AZ)
• Start the application
• Load the model into GPU memory
• Pass health checks
• Join the load balancer

If the replacement isn't ready in time, the remaining instance handles traffic until it is. With a minimum of 2 instances, this is fine.

4. Keep Model Artifacts External

Don't bake model weights into the Docker image. Store them in S3/GCS/Blob and download on startup. This keeps images small (faster pulls during spot replacement) and lets you update models without rebuilding the image.

import boto3

MODEL_PATH = os.getenv("MODEL_PATH", "/app/model")
if not os.path.exists(f"{MODEL_PATH}/pytorch_model.bin"):
    s3 = boto3.client("s3")
    s3.download_file("my-models", "sentiment-v3/pytorch_model.bin",
                     f"{MODEL_PATH}/pytorch_model.bin")

5. Use a Fallback Pool

Don't go 100% spot. Keep 1 on-demand instance as a floor, plus 2-4 spot instances for scaling. If spot capacity dries up entirely (rare but happens during GPU crunches), the on-demand instance keeps the service alive.

Base: 1x on-demand (guaranteed)
Scale: 2-4x spot (cost savings)

Spot instances handle 80% of traffic. On-demand handles 20%. If spot capacity vanishes, on-demand scales up.

Real Example: DistilBERT Inference @ 10,000 req/day

All on-demand:

• 2x g4dn.xlarge, 24/7: $504/month

Mixed (1 on-demand + 2 spot):

• 1x g4dn.xlarge on-demand: $252/month
• 2x g4dn.xlarge spot (average $0.12/hr): $173/month
• Total: $425/month
• Savings: 16%

All spot (minimum 2):

• 2x g4dn.xlarge spot (average $0.12/hr): $173/month
• Savings: 66% from on-demand pricing

For higher-traffic deployments with more instances, the savings increase. A fleet of 8 g5.xlarge instances on spot costs about $1,382/month. On-demand: $4,320/month. That's $35,256/year in savings.

When Not to Use Spot

• Single-instance deployments with no load balancing (termination = outage)
• Real-time inference with hard latency SLOs under 50ms (spot replacement adds jitter)
• GPU types that rarely have spot availability (H100, some A100 regions)
• Compliance requirements that mandate guaranteed capacity

Automating Spot Management

Managing spot instance replacement, health checks, and load balancer draining manually is tedious. Roptal handles spot instance lifecycle across AWS, GCP, and Azure. It maintains minimum instance counts, replaces terminated instances, and drains unhealthy ones — without manual intervention.