ML Inference Cost Calculator: GPU Hours, Requests, and Break-Even Points

GPU pricing is easy to quote and hard to budget. A T4 may cost $0.35/hour, but the real monthly cost depends on uptime, utilization, request volume, storage, egress, and engineering overhead.

Use this model to estimate a production inference budget before committing to an architecture.

Basic GPU Cost

Monthly GPU cost = hourly GPU price × hours per month × number of replicas

For an always-on T4 endpoint:

$0.35/hour × 730 hours = $255.50/month

Two replicas for high availability:

$255.50 × 2 = $511/month

That is the baseline before storage, logs, data transfer, or load balancers.

Cost Per Request

Cost per request = monthly infrastructure cost / monthly request count

Example: one $255.50 T4 endpoint processing 100,000 requests/month.

$255.50 / 100,000 = $0.00255 per request

At 1,000,000 requests/month, that becomes $0.000255 per request without changing the hardware. This is why utilization matters more than the headline hourly price.

Include the Hidden Line Items

Cost item	Typical monthly range	Notes
GPU compute	$200-$2,000+	Depends on GPU and uptime
Container registry	$1-$20	Image storage and pulls
Model artifact storage	$1-$100	S3/GCS/Blob storage
Load balancer	$20-$60	Internal or public ALB/LB
Logs and metrics	$10-$100	Depends on retention and volume
Data egress	Variable	Often overlooked
NAT gateway	$35+	Avoid with VPC endpoints where possible

For a single small production endpoint, the all-in cloud bill is commonly 1.2-1.5× the raw GPU line item.

Serverless vs Always-On Break-Even

Assume a serverless GPU costs $0.000044/sec and a dedicated T4 costs $0.35/hour.

Dedicated monthly cost: $255.50
Serverless break-even seconds: $255.50 / $0.000044 = 5,806,818 sec

That is about 1,613 GPU-hours. A month has 730 hours, so this simple calculation says dedicated looks cheaper. But serverless only runs while serving requests, while dedicated runs all month.

If each request consumes 0.25 seconds of GPU time:

Serverless requests at break-even = 5,806,818 / 0.25 = 23.2M requests/month

Actual break-even changes with cold starts, concurrency, batch size, and provider pricing. The lesson: serverless is generally cheaper for low or bursty traffic; dedicated wins for steady, high-volume traffic.

Batch Size Changes the Math

Batching improves GPU utilization. A model serving one request at a time might use 15% of a T4. Batch 16 requests together and it may use 70% of the GPU with only a modest latency increase.

Batch size	Typical throughput	Latency tradeoff
1	Lowest	Lowest individual wait
4	2-3× higher	Small queue delay
16	4-8× higher	Requires enough incoming traffic
32+	Workload dependent	Risks GPU memory pressure

Do not choose a batch size from a benchmark alone. Test it with your real request distribution and latency objective.

Simple Budget Template

Replicas:                    2
GPU hourly price:            $0.35
Hours/month:                 730
GPU cost:                    $511
Load balancer:               $30
Logs/metrics:                $30
Storage/registry:            $15
Estimated monthly total:     $586

Then divide by expected monthly requests. If the number is too high, test a lower-cost GPU, batching, quantization, serverless, spot capacity, or a different provider.

Roptal’s cost explorer tracks deployments by cloud and provider so this calculation is based on actual spend rather than spreadsheet estimates. Join the early-access list to test it with your own account.