Deploying FastAPI + PyTorch Models Without Kubernetes

Kubernetes is amazing. It's also the #1 reason ML engineers quit their jobs to become farmers. Here's how to deploy FastAPI + PyTorch models in production without touching a single YAML manifest.

The Problem with K8s for ML

Kubernetes was built for stateless microservices. ML models are stateful, GPU-hungry, and have unique deployment patterns. Adding K8s to the mix means:

• GPU scheduling is painful: Node selectors, tolerations, device plugins, NVIDIA operator — the setup alone takes a week
• Container images are massive: 5-8GB for PyTorch + CUDA base images. Your pod startup time is 3-5 minutes minimum
• Autoscaling doesn't work well for inference: HPA scales on CPU/memory — but GPU inference is latency-bound, not compute-bound
• Observability is fragmented: You need Prometheus + Grafana + custom exporters just to see GPU utilization
• Cost: K8s control plane alone costs $73/month on AWS EKS, plus worker nodes

Kubernetes makes sense at Google scale. For 90% of ML teams, it's overengineering.

Alternative 1: Cloud-Specific Managed Services

Every major cloud provider has a managed ML inference service:

• AWS SageMaker: Best for AWS-native teams. Costly. ~$255/month base per endpoint.
• GCP Cloud Run + GPU: Serverless inference. Pay-per-request pricing. Cold starts are 15-30s.
• Azure Container Apps: Similar to Cloud Run. Good if you're on Azure already.

Pros: No infrastructure to manage. Auto-scaling built in. Cons: Vendor lock-in. Different API/config per provider. Hard to multi-cloud.

Alternative 2: Standalone Containers on Compute

Run your FastAPI app as a plain Docker container on a GPU VM:

FROM pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8000
HEALTHCHECK --interval=30s CMD curl -f http://localhost:8000/health || exit 1
USER 1000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Run on any VM with Docker. Add nginx or Cloudflare in front for HTTPS.

Pros: Simple. Portable. Same Dockerfile works everywhere. Cons: You manage the VM, SSL, monitoring, updates, rollbacks manually.

Alternative 3: Roptal's Approach (BYOC Orchestration)

Roptal takes the Docker container approach and adds an orchestration layer:

1. Auto-generate production Dockerfiles from your repository
2. Deploy the container to your cloud of choice (AWS, GCP, Azure, RunPod, Railway)
3. Operate — monitoring, health checks, canary deployments, rollbacks, migrations — from one dashboard

No YAML. No K8s manifests. No cloud-specific config. The Dockerfile is the only artifact, and Roptal generates it for you.

The FastAPI + PyTorch Deployment Checklist

Whether you use Roptal or roll your own, here's what every production ML deployment needs:

Build

• Slim base image (runtime, not devel — saves 2-4GB)
• Layer caching optimized (copy requirements.txt first)
• Non-root user (uid 1000)
• Health check endpoint (/health returns 200)
• Graceful shutdown signal handling

Deploy

• HTTPS termination (nginx, Cloudflare, or cloud LB)
• Resource limits (not just requests — prevent OOM)
• GPU availability zone (not all zones have T4/A100)
• Secrets management (never bake API keys into images)

Operate

• Request logging + latency percentiles (p50, p95, p99)
• GPU utilization monitoring
• Model prediction drift detection (PSI > 0.2 = investigate)
• Budget alerts (GPU instances are $500+/month)
• Rollback plan (keep last 3 versions warm)

Why FastAPI + PyTorch Is a Great Stack

FastAPI is the most popular framework for serving ML models. It's:

• Fast: Async support, automatic OpenAPI docs, request validation via Pydantic
• Lightweight: No Django overhead. Just what you need for an API
• Standard: Python ecosystem. Easy to containerize. Libraries like torchserve exist but add complexity

PyTorch is the most widely used ML framework. Together, they're the default stack for production inference.

Getting Started Today

If you're deploying a FastAPI + PyTorch model and want to skip the K8s grind:

1. Local: Test with uvicorn main:app --host 0.0.0.0 --port 8000
2. Single VM: Docker + nginx reverse proxy. Takes 30 minutes.
3. Multi-cloud: Roptal — connect your repo, we generate the Dockerfile, deploy to any cloud

The platform is in early access. We're onboarding teams now.

Join the waitlist →