Docker Image Size Matters: How to Reduce Your Inference Container by 70%

A typical PyTorch inference image is 5-8GB. Every deployment pulls that image to a new instance. During autoscaling events or spot instance replacement, image pull time directly adds to cold start latency.

Here's how to get it under 2GB.

Where the Bloat Comes From

A standard pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime image is 2.5GB base. After adding your code and dependencies:

Layer	Typical Size	Optimized Size
Base image (runtime)	2.5 GB	2.5 GB
System packages	200 MB	60 MB
Python packages	1.5 GB	400 MB
Application code	10 MB	10 MB
Model weights (in image)	2-4 GB	0 (external)
Total	6-8 GB	~2.1 GB

Step 1: Use the Runtime Image

# Bad: 5GB
FROM pytorch/pytorch:2.1.0-cuda12.1-cudnn8-devel

# Good: 2.5GB
FROM pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime

The devel image includes nvcc, cuDNN headers, build tools. You don't need these at inference time.

Step 2: Combine RUN Commands

Each RUN creates a layer. More layers = more metadata, slower pulls. Combine related RUNs:

# Bad: 3 layers
RUN apt-get update
RUN apt-get install -y libgl1
RUN rm -rf /var/lib/apt/lists/*

# Good: 1 layer
RUN apt-get update \
    && apt-get install -y --no-install-recommends libgl1-mesa-glx \
    && rm -rf /var/lib/apt/lists/*

--no-install-recommends alone saves ~80MB on a minimal install.

Step 3: Clean pip Cache

RUN pip install --no-cache-dir -r requirements.txt

Without --no-cache-dir, pip stores downloaded wheels in /root/.cache/pip. That's ~500MB for PyTorch + transformers + other ML packages. This flag removes it.

Step 4: Minimal Requirements

Don't install packages you don't need:

# Bad
torch==2.1.0
torchvision==0.16.0
torchaudio==2.1.0
transformers==4.35.0
datasets==2.15.0
pandas==2.1.0
matplotlib==3.8.0

# Good
torch==2.1.0
transformers==4.35.0
fastapi==0.104.0
uvicorn==0.24.0
pydantic==2.5.0

For inference, you don't need datasets (used for loading training data), matplotlib (plotting), pandas (data wrangling), or torchvision/torchaudio (unless you're processing images/audio). Audit your requirements.txt — remove everything not imported at inference time.

Step 5: External Model Storage

Never bake model weights into the Docker image:

# Bad: adds 3GB to the image
COPY ./models/sentiment-v3.bin /app/models/

# Good: download at runtime

Store weights in S3/GCS/Blob Storage. Download on container startup. This keeps the image small and lets you update the model without rebuilding the image.

if not os.path.exists("/app/model/pytorch_model.bin"):
    boto3.client("s3").download_file(
        "my-models", "sentiment-v3/model.bin",
        "/app/model/pytorch_model.bin"
    )

Step 6: .dockerignore

Files you don't need in the image but are in the repo:

.dockerignore
.git
.gitignore
.env
.env.local
venv/
__pycache__/
*.pyc
.cache/
.pytest_cache/
.notebooks/
data/
logs/
README.md

Prevents accidentally copying your virtualenv (300MB+) or data files into the image.

Step 7: Slim Base (Optional, Extreme)

If you want to go below 2GB, don't use the PyTorch image at all. Start from python:3.11-slim and install PyTorch with --index-url:

FROM python:3.11-slim

RUN pip install --no-cache-dir \
    torch==2.1.0 --index-url https://download.pytorch.org/whl/cu121 \
    transformers==4.35.0 \
    fastapi==0.104.0

# Total: ~1.2GB

The tradeoff: you lose libgl1, libglib2.0-0, and other CUDA system libraries that come pre-installed in the PyTorch image. You'll need to install them manually if your model needs them.

Results

For a FastAPI + DistilBERT inference container:

Optimization	Image Size
Unoptimized (full devel image)	6.8 GB
Runtime image	3.4 GB
+ Combine RUNs, clean cache	2.8 GB
+ Trim requirements	2.3 GB
+ Remove model weights	2.3 GB (but model is 3GB external)
+ .dockerignore	2.2 GB
+ Slim base	1.4 GB

From 6.8GB to 1.4GB. Cold start went from ~90 seconds to ~15 seconds on a decent connection.

Roptal generates optimized Dockerfiles by default. The analysis layer detects what your app actually imports, strips unused dependencies, and produces a Dockerfile with these optimizations applied.