Deploy a Hugging Face Model to AWS SageMaker: Docker, IAM, and Endpoint Setup

Hugging Face has an AWS integration, but production deployment still comes down to three things: a container image, an IAM role, and a SageMaker endpoint configuration. This guide covers the parts that usually cause failed deployments.

The Deployment Shape

For a custom Hugging Face model, the request path is:

Client -> SageMaker endpoint -> Docker container -> FastAPI handler -> model

SageMaker does not need to know whether you use transformers, PyTorch, or a custom tokenizer. It starts a container and routes HTTP requests to it. Your container needs to expose a predictable handler and return JSON.

Start With a Small FastAPI Handler

from fastapi import FastAPI
from pydantic import BaseModel
from transformers import pipeline

app = FastAPI()
classifier = pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english")

class PredictionRequest(BaseModel):
    text: str

@app.get("/health")
def health():
    return {"status": "ok"}

@app.post("/predict")
def predict(request: PredictionRequest):
    return classifier(request.text)[0]

Keep the health endpoint separate from inference. An endpoint can have a running web process but a failed model load. Your health check should only return 200 after the model is ready.

Dockerfile Requirements

FROM pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .
RUN useradd --create-home --uid 1000 appuser && chown -R appuser:appuser /app
USER appuser

EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
  CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Use the CUDA runtime image rather than the development image. It avoids shipping compilers and headers that are unnecessary at inference time.

IAM Role: Minimum Permissions

The SageMaker execution role needs access to the image in ECR and any model artifacts stored in S3.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": "arn:aws:s3:::your-model-bucket/*"
    }
  ]
}

Avoid attaching AdministratorAccess just to get the first deployment working. It makes later security review much harder.

Build and Push to ECR

aws ecr create-repository --repository-name sentiment-api
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com

docker build -t sentiment-api .
docker tag sentiment-api:latest ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/sentiment-api:git-sha
docker push ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/sentiment-api:git-sha

Tag images with a Git SHA or release version. Do not deploy latest; it makes rollback ambiguous.

Endpoint Sizing

For a small BERT-style classifier, ml.g4dn.xlarge is a reasonable starting point:

• NVIDIA T4 GPU with 16GB VRAM
• 4 vCPU
• 16GB system memory
• Around $0.35/hour on-demand in us-east-1

For low traffic, CPU inference may be cheaper. Run a load test first. A model that takes 30ms on GPU but receives 100 requests per day does not need a permanent GPU endpoint.

Common Deployment Failures

Endpoint stuck in Creating

Usually an ECR pull failure or container crash. Check CloudWatch logs for ModuleNotFoundError, missing system libraries, or an incorrect entrypoint.

Health check fails but the container starts

The model probably loads after the health check starts. Add a startup grace period or only expose the health endpoint after model initialization completes.

CUDA out of memory

Use FP16 for compatible models, reduce batch size, or move to a GPU with more memory. Do not solve it by restarting the endpoint repeatedly.

A Better Operating Model

The container build is only one part of production. You also need a release strategy, logs, health monitoring, rollback, and cost controls. Roptal connects the repository, generates reviewable deployment files, and deploys the same image to your AWS account without taking ownership of your infrastructure.