Localized reading guide
hf-cloud-serving-image-selectionPurpose, usage and source
Pick the right serving container for a SageMaker model deployment and find its current image URI. Use this skill whenever about to deploy a model to a SageMaker endpoint and an image URI needs to be chosen — including when the user says "deploy this LLM", "host this HuggingFace model", "serve this fine-tuned model", "deploy this embedding model", "host a reranker", "serve a sentence-transformers model", or when about to hardcode any container URI in deployment code. HuggingFace-curated Deep Learning Containers are ALWAYS preferred: HuggingFace vLLM (LLMs and generative rerankers), HuggingFace vLLM-Omni (multimodal), TEI (embeddings/cross-encoder rerankers), HF Inference Toolkit (other transformers). Generic images (AWS vLLM, DJL-LMI, SGLang) are used only when no HuggingFace image is compatible — never merely because they carry a newer version. Never hardcode a container URI from memory and never default to TGI. Prevents stale-image failures and wrong-region URIs.
Source retrieved:
- Source repository
- huggingface/skills
- SKILL.md
skills/hf-cloud-serving-image-selection/SKILL.md- Repository stars
- 11,241
- Repository updated
- 2026-10-30
- Categories
- Cloud
- Last collected
License: Apache-2.0
Workflow excerpts are reading aids, not executable instructions from Codez. No third-party Skill was run to produce this page.
The purpose and workflow passages below are upstream material, organized and translated for reading. They are distinct from the separately labeled Codez editorial analysis.
What this Skill does
Pick the right serving container for a SageMaker model deployment and find its current image URI. Use this skill whenever about to deploy a model to a SageMaker endpoint and an image URI needs to be chosen — including when the user says "deploy this LLM", "host this HuggingFace model", "serve this fine-tuned model", "deploy this embedding model", "host a reranker", "serve a sentence-transformers model", or when about to hardcode any container URI in deployment code. HuggingFace-curated Deep Learning Containers are ALWAYS preferred: HuggingFace vLLM (LLMs and generative rerankers), HuggingFace vLLM-Omni (multimodal), TEI (embeddings/cross-encoder rerankers), HF Inference Toolkit (other transformers). Generic images (AWS vLLM, DJL-LMI, SGLang) are used only when no HuggingFace image is compatible — never merely because they carry a newer version. Never hardcode a container URI from memory and never default to TGI. Prevents stale-image failures and wrong-region URIs.
How the upstream workflow works
Workflow
For every family: read the URI from the AWS catalog page. Find the section for the right family (e.g. "HuggingFace vLLM Inference" for HuggingFace LLMs, "HuggingFace Text Embeddings Inference" for embeddings) Pick the newest row marked SageMaker for the platform column — newest within that family. Do not switch to another family's section because it lists a higher engine version (see "Rule zero") Substitute with the user's region (from hf-cloud-aws-context-discovery)
Serving Image Selection
The serving container is the single thing most likely to break a SageMaker deployment that "looked correct on paper". Wrong container, stale tag, or the wrong AMI — all produce the same opaque Failed to pass health check error.
Rule zero: HuggingFace images always win
When both a HuggingFace-curated family (huggingface-vllm, huggingface-vllm-omni, huggingface-sglang, tei, huggingface-pytorch-inference) and a generic family (vllm, vllm-omni, sglang, djl-inference) can serve the model, the HuggingFace one is mandatory, not preferred. The only valid reasons to use a generic image: Verified incompatibility — the model needs an architecture/modality/feature no available HuggingFace tag supports, confirmed against the catalog (not assumed).
curl -s https://huggingface.co/<model-id>/raw/main/config.json
# "architectures": ["Qwen3ForCausalLM"] → vLLM
# "architectures": ["XLMRobertaForSequenceClassification"] → TEIRequirements and setup
vLLM AMI requirement
vLLM DLC images with CUDA 13 or higher (current default: cu130) require setting InferenceAmiVersion=al2-ami-sagemaker-inference-gpu-3-1 on the ProductionVariant. This applies equally to huggingface-vllm and huggingface-vllm-omni (layered on the same cu130 base) and to the AWS vllm repo. Without it the container dies on startup with no CloudWatch logs ever created. The failure looks identical to many other things (account-level issues, quota, networking) and routinely sends people down wrong diagnostic paths. Rule of thumb: if the vLLM tag you picked contains cu130 or later, pass --inference-ami-version al2-ami-sagemaker-inference-gpu-3-1 to deploy.py. If a future CUDA version (cu140+) needs a different AMI, add a row to the table when AWS publishes the new image.
CUDA / instance compatibility
Critical and easy to get wrong: The driver comes from the host AMI, not the instance family — so passing the gpu-3-1 AMI (which vLLM cu130 images require anyway) also makes ml.g5.* viable for cu129+ images.
Limits and quality checks
No separate limits passage was selected from this file. This does not mean the Skill is safe for every input or environment. Review the complete source and validate your own result.
Start with a safe trial
- Check the supported host and the repository installation instructions. Keep the Skill directory and its referenced scripts, assets and documentation together; SKILL.md alone may omit dependencies.
- Give the agent your real input, desired output and acceptance criteria. Invocation syntax and available tools depend on the host; this site does not assume a universal install command.
- Try the source workflow in a disposable project or on a copy. Review operations that change files, remote services or billable resources before allowing them.
- Validate the output against the source quality checks and your own requirements. A successful trigger or a high repository Star count does not establish a correct result.
Read further in the source
- Serving Image Selection
- Rule zero: HuggingFace images always win
- Where image URIs come from
- Quick decision
- Rerankers: TEI or vLLM?
- Workflow
- TEI: pick the right variant
- vLLM AMI requirement
- Configuring the vLLM DLCs (HuggingFace vLLM and AWS vLLM)
- Required for every HuggingFace LLM deployment
- Tuning (optional)
- Configuring TEI
- CUDA / instance compatibility
- VPC / NAT gateway problem
- When the catalog page won't render, is stale, or is wrong
- Known-broken images (last checked July 2026)
- Hub download time at first boot
Related Skills
Translation and scope
This reading guide selects passages from the pinned upstream SKILL.md. Translation is assisted. It is not a complete manual or a claim of tested compatibility. Read the original and its referenced files before use.
This introduction refers to the pinned version below; current repository activity and collection time are shown separately.
- Pinned source
- Read the original ↗
- Source retrieved
- Source commit
fabc634a893f62ad5c2039830397efa35fa62715- Source SHA-256
6af6adfe1cdcf7bcc061ee11e1e2cf88bae9fbb08bf79d16945ce748839c1d76