@app.task in the Runpod SDK or to a Pod. At the end, each workload runs on Runpod, your clients call Runpod URLs with a Runpod API key, and the Hugging Face compute is paused or deleted.
How Hugging Face maps to Runpod
What’s different
What works differently
Where the model lands inside the container. Inference Endpoints mounts the selected repository at/repository. Runpod’s cached models land in the standard Hugging Face cache layout at /runpod-volume/huggingface-cache/hub/models--ORG--NAME/snapshots/HASH/. Code that loads from /repository needs the snapshot path instead. Step 4 has a helper that resolves it.
What happens to a request when nothing is running. An Inference Endpoint that has scaled to zero answers with status 503 while the new replica initializes, unless the client sends the X-Scale-Up-Timeout header to make the proxy hold the request for a set number of seconds. A Runpod queue-based endpoint puts the request in the endpoint’s queue and a worker picks it up when it’s ready, so you can remove 503 retry loops. Load balancing endpoints don’t queue: a request that arrives before a worker is ready can return no workers available, and it times out after 2 minutes without a worker.
The first synchronous request after a cold start. /runsync waits up to 90 seconds by default, and synchronous routes sit behind an edge timeout of about 100 seconds. A cold vLLM worker loading a large model can take longer than that. Use /run and poll /status for the first request, or keep one active worker for latency-sensitive endpoints.
Idle time is measured in seconds. Scale-to-zero on Inference Endpoints is set in minutes (1 hour by default). Runpod’s idle timeout is set in seconds, from 1 to 3600. The console default is 5 seconds and the REST v2 default is 10 seconds. A worker is billed while it waits out the idle timeout.
Authentication. Inference Endpoints has three endpoint types: public (no authentication), authenticated (a Hugging Face token), and private. Requests to a Runpod Serverless endpoint carry Authorization: Bearer RUNPOD_API_KEY. If you run a public endpoint today, put your own API or gateway in front of the Runpod endpoint and keep the key server-side.
Request and response envelope. The Inference Toolkit accepts your JSON body as is, with inputs at the top level. A queue-based endpoint expects {"input": {...}} and returns your handler’s result under output, next to id, status, delayTime, and executionTime.
The model field on OpenAI-compatible calls. On a vLLM worker, model must be the Hugging Face repo ID you deployed, or the name you set in OPENAI_SERVED_MODEL_NAME_OVERRIDE. If your clients send the Inference Endpoint’s name as model, set that override to the same string and they won’t need a code change beyond the URL and key.
TGI’s native routes. TGI serves /generate and /generate_stream next to its OpenAI-compatible routes. The vLLM worker serves /chat/completions, /completions, and /models. Move /generate callers to /completions. Hugging Face put TGI in maintenance mode in late 2025 and recommends vLLM or SGLang, so this is the same move you’d make on Inference Endpoints.
Autoscaling signals. Inference Endpoints scales on hardware utilization, pending requests, or (in beta, for vLLM and SGLang) inference load. Runpod scales on queue delay (seconds a request waits) or request count (queued plus in-progress requests divided by a target). Request count is the closer match to pending requests.
Pause and resume. A paused Inference Endpoint stays defined and isn’t billed. On Runpod, an endpoint at zero workers isn’t billed for compute, so there’s no separate paused state. To block new workers from starting, set max workers to 0.
Unused endpoints scale down on their own. After 3 days without a request, Runpod reduces an endpoint’s max workers to 2 and emails you. After 7 days it sets max workers to 0, and it stays there until you raise it. An Inference Endpoint with min replicas above 0 keeps running until you change it.
Worker limits. Your account balance sets the maximum number of workers across all endpoints (5 by default). See Max worker limits before you size a large endpoint.
What has no direct equivalent
In the other direction, moving gives you a few things to build on: a request queue with async jobs, status polling, and webhooks on queue-based endpoints, persistent network volumes shared across workers, and per-second billing.
Requirements
- A Hugging Face account with access to the endpoints, Spaces, and Jobs you’re moving, and the
hfCLI logged in (hf auth login). - A Hugging Face access token with read access to every gated or private repo you serve.
- A Runpod account with credits and a Runpod API key.
- Docker, if you’re moving a custom handler or a custom container.
- Python 3.10 to 3.14, if you use the Runpod SDK.
Step 1: Take inventory of what you run on Hugging Face
List every Inference Endpoint in every namespace you belong to, then print the full configuration of each one:export_endpoints.py
model.image in raw, such as vllm, tgi, tei, llamacpp, huggingface, or custom) and whether the model repository contains a handler.py. Those two facts decide where the endpoint lands in Step 3.
Then list the rest of your Hugging Face compute:
cpu-basic is using paid compute. Note each Space’s SDK (Gradio, Docker, or static), its app_port, and its variables and secrets.
Finally, find the code that calls Hugging Face compute or manages it:
from_pretrained, hf_hub_download, snapshot_download, load_dataset) doesn’t need to change.
Step 2: Set up Runpod
Installrunpodctl and the Runpod SDK, which includes the rp CLI, then store your API key:
runpodctl doctor walks through first-time setup, and rp login saves credentials to ~/.runpod/config.toml. To let a coding agent manage resources, add the MCP server. If you manage infrastructure as code, the Terraform provider covers endpoints, templates, and volumes.
Next, store your Hugging Face token as a Runpod secret so it never appears in an endpoint’s configuration:
runpod. Reference the secret from any environment variable as {{ RUNPOD_SECRET_hf_token }}.
Step 3: Choose where each workload lands
Queue-based or load balancing is the main decision for an Inference Endpoint. Choose queue-based when requests can wait for a worker, run longer than a few minutes, or need retries. Choose load balancing when you need your own routes, non-JSON payloads, or WebSockets, and your requests finish within the 5.5 minute processing limit and the 30 MB payload limit.
GPU mapping
Match on VRAM and GPU generation, then confirm the model fits. Instance names come from the Inference Endpoints pricing page, and pools and GPU IDs come from the GPU types reference.
The instance size (
x1, x2, x4, x8) is the number of GPUs per replica. Set the same number with --gpu-count, or gpu.count in REST v2. You can list up to three GPU choices in priority order on one endpoint, which helps when your first choice is busy. Stock varies by data center, so check live availability with runpodctl gpu list or GET /v2/catalog/gpus?include=AVAILABILITY&product=SERVERLESS before you commit to a GPU.
Step 4: Move your images and code
LLM endpoints (vLLM, TGI, SGLang)
There’s no image to build. The vLLM worker takes the same repo ID, and engine settings become environment variables. Anyvllm serve flag works as an upper snake case variable (--max-model-len 8192 becomes MAX_MODEL_LEN=8192), and VLLM_EXTRA_ARGS passes flags through as written.
TGI’s batching limits (Max Input Tokens, Max Batch Prefill Tokens, Max Batch Total Tokens) have no one-to-one vLLM setting. Start with
MAX_MODEL_LEN and MAX_NUM_SEQS, then tune from there. See vLLM environment variables and Configure vLLM for model-specific settings.
Custom handlers (handler.py)
On Inference Endpoints, the Inference Toolkit imports EndpointHandler from a handler.py in your model repository, calls __init__ once with the path to the weights, and calls __call__ with the request body for every request. Here’s a typical handler:
handler.py (Hugging Face)
job["input"]:
handler.py (Runpod)
path is now the snapshot directory of the cached model, which Runpod places at /runpod-volume/huggingface-cache/hub/ when you attach a model to the endpoint (see Use Hugging Face models). The handler keeps the inputs and parameters keys, so clients only wrap their existing body in input. Errors are returned as {"error": "..."}, which marks the job as FAILED.
Package the handler with a Dockerfile. Copy the dependencies from the requirements.txt in your model repository and add runpod:
Dockerfile
requirements.txt
linux/amd64 and push to a registry Runpod can pull from:
runpodctl registry create and pass the ID with --registry-auth-id on the template. You can also skip the registry and deploy from GitHub.
Custom handlers without a Dockerfile
The Runpod SDK builds and deploys the worker from a Python file.runpod.Model stages the weights from the Hub before the worker starts, and model.path points to them:
main.py
rp flash dev main.py, then deploy with rp flash deploy. For a gated model, add env={"HF_TOKEN": runpod.Secret("hf_token")} to the decorator. This example builds the pipeline inside the function, the same way the SDK docs do. To load the model once per worker, serve it from an @app.api class and load it in a method marked @runpod.init. See Build an app and Resource parameters.
Custom containers, TEI, and llama.cpp
A container that serves HTTP on Inference Endpoints runs on a load balancing endpoint with three changes:
If you don’t want to rebuild the image, set
HEALTH_CHECK_PATH to your existing health route. Changing PORT doesn’t change PORT_HEALTH, so set both, and expose the port on the endpoint. If the server loads the model from /repository, make that path configurable and point it at the cached snapshot directory or a Hub repo ID.
For TEI, run the image from the TEI docs (for example ghcr.io/huggingface/text-embeddings-inference:cuda-1.9). TEI reads MODEL_ID, HF_TOKEN, and PORT from the environment and serves /health, /embed, /rerank, /predict, and /v1/embeddings. Runpod’s docs don’t cover a dedicated embeddings worker, but the Hub has an embedding category you can search:
runpodctl hub get before you deploy it.
Spaces and Jobs
A Docker Space is already a container image. Build it forlinux/amd64, push it, and run it on a Pod with the Space’s app_port exposed (the default is 7860). For a Gradio or Streamlit Space, add a Dockerfile that installs requirements.txt and starts the app bound to 0.0.0.0. Space variables and secrets become Pod environment variables and Runpod secrets.
A Job is an image, a command, and a hardware flavor. With the Runpod SDK, the same work is a function:
- Hugging Face Jobs
- Runpod SDK
train provisions one Pod and terminates it when the function returns. Scheduled Jobs (hf jobs scheduled run) have no built-in counterpart: trigger the task from your own scheduler or CI.
Step 5: Move your data and models
In most migrations there’s nothing to copy, because the weights are on the Hub. Pick how each endpoint gets them:
Cached models are the closest match to how Inference Endpoints delivers weights. Runpod schedules workers on hosts that already hold the model where it can, and you aren’t billed while a model downloads. An endpoint supports one cached model, and a repo with several quantization variants downloads all of them. The reference includes a ref (
:main, a tag, or a commit), so pin the commit you serve today if your Inference Endpoint pins a revision.
To pre-load a network volume, create it in the data center you’ll serve from, start a small Pod with the volume attached, and download from the Hub:
/runpod-volume/models/. Delete the loader Pod when the download finishes.
Fine-tuned weights that only exist on a replica’s disk or in a Space’s ephemeral storage should be pushed to a Hub repository first (hf upload), then served from there. Datasets on the Hub load with load_dataset from Runpod as they do today.
Step 6: Deploy on Runpod
vLLM worker for an LLM endpoint
Here’s an Inference Endpoint definition and its Runpod equivalent:- Hugging Face
- Runpod
--hub-id pulls the image, container disk size, and default environment from the Hub release, and --env overrides or adds variables. --model-reference attaches the cached model, so workers start with the weights already on the host where possible. The command prints the endpoint ID that goes in every request URL. In the console, the same deployment is Serverless > New Endpoint > Hugging Face, where you enter the repo ID and token.
Replicas translate directly: min_replica is --workers-min (active workers, billed continuously) and max_replica is --workers-max. scale_to_zero_timeout=15 is in minutes, and --idle-timeout is in seconds with a maximum of 3600. A short idle timeout costs less and causes more cold starts.
Queue-based endpoint for a custom handler
Create a Serverless template for the image you pushed in Step 4, then create the endpoint from it:- runpodctl
- REST API v2
--template-id, environment variables come from the template, so set them on template create. Each Serverless template binds to one endpoint. In REST v2, type and scaling are required, timeout is in milliseconds, and FlashBoot is off unless you set it. The REST example creates the endpoint without a cached model, and the handler above loads from the cache, so its workers fail at startup with Cached model not found until you attach the model in the Model field in the console. To attach it at creation time, use the runpodctl tab.
Load balancing endpoint for a custom container or TEI
Create load balancing endpoints through REST v2 or the console (Serverless > New Endpoint, endpoint type Load Balancer). This example runs TEI:https://ENDPOINT_ID.api.runpod.ai/, and every path your server defines is available under it. For your own container, swap in your image and port, and set HEALTH_CHECK_PATH to the health route you configured on Inference Endpoints. This TEI configuration downloads the model when a worker starts, so expect a longer cold start than with a cached model.
Public Endpoint for an Inference Providers call
There’s nothing to deploy. Check the model reference for the model you call throughrouter.huggingface.co. If it’s listed, switch the client in Step 7. If it isn’t, deploy it with the vLLM worker above.
Pod for a Space
Run the Space’s image on a Pod and expose its port over HTTP:https://POD_ID-7860.proxy.runpod.net.
A Pod bills per second while it runs, whether or not anyone is using the app, like an upgraded Space with no sleep time. If you relied on a Space’s sleep time, stop the Pod yourself with runpodctl pod stop POD_ID and start it again with runpodctl pod start POD_ID.
Step 7: Update the code that calls Hugging Face
OpenAI-compatible clients
Swap the base URL and the key. Everything else in the call stays the same.- Before (Inference Endpoints)
- After (Runpod)
model is the repo ID you deployed, or the value of OPENAI_SERVED_MODEL_NAME_OVERRIDE. Streaming works with stream=True. Raise the client’s timeout for the first request after a cold start, and drop the X-Scale-Up-Timeout header.
InferenceClient
InferenceClient accepts a base_url and exposes the same chat.completions.create method, so you can keep it and point it at Runpod:
client.text_generation() or endpoint.client.text_generation(), have no route on the vLLM worker. Replace them with chat.completions.create or the OpenAI client’s completions.create.
TGI /generate callers
Move raw /generate requests to the completions route and rename the parameters:
- Before (TGI)
- After (Runpod)
choices[0].text. /generate_stream callers move to the same route with "stream": true.
Custom handler clients
Wrap the body ininput, change the URL and key, and read the result from output:
- Before (Inference Endpoints)
- After (Runpod)
/runsync returns the result when the job finishes within the wait window. If the job is still queued or running when the window closes, the response has a status of IN_QUEUE or IN_PROGRESS, and you fetch the result from /status/JOB_ID. For anything longer than a few seconds, submit to /run, which returns a job ID right away, then poll /status/JOB_ID or pass a webhook URL in the request. Results are kept for 30 minutes after an async job and 1 minute after a sync job. Payloads are limited to 10 MB on /run and 20 MB on /runsync, so send large inputs as URLs. The same calls are available from Python through runpod.Endpoint("ENDPOINT_ID").run_sync({...}). See Send API requests.
Custom container and TEI clients
Only the host and the key change. Paths and payloads stay the same:no workers available. Use at least 3 retries with a 5 to 10 second delay.
Inference Providers clients
For a model that’s listed as a Public Endpoint with OpenAI compatibility, such as Qwen3 32B, swap the router URL for the model’s endpoint:- Before (Inference Providers)
- After (Runpod Public Endpoint)
:fastest, :cheapest, :groq) on the model name. The Public Endpoint may serve a different variant of the model than the provider you used (here, an AWQ quantization), so compare outputs before you switch. Image, video, and audio models use /runsync with an input object. See Make API requests.
Control plane automation
After any update that sets a worker count to 0, confirm it with
runpodctl serverless get ENDPOINT_ID, and update runpodctl first if it’s an old build.
A batch script that resumes an endpoint, waits, runs inference, and pauses it again gets simpler: submit the jobs to /run on an endpoint with zero active workers. Workers start for the queue and stop when it’s empty.
Step 8: Verify, cut over, and shut down
-
Send a real request from outside Runpod. Check the endpoint’s health, then send the same prompt or payload you send to Hugging Face:
-
Compare outputs and latency. Run a fixed set of requests against both platforms with the same sampling settings (set
temperatureto 0 where you can). Expect small differences when the engine changes, for example TGI to vLLM, or when the quantization differs. Measure a cold start as well as warm latency: the response includesdelayTimeandexecutionTime, and the endpoint’s Metrics tab shows cold start percentiles. - Run both in parallel. Keep the Inference Endpoint up and send a share of traffic to Runpod through a feature flag or your gateway.
- Shift the rest of the traffic, then watch error rates and queue delay for a full traffic cycle.
-
Stop billing on Hugging Face. Pause first so you can roll back, then delete once you’re sure:
Deleting an Inference Endpoint is permanent and removes its configuration, logs, and usage metrics. Keep the model repositories and the token that Runpod uses to read them. Revoke tokens that only the old clients used.
-
Set cost guards on Runpod. Keep
--workers-minat 0 unless you need warm capacity, set--workers-maxto the concurrency you’re willing to pay for, set an execution timeout on every endpoint, and keep the idle timeout short. Delete loader Pods and unused network volumes, which bill for storage until you remove them. Track spend on the billing page.
Reference
CLI command map
API map
The Inference Endpoints API is athttps://api.endpoints.huggingface.cloud. Runpod’s REST API v2 is at https://api.runpod.io. Both use Authorization: Bearer.
Configuration field map
Environment variable map
Status map
Inference Endpoints reports one state per endpoint. Runpod reports a state per worker, and a state per job on queue-based endpoints.
Job states on queue-based endpoints are
IN_QUEUE, IN_PROGRESS, COMPLETED, FAILED, CANCELLED, and TIMED_OUT.
Migrate with a coding agent
Install the Runpod skills and connect the MCP server:- Claude Code
- Other agents
Troubleshooting
The worker fails withCached model not found or tries to load /repository. The handler still expects Hugging Face’s mount path, or the endpoint has no cached model attached. Attach the model with --model-reference or the Model field, and load from the snapshot directory under /runpod-volume/huggingface-cache/hub/.
A gated model returns 401 or 403 while a worker starts. The endpoint has no HF_TOKEN, or the token’s account hasn’t accepted the model’s terms. Set HF_TOKEN to {{ RUNPOD_SECRET_hf_token }} and confirm the token can read the repo. Workers pick up a changed secret on their next start.
The first request after idle returns a 524 or times out. The cold start took longer than the synchronous wait. Send the first request to /run and poll /status, raise the idle timeout, or set one active worker.
OpenAI calls return an invalid model error. The client still sends the Inference Endpoint’s name as model. Send the Hugging Face repo ID, or set OPENAI_SERVED_MODEL_NAME_OVERRIDE to the name your clients use.
/generate returns 404 or an error. TGI’s native routes aren’t served by the vLLM worker. Call /openai/v1/completions or /openai/v1/chat/completions.
A load balancing worker never becomes healthy. The load balancer polls /ping on port 80 by default. Set HEALTH_CHECK_PATH to your health route, set PORT and PORT_HEALTH to your server’s port, and expose that port on the endpoint. With misconfigured ports, workers stay up for 8 minutes and return 502.
Requests return no workers available. A load balancing endpoint has no healthy worker yet. Retry with a delay, or keep an active worker. If the request can wait, a queue-based endpoint holds it for you.
The vLLM worker runs out of GPU memory on a model that fit before. vLLM pre-allocates VRAM for its KV cache, and your new GPU may have less memory than the old instance. Lower MAX_MODEL_LEN or GPU_MEMORY_UTILIZATION, or move up a pool.
Workers stay Throttled or Initializing for a long time. The selected GPU is busy in the allowed data centers, or the model is still downloading to the host. Add a second and third GPU choice, remove data center restrictions, and for large models raise RUNPOD_INIT_TIMEOUT.
An endpoint that worked last week returns no workers. Runpod set max workers to 0 after 7 days without requests. Raise max workers again.
Next steps
vLLM workers
Configure and tune LLM endpoints.
Cached models
Serve Hugging Face models without baking them into images.
Endpoint settings
Set workers, timeouts, and autoscaling.
Runpod SDK
Define endpoints and tasks in Python.