Skip to main content
This guide is for teams that run GPU workloads on Amazon Web Services and want to move the GPU-heavy part (training, inference, batch) to Runpod. Most teams keep S3, IAM, databases, and the rest of the application on AWS, so the guide treats that hybrid setup as the default and a full exit as the second case. Amazon EC2 GPU instances map to Pods. Amazon SageMaker AI real-time, asynchronous, and batch inference map to Serverless endpoints. SageMaker training and processing jobs map to Pods or the Runpod SDK. Amazon Bedrock maps to Public Endpoints or a Serverless vLLM endpoint. AWS ParallelCluster and SageMaker HyperPod map to Instant Clusters. When you finish, the GPU workload runs on Runpod, your callers use a Runpod URL and API key, your Terraform manages both clouds, and the AWS resources you replaced are shut down.
You can hand this migration to a coding agent. See Migrate with a coding agent for the setup commands and a prompt.

How AWS maps to Runpod

What’s different

What works differently

Pods are containers. An EC2 instance boots an AMI with its own kernel, systemd, and Docker daemon. A Pod runs one container image. Docker Compose isn’t available, UDP ports aren’t exposed, and Windows images aren’t supported. Images must be linux/amd64, so images built for Graviton (g5g) or other arm64 instances need a rebuild. Setup moves from boot time to build time. A user data script runs as root at first boot. On Runpod, put the same commands in a Dockerfile, and keep only the final long-running command as the container’s start command. Authentication is a Bearer token. Calls to sagemaker-runtime and bedrock-runtime are signed with SigV4 from IAM credentials. Runpod calls are HTTPS requests with Authorization: Bearer YOUR_API_KEY. There is no request signing and no instance role. Health checks are similar, ports aren’t. SageMaker sends GET /ping to port 8080 and expects a 200. Runpod load balancing endpoints also poll GET /ping by default and treat 200 as healthy, 204 as initializing, and anything else as unhealthy. The default port is 80, set through the PORT and PORT_HEALTH environment variables. See Step 4. Model artifacts aren’t unpacked for you. SageMaker copies model.tar.gz from S3 into /opt/ml/model before your container starts. On Runpod you bake the model into the image, attach a network volume, use cached models for Hugging Face models, or download from S3 at startup. Autoscaling is built into the endpoint. SageMaker registers a production variant with Application Auto Scaling and attaches a policy. A Runpod endpoint has min workers, max workers, an idle timeout, and one scaler (queue delay or request count). Scale to zero is the default (min workers is 0). On SageMaker, scale to zero needs asynchronous inference, Serverless Inference, or inference components with a step scaling policy. Timeouts differ. A SageMaker real-time container must answer /invocations within 60 seconds. A Runpod load balancing endpoint allows 5.5 minutes of processing per request and 30 MB payloads. A queue endpoint’s execution timeout defaults to 10 minutes in the console (5 minutes when you create the endpoint through REST API v2) and can be raised up to 7 days. Storage is tied to a data center. A network volume lives in one data center, and anything that mounts it runs there. Choose the data center by GPU stock first, then create the volume. Billing is prepaid by default. Runpod accounts hold a credit balance. When the balance reaches $0, Pods with a network volume are stopped and Pods without one are terminated. Organizations, the enterprise account model, are invoiced post-paid instead. See Pod pricing.

What has no direct equivalent

On the Runpod side:
  • VPC, security groups, PrivateLink. Runpod has no VPC peering or private link into AWS. Exposed Pod ports are reachable from the internet, and the HTTP proxy URL is unauthenticated, so put your own auth in front of anything you expose. Workaround: call AWS over its public endpoints with scoped credentials, or call your own API gateway.
  • IAM roles and instance profiles. A Pod or worker has no AWS identity. Workaround: create a narrowly scoped IAM user or role, store its keys as Runpod secrets, and inject them as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. These are long-lived credentials that work from anywhere, so limit the policy to the buckets and actions the workload needs and rotate the keys. For single objects, pass presigned S3 URLs in the request instead of credentials.
  • KMS and customer-managed keys. The Runpod S3-compatible API doesn’t support bucket policies or ACLs, so there’s nowhere to attach a KMS key policy. Runpod can encrypt a Pod’s volume disk with a key it manages, but bring-your-own-key isn’t supported, and container disks and network volumes can’t be encrypted this way. Workaround: keep regulated data in S3 under your KMS keys and read it at run time, or encrypt before writing to a volume.
  • Multi-model endpoints, production variants, shadow variants, Model Monitor. One Runpod endpoint runs one image. Workaround: one endpoint per model, a router in your own code inside a load balancing worker, and rolling releases for updates.
  • Bedrock features beyond inference (Guardrails, Knowledge Bases, Agents) and models that are only offered through Bedrock. A vLLM endpoint serves open-weight models, and Public Endpoints serve the listed models. Keep the other calls on Bedrock.
  • Capacity Blocks, Capacity Reservations, and Savings Plans. Runpod has no self-serve future-dated reservation, and its savings plans can no longer be purchased. A running on-demand Pod isn’t displaced by other users, but a stopped Pod releases its GPU, so starting it again depends on stock at that moment. For committed capacity, talk to Runpod sales.
  • Spot Instances. runpodctl, REST API v2, and the Terraform provider create on-demand Pods. Workaround: run Spot-style batch work on on-demand Pods that you delete when the job ends, or on a Serverless queue endpoint that scales to zero. Keep checkpointing to a network volume so a job can resume on a new Pod.
  • Compliance controls as IAM policy. Runpod expresses placement as deploy-time filters instead: Secure Cloud runs in T3 and T4 data centers with partners that hold SOC 2, ISO 27001, and PCI DSS certifications, and runpodctl pod create accepts --country-code and --compliance to restrict where a Pod lands. See Security and compliance.
On the AWS side:
  • Per-second GPU Serverless with scale to zero as the default. SageMaker Serverless Inference doesn’t support GPUs.
  • No data transfer fees. Runpod doesn’t charge for ingress or egress. AWS bills data transferred out of S3 and EC2 to the internet.
  • Cached models. Runpod can cache a Hugging Face model on the host so workers load it without baking it into the image. See cached models.

Requirements

  • An AWS identity that can list and describe EC2, SageMaker, ECR, S3, CloudWatch Logs, and Application Auto Scaling resources, plus permission to delete them at the end.
  • The AWS CLI configured for each account and Region you use.
  • A Runpod account with credits and an API key.
  • Docker, to rebuild or retag images for linux/amd64.
  • Terraform 1.6 or later, if you manage infrastructure as code.

Step 1: Take inventory of what you run on AWS

List every GPU resource before you change anything. Run the commands once per Region. List running GPU instances with their type, AMI, and Availability Zone:
Stopped instances appear too, because their EBS volumes still bill. For each instance, export the user data script, which is your setup recipe:
List launch templates and Auto Scaling groups, which hold the same settings for fleets:
List SageMaker endpoints, then walk each one from endpoint to endpoint configuration to model. The model holds the image URI, the model artifact location, and the environment variables:
An AsyncInferenceConfig in the output means the endpoint is asynchronous and maps to a queue endpoint. Export the autoscaling settings for each variant:
List the remaining SageMaker and Bedrock resources that bill while idle:
List images, storage, and cluster resources:
Then find the code that calls AWS. This search covers boto3 clients, the SageMaker Python SDK, the SageMaker container contract, CLI calls, and Terraform:
A match on 169.254.169.254 is a call to the EC2 instance metadata service, which doesn’t exist on Runpod. Record, for each workload: image, start command, environment variables, ports, disk sizes, GPU model and count, Region, autoscaling limits, and where its data lives.

Step 2: Set up Runpod

Create an account, add credits, and create an API key. Then install the tools.
rp login stores credentials in ~/.runpod/config.toml. On macOS you can install runpodctl with brew install runpod/runpodctl/runpodctl. To let an AI tool manage resources, connect the Runpod MCP server. For infrastructure as code, see Migrate your Terraform. Check which GPUs and data centers have stock before you plan placements:

Step 3: Choose where each workload lands

Choose queue endpoints when every request must be processed, jobs run long, or traffic arrives in bursts. Choose load balancing endpoints for low-latency request and response traffic with your own routes. Load balancing endpoints don’t queue: requests are dropped when all workers are busy, and there’s no built-in retry.

Map EC2 instance types to Runpod GPUs

Runpod selects GPUs by type ID, and Serverless endpoints select by pool. The GPU details below come from the EC2 instance type specifications and the Runpod GPU reference. The EC2 specifications list the A10G and L4 at 22 GiB of accelerator memory and the L40S at 44 GiB. The fractional-GPU g6f sizes and the AMD Radeon g4ad family have no like-for-like type. Use a full 16 GB or 24 GB GPU. Choose by fit for the workload: the VRAM the model needs, then throughput. On EC2 the instance type fixes vCPUs, RAM, and GPU count together. On Runpod you set the GPU type and --gpu-count, and vCPU and RAM come with the GPU. Stock varies by data center and changes through the day, so run runpodctl gpu list before you commit to a type, and give Serverless endpoints more than one pool where your model allows it. For an 8-GPU node or several nodes working together, check stock for that GPU count or use an Instant Cluster.

Step 4: Move your images and code

Make images portable

Runpod runs linux/amd64 images from any registry. Build with the platform flag, especially on Apple silicon or when the image was built for Graviton:
Use a version tag or digest instead of :latest. If your image is built on an AWS Deep Learning Container, it runs on Runpod as long as it doesn’t depend on SageMaker starting it with the serve or train argument. To drop the AWS base, start from a Runpod PyTorch image or an nvidia/cuda image. The host provides the NVIDIA driver, as it does on SageMaker, so don’t install drivers in the image. Set a CUDA floor at deploy time with --min-cuda-version so workers land on hosts that match your framework build.

Keep images in ECR

You can leave images in a private ECR repository. Runpod pulls them through an ECR delegation: you grant two fixed Runpod IAM roles pull access on the repository, then register the repository with Runpod. There are no tokens to rotate.
1

Grant pull access on the repository

Apply this repository policy. The two role ARNs belong to Runpod and must be used exactly as shown.
If your AWS organization uses service control policies, confirm they don’t block ecr:GetAuthorizationToken.
2

Register the delegation with Runpod

Register the repository ARN with REST API v2. Runpod validates the policy at this point, so the call fails if step 1 is missing.
In the console, the same step is Settings, then Container Registry Authentication, Add Credential, and the AWS ECR tab, where you enter the full image URI including a tag. In Terraform, use runpod_registry_delegation.
3

Deploy with the ECR image URI

Use 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-worker:v1.0.0 as the image, and leave the registry credential unset. Runpod matches the image to the delegation. Any tag in a registered repository works.
The private ECR tutorial walks through this for Pods. The REST API v2 reference describes delegations as covering Pods and Serverless endpoints, so test a Serverless deploy from ECR before you depend on it. For other private registries, create a credential with runpodctl registry create and pass its ID as --registry-auth-id.
Don’t store the output of aws ecr get-login-password as a Runpod registry credential. An ECR authorization token is valid for 12 hours, so image pulls for new workers start failing after it expires.

Translate a SageMaker inference.py to a queue endpoint

The SageMaker framework containers load a script that defines model_fn, input_fn, predict_fn, and output_fn. Here is a typical PyTorch script:
inference.py
On Runpod, a handler calls the same four functions. Keep inference.py as it is and add this file:
handler.py
The request body that SageMaker passed as raw bytes arrives as job["input"], already parsed from JSON. The handler’s return value becomes the output field of the response. Package it with the model files that SageMaker used to unpack from model.tar.gz:
Dockerfile
Baking the model into the image keeps /opt/ml/model valid, so inference.py doesn’t change. For large models, put the files on a network volume and set MODEL_DIR=/runpod-volume/my-model instead. If model.tar.gz contains code/requirements.txt, install it in the Dockerfile.

Run a SageMaker inference container on a load balancing endpoint

A bring-your-own SageMaker container already serves POST /invocations and GET /ping on port 8080. That contract lines up with a Runpod load balancing endpoint: To run the container unchanged, set three things when you create the endpoint: PORT=8080 and PORT_HEALTH=8080 as environment variables, 8080/http as the exposed port, and serve as the container command if your entrypoint dispatches on that argument. If /ping returns 200 before the model is loaded, change it to return 204 until the model is ready, so Runpod doesn’t route requests to a worker that can’t serve them. This minimal server shows the contract:
serve.py
Callers reach it at https://ENDPOINT_ID.api.runpod.ai/invocations with a Bearer token. Runpod checks the API key before the request reaches your server.

Turn an EC2 user data script into a Pod image

A GPU instance launched from a Deep Learning AMI often has a user data script like this one:
user-data.sh
On Runpod, the install steps move into the image, and the commands that need run-time data stay in a start script:
Dockerfile
start.sh
The script ends with exec so the server is the container’s main process. Checkpoints go to /workspace, the volume mount, so they survive a Pod stop and the next start skips the download. The instance profile that gave the EC2 instance S3 access is replaced by scoped IAM keys stored as Runpod secrets. Step 6 creates the Pod.

Move training jobs

A SageMaker training container reads data from /opt/ml/input/data/CHANNEL, reads hyperparameters from /opt/ml/input/config/hyperparameters.json, and writes the model to /opt/ml/model, which SageMaker uploads to S3. On Runpod, pass hyperparameters as environment variables or function arguments, read data from a network volume or sync it from S3 at start, and upload results yourself. With the Runpod SDK, a training job is a decorated function that runs on its own Pod and terminates when it returns:
train.py
Run it with rp flash dev train.py. See Build an app for the full set of options.

Step 5: Move your data and models

You don’t have to move data. A Pod or worker can read and write S3 over the public internet with IAM keys you store as secrets. Move data to Runpod when load time or transfer cost matters, which is usually true for model weights and training sets.
AWS charges for data transferred out of S3 to the internet. Copying a dataset to Runpod is a one-time transfer, so estimate it from your bucket size and the Amazon S3 pricing page before you start. Runpod doesn’t charge for ingress or egress. In a hybrid setup, every read from S3 at run time is also egress, which is a reason to keep hot data on a network volume.
Create the network volume in the data center where you plan to run, since everything that mounts it must run there:
The most direct transfer runs inside a Pod with the volume attached, because data flows from S3 straight to Runpod. Create the IAM keys as secrets named aws_access_key_id and aws_secret_access_key, start a Pod that maps them to environment variables, then sync:
Delete the transfer Pod when the sync finishes. The volume keeps the data. There are three other routes:
  • Cloud Sync. On a Pod’s page in the console, Cloud Sync copies between the Pod and an S3 bucket using an access key, secret key, Region, and bucket path. See Sync data with cloud storage.
  • runpodctl. runpodctl send on your machine and runpodctl receive on a Pod copy files with a one-time code. This suits small and medium transfers. See Transfer files, which also covers scp and rsync over SSH.
  • The S3-compatible API. Point the AWS CLI at the volume to upload from your machine without a Pod, as shown below. It needs Runpod S3 API keys and a data center that supports the API.
The Runpod S3-compatible API is not Amazon S3. It uses a per-data-center endpoint, separate S3 API keys from the Runpod console, the data center ID as the Region, and the volume ID as the bucket. Presigned URLs, versioning, bucket policies, ACLs, and object tagging aren’t supported, and aws s3 sync can run into issues on directories with more than 10,000 files. Code that needs those features should keep using S3. See the compatibility reference.
Network volumes mount at /workspace on Pods and /runpod-volume on Serverless workers, so paths in your code may need a prefix change between the two.

Step 6: Deploy on Runpod

Deploy a Pod

Create the Pod from the image you built in Step 4. This replaces aws ec2 run-instances with a user data file:
The security group becomes the --ports list, the instance profile becomes two secret-backed environment variables, and the root volume size splits into a container disk and a persistent volume at /workspace. The server is reachable at https://POD_ID-8000.proxy.runpod.net. That URL is public and has a 100-second request timeout, so turn on your server’s own API key check before you send it real traffic. Check status and logs with:
To override the image’s command at deploy time, use --docker-args in runpodctl or the cmd field in REST API v2. In the console, deploy from Pods, then Deploy.

Deploy a queue endpoint

Push the handler image from Step 4, create a Serverless template for it, then create the endpoint:
--workers-min and --workers-max replace the MinCapacity and MaxCapacity of the Application Auto Scaling target. With --workers-min 0 the endpoint bills nothing while idle. Set --workers-min 1 to keep one worker warm, as an always-on SageMaker instance did. Send a test job:

Deploy a load balancing endpoint

runpodctl serverless create creates queue endpoints, so create load balancing endpoints with REST API v2, Terraform, or the console (Serverless, New Endpoint, then Load Balancer under Endpoint Type). This request deploys the SageMaker container from Step 4 on port 8080:
Load balancing endpoints accept only the REQUEST_COUNT scaler. The response includes requestUrls.base, which is the URL your callers use. Remove "cmd": ["serve"] if your image starts its server without that argument. The endpoint type can’t be changed after creation.

Deploy an LLM endpoint

For Bedrock and JumpStart replacements, check the Public Endpoint model reference first. If your model is listed, there is nothing to deploy. Otherwise deploy the vLLM worker from the Hub with your model:
The endpoint serves an OpenAI-compatible API at https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1. For gated models, add a Hugging Face token. See vLLM configuration for the environment variables.

Deploy a cluster

Create Slurm clusters from the Instant Clusters page by choosing Slurm Cluster as the cluster type, or with the runpod_cluster Terraform resource. Slurm and munge are preinstalled and controller and agent roles are assigned for you. Review your sbatch scripts for paths, modules, and partition names that came from the ParallelCluster configuration. Cluster capacity depends on stock at the time you create it.

Step 7: Update the code that calls AWS

Replace SageMaker runtime calls

A real-time endpoint call moves from a SigV4-signed boto3 call to an HTTPS request with a Bearer token.
On a queue endpoint the raw HTTP form is POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync with the body {"input": {...}}, and the result is under output in the response. /runsync accepts payloads up to 20 MB. A load balancing endpoint that has scaled to zero can return an error while the first worker starts, so retry with a delay, or keep one worker warm.

Replace asynchronous inference

SageMaker asynchronous inference reads the request from S3, writes the result to S3, and notifies through SNS. A queue endpoint takes the request in the call, and posts the result to a webhook.
/run accepts payloads up to 10 MB, so keep large inputs in S3 and pass a presigned URL, which your handler downloads. Anyone who holds a presigned URL can use it until it expires, so set an expiry that covers the time the job can wait in the queue and no longer. Have the handler upload large outputs to S3 with a presigned PUT URL or scoped keys, and return the object location. The webhook receives the job result when it finishes and should return 200. Without a webhook, poll GET /v2/ENDPOINT_ID/status/JOB_ID. Results are kept for 30 minutes after completion. For batch transform, submit one /run job per record or per shard and let the endpoint scale out to its max workers.

Replace Bedrock calls

The Converse API has its own request shape. Runpod’s vLLM workers and LLM Public Endpoints use the OpenAI Chat Completions shape, so the request needs translating.
For a Public Endpoint, use the model’s slug in place of ENDPOINT_ID, for example https://api.runpod.ai/v2/qwen3-32b-awq/openai/v1. If you already call Bedrock through its OpenAI-compatible Chat Completions endpoint, the change is the base URL, the API key, and the model name. Code that uses invoke_model with a model’s native body needs the same translation as Converse, starting from that model’s request format. Tool use and model-specific fields depend on the model and vLLM settings, so test them before cutover.

Replace control-plane automation

Scripts and CI jobs that manage AWS resources move to runpodctl or REST API v2. The CLI command map and API map list the equivalents. Two patterns change shape:
  • Deploying a new model version. UpdateEndpoint with a new endpoint configuration becomes an image change on the endpoint: send PATCH /v2/serverless/{id} with the new image. Runpod creates a new release and deploys it as a rolling release, replacing workers as they cycle. Track it with GET /v2/serverless/{id}/releases. If you change the image with runpodctl template update TEMPLATE_ID --image ... instead, confirm that the endpoint’s workers picked up the new image before you rely on it.
  • Waiting for readiness. Polling for InService becomes waiting for a worker (runpodctl serverless create ... --wait) or polling runpodctl pod get POD_ID until runtimeStatus is running.

Step 8: Verify, cut over, and shut down

1

Send real requests from outside

Call the Runpod endpoint from the same network your callers use, with production-shaped payloads. Compare outputs with the AWS endpoint on a fixed set of inputs. Measure latency for warm requests and for a cold start from zero workers, and decide whether you need a warm worker.
2

Run both in parallel

Keep the AWS endpoint serving. Send a copy of traffic, or a small share, to Runpod and watch error rates, queue delay, and worker counts with runpodctl serverless health ENDPOINT_ID and the endpoint’s Metrics tab.
3

Shift traffic

Move traffic in steps behind a flag or a weighted route in your own gateway. Keep the AWS endpoint until Runpod has carried full load through a peak.
4

Set a cost guard on Runpod

Cap --workers-max, set an idle timeout and an execution timeout on each endpoint, and stop or delete Pods you aren’t using. A stopped Pod still bills for its volume disk. Runpod accounts have a default spend limit of $80 per hour across all resources. Turn on low balance notifications in billing, and review spend with runpodctl billing pods and runpodctl billing serverless.
5

Shut down AWS resources

Delete what you replaced, in this order. Deleting a SageMaker endpoint doesn’t delete its endpoint configuration or model, and terminating an instance doesn’t always delete its volumes.
Remove SageMaker autoscaling, endpoints, and notebooks:
Terminate instances, then find what they left behind:
Delete each leftover resource after you confirm nothing else uses it:
The shutdown checklist:
  • SageMaker endpoints, endpoint configurations, models, notebook instances, and Studio apps.
  • Bedrock Provisioned Throughput and imported models.
  • EC2 instances, Auto Scaling groups (set desired capacity to 0 first, or they relaunch instances), and launch templates.
  • EBS volumes in the available state, and EBS snapshots and AMIs you no longer need.
  • Elastic IPs that aren’t associated, and NAT gateways that only served the GPU subnets.
  • ECR repositories, unless Runpod still pulls from them.
  • EFS and FSx file systems, and S3 buckets you copied in full.
  • CloudWatch log groups, alarms, and dashboards for the deleted resources.
  • Capacity Reservations, which bill whether or not instances run in them. Savings Plans run to the end of their term, so plan the cutover date around them.

Migrate your Terraform

The Runpod Terraform provider 2.x is built on REST API v2. It runs in the same configuration as the AWS provider, so you can add Runpod resources next to the AWS ones, shift traffic, then remove the AWS blocks.

Resource map

Run both providers side by side

Declare both providers in one configuration. The Runpod provider reads RUNPOD_API_KEY from the environment.
versions.tf
Keep whatever AWS provider version constraint you already use. With both providers loaded, Runpod resources can reference AWS ones. This example keeps images in ECR and lets Runpod pull them:
ecr.tf
The depends_on matters because Runpod validates the repository policy when the delegation is created. A delegation can’t be imported or updated in place, so any change replaces it.

GPU instance to Pod

The subnet, security group, and instance profile have no Runpod resource. Ports replace the security group, and secret-backed environment variables replace the instance profile. The key pair becomes account-level SSH keys (runpod_account_ssh_keys), which a Pod receives when you set start_ssh = true. Changing gpu_type_id, gpu_count, cloud, data_center_ids, or min_cuda_version replaces the Pod, so keep state you care about on the network volume. To stop a Pod without removing it from the configuration, set desired_state = "EXITED".

SageMaker endpoint with autoscaling to a Serverless endpoint

Five AWS resources become two. The IAM execution role has no counterpart, and model_data_url has no field: the model must be in the image or on a volume (network_volumes on the endpoint). scaling_value = 4 adds a worker for every four in-flight requests, which is a concurrency target and not the per-minute invocation count that SageMakerVariantInvocationsPerInstance tracks, so derive it from how many concurrent requests one worker handles. workers_min = 1 matches the always-on instance. Set it to 0 to scale to zero. For a queue endpoint, set endpoint_type = "QUEUE", drop ports, and choose QUEUE_DELAY (seconds a request may wait) or REQUEST_COUNT. A queue endpoint that scales on REQUEST_COUNT doesn’t take idle_timeout, so drop that too. The endpoint copies the template’s settings when it’s created, so pass later image changes through the endpoint.

Cut over in Terraform

  1. Add the Runpod resources and run terraform apply. The AWS resources are untouched.
  2. Expose the Runpod URL as an output or pass it to the resource that configures your callers, then shift traffic.
  3. Remove the AWS resource blocks in a separate change and apply it. Terraform destroys them.
  4. Check for leftovers that Terraform didn’t manage, using the commands in Step 8.
For every argument, see the provider reference on the Terraform Registry.

Reference

CLI command map

Run runpodctl COMMAND --help for current flags.

API map

Management routes use https://api.runpod.io/v2. Inference routes use https://api.runpod.ai.

Configuration field map

Environment variable map

Status map

A Runpod endpoint has no single endpoint-level status like InService. Read worker states from runpodctl serverless health ENDPOINT_ID.

Migrate with a coding agent

A coding agent with the Runpod skills and MCP server can run this migration with you: it reads your AWS inventory, proposes a plan, and creates Runpod resources after you approve them.
The MCP server signs in with OAuth (“Sign in with Runpod”), so you don’t need an API key to start. See Agent skills and MCP servers for details. Then give the agent this prompt from the root of your repository:

Troubleshooting

Workers start, then requests return 502 and workers recycle after about 8 minutes. The load balancer can’t reach your health check. SageMaker containers listen on 8080, and Runpod checks port 80 unless told otherwise. Set both PORT and PORT_HEALTH to 8080 and expose 8080/http. The container exits immediately on Runpod but ran on SageMaker. SageMaker starts inference containers with the serve argument. Without it, an entrypoint that dispatches on serve or train exits. Set the container command to serve, or change the image’s CMD. FileNotFoundError under /opt/ml/model. Nothing unpacks model.tar.gz on Runpod. Bake the files into the image, or mount a network volume and point your code at it. Unable to locate credentials or NoCredentialsError from boto3. There’s no instance profile or execution role. Create a scoped IAM user, store its keys as Runpod secrets, and map them to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the template’s environment variables. Set AWS_DEFAULT_REGION too. Requests to 169.254.169.254 hang. Code is calling the EC2 instance metadata service. Replace those lookups with the RUNPOD_ environment variables and explicit configuration. exec format error when the container starts. The image was built for arm64 (Graviton or Apple silicon). Rebuild with --platform linux/amd64. Image pull fails for an ECR image. Confirm the repository policy lists both Runpod role ARNs exactly, that a delegation exists for that repository (GET /v2/registries/delegations), and that the tag exists. If you stored a get-login-password token as a registry credential, it expired after 12 hours. Use a delegation instead. {"error":"no workers available"} from a load balancing endpoint. The endpoint scaled to zero and the first worker hasn’t passed its health check. Retry with a delay of 5 to 10 seconds, or set min workers to 1. CUDA or driver errors on some workers only. Hosts differ in driver version. Set --min-cuda-version (REST gpu.minCudaVersion) to the version your framework build needs. A Pod can’t see a network volume, or deploy fails for lack of capacity. The volume pins the Pod to its data center, and the GPU you asked for isn’t in stock there. Pick a GPU type that data center has, or create the volume in a data center with stock for your GPU.

Next steps

Serverless overview

Learn how endpoints, workers, and scaling fit together.

Load balancing endpoints

Review the health check, port, and timeout contract.

Terraform provider

Manage Pods, endpoints, and volumes as code.

Private ECR images

Set up cross-account pulls from Amazon ECR.
Found a gap or an error in this guide? Open an issue in the Runpod docs repository.
Last modified on November 14, 2026