Google’s documentation now calls Vertex AI “Gemini Enterprise Agent Platform”. The
gcloud ai commands, the Vertex AI SDK for Python, the aiplatform.googleapis.com API, and the Terraform google_vertex_ai_* resources keep their names, so this guide says Vertex AI throughout.How Google Cloud maps to Runpod
What’s different
What works differently
Pods are containers, not VMs. A Compute Engine VM boots an OS image and runs a startup script as root on every boot. A Pod starts your container image and runs its start command. Move startup-script logic into the image or the Pod’s start command. Pods don’t run Docker Compose, Windows, or UDP services. Health checks are required for load-balancing endpoints. Vertex AI sends health checks to the route inAIP_HEALTH_ROUTE and Cloud Run only needs your server to listen on PORT. A Runpod load-balancing worker must answer GET /ping (or the path in HEALTH_CHECK_PATH) with 200 when ready and 204 while it’s still loading. Any other code marks the worker unhealthy.
Ports default to 80, not 8080. Vertex AI sets AIP_HTTP_PORT to 8080 by default and Cloud Run sets PORT to 8080 by default. Runpod load-balancing workers read PORT and PORT_HEALTH, which both default to 80. Set both to your server’s port and expose that port on the endpoint.
Request shape for queue endpoints. Vertex AI sends {"instances": [...], "parameters": {...}} and expects {"predictions": [...]}. A Runpod queue endpoint takes {"input": {...}} and returns your handler’s return value under output. On a load-balancing endpoint the request and response bodies are whatever your server defines, so you can keep the Vertex AI shape.
Limits on load-balancing endpoints. Cloud Run requests time out after 5 minutes by default and can be extended to 60 minutes. Vertex AI shared public endpoints time out at 60 seconds and dedicated endpoints are configurable up to 1 hour. Runpod load-balancing endpoints allow 5.5 minutes of processing per request and 30 MB per request or response, and they don’t queue: a request that finds no available worker within 2 minutes fails. For longer work, or work that must not be dropped, use a queue endpoint, where the execution timeout can be raised up to 7 days.
Scale to zero is the default. A Runpod endpoint starts with zero active workers and scales up on demand. Vertex AI deployments default to a minimum of one replica, and setting the minimum to 0 opts in to scale to zero, which Google lists as a Preview feature. On Runpod, set active (minimum) workers above zero only when you need to remove cold starts, because active workers bill continuously.
Concurrency is expressed through the scaler. Cloud Run has a per-instance concurrency setting (up to 1,000). On Runpod you set the REQUEST_COUNT scaler value, and Runpod sizes the worker count as in-flight requests divided by that value. Your server handles the concurrent requests it receives.
Authentication is an API key. Google Cloud callers use OAuth access tokens (Vertex AI) or ID tokens (private Cloud Run services), usually through Application Default Credentials. Every Runpod Serverless request carries Authorization: Bearer YOUR_API_KEY, and Runpod checks the key before it routes the request. For traffic from browsers or anonymous users, put your own API or proxy in front so the key stays server-side.
GPU selection is by GPU type or pool, not machine type. A Pod takes one GPU type ID and a GPU count. A Serverless endpoint takes one or more GPU pools, and workers land on whichever pool has capacity. vCPU and RAM come with the GPU instead of being a separate machine-type choice. On Pods you can set a floor with min_ram_per_gpu and min_vcpu_count_per_gpu placement filters.
Storage is tied to a data center. A network volume lives in one data center, and anything that mounts it must run there. That narrows GPU choice the same way a zonal Persistent Disk does. A global volume removes the constraint for read-heavy data such as model weights.
Model artifacts. Vertex AI copies artifactUri to a managed location and exposes it as AIP_STORAGE_URI. On Runpod, bake weights into the image, load them from a network or global volume, or use cached models for Hugging Face models.
What has no direct equivalent
On Runpod:- VPC networks, firewall rules, and Private Service Connect. Runpod has no customer-defined private network or firewall rules. Pods can use global networking (a private network between your own Pods, at
POD_ID.runpod.internal, which you enable at deploy time on NVIDIA GPU Pods in supported data centers) and explicitly exposed ports. HTTP ports are served through a public HTTPS proxy URL that isn’t authenticated by Runpod, so add authentication in your app. If you need a private path into your Google Cloud VPC, run a VPN or proxy that you control, or keep that workload on Google Cloud. - IAM service accounts, Workload Identity, and the metadata server. A Pod or worker has no Google identity and no
metadata.google.internal. To call Google Cloud APIs from Runpod, create a least-privilege service account, store its key as a Runpod secret, and load it at startup. Where a caller only needs one object, hand it a signed URL instead of a key. - Customer-managed encryption keys (CMEK). You can encrypt a Pod’s volume disk at creation, with a key Runpod manages. Bring-your-own-key isn’t supported, and container disks and network volumes can’t be encrypted this way.
- Spot pricing through the CLI or REST API v2.
runpodctland REST API v2 create on-demand Pods. Checkpoint to a volume and plan capacity withrunpodctl gpu listrather than relying on a spot fallback. - Traffic splitting between model versions on one endpoint. Vertex AI splits traffic across the DeployedModels on an endpoint. On Runpod, run two endpoints and split in your client or gateway. Rolling out a new image to one endpoint is covered in rolling releases.
- Managed batch input and output. Vertex AI batch inference reads JSON Lines, CSV, or BigQuery and writes results to Cloud Storage or BigQuery. On Runpod you write the loop: read inputs, submit
/runjobs, and write results where you want them. - A Kubernetes API. There’s no control plane to
kubectl applyagainst. Use Serverless endpoints, Pods, or an orchestrator such as SkyPilot or dstack.
- A built-in job queue for inference. Runpod queue endpoints accept
/run, hold jobs while workers scale, retry failures, and call a webhook on completion. Vertex AI online inference and Cloud Run services are request and response only. - No data transfer fees. Runpod doesn’t charge for ingress or egress.
- GPU pools with fallback. A Serverless endpoint can list several GPU pools and place workers wherever stock exists.
runpodctl pod create --compliance restricts placement to data centers holding the certifications you name. Review Security and compliance against your own requirements before you move regulated data.
Requirements
- A Google Cloud project with the gcloud CLI authenticated and permission to list and delete the resources you’re migrating.
- A Runpod account with a payment method or credits. Deploying an on-demand Pod requires at least one hour of credits for that configuration.
- Docker, if you need to rebuild images.
- Terraform 1.6 or later, if you manage infrastructure as code.
Step 1: Take inventory of what you run on Google Cloud
List every GPU resource, export its configuration, and find the code that calls Google Cloud. Start with a project-wide search so nothing is missed, then drill into each product.--asset-types, for example compute.googleapis.com/Instance.
Next, export the details for each product you use. Each command prints what you need to recreate the workload: machine and accelerator type, image, environment, ports, disks, region, and scaling limits.
vm.json, read machineType, guestAccelerators, disks, scheduling, serviceAccounts, and the startup-script key under metadata. In endpoint.json, each entry under deployedModels has dedicatedResources with the machine type, accelerator, and minReplicaCount and maxReplicaCount. In service.yaml, read the container image, ports, env, resource limits (including nvidia.com/gpu), the run.googleapis.com/accelerator node selector, concurrency, timeout, and scaling annotations.
Then find the code and configuration that depends on Google Cloud.
Step 2: Set up Runpod
Create an API key on the Credentials page, then install the tools.rp login opens a browser to approve access. Tools also read the RUNPOD_API_KEY environment variable, which is the simplest option in CI.
If you’ll connect to Pods over SSH, add your public key on the Credentials page first. If you manage infrastructure as code, set up the Terraform provider. To let a coding agent manage Runpod resources, connect the MCP server.
Step 3: Choose where each workload lands
Pick a Runpod target for each workload from your inventory.
Check the Public Endpoints reference for the hosted models Runpod offers. Keep calls to models that aren’t listed there on Google Cloud, or move them to an open model you deploy yourself.
Map your GPUs
Use this table to translate Compute Engine accelerator types to Runpod GPU type IDs (used by Pods) and Serverless GPU pools. Choose by fit for the workload: VRAM first, then GPU generation.
Machine types that carry several GPUs, such as
a3-highgpu-8g or a2-ultragpu-4g, become one Pod with a GPU count (--gpu-count 8) or an Instant Cluster when you need several nodes.
GPU stock varies by data center and changes through the day. Before you commit to a type, check live stock with runpodctl gpu list and runpodctl datacenter list, or call GET /v2/catalog/gpus, where each GPU’s pool field gives its Serverless pool ID. The full list of IDs is in the GPU types reference.
Step 4: Move your images and code
Check image portability
Runpod runslinux/amd64 images. Cloud Run and Compute Engine GPU VMs are also x86-64, so most images run unchanged. If you build on an Arm machine, build for the right platform.
nvidia/cuda base image). Set a CUDA floor that matches your image when you create Pods or endpoints (--min-cuda-version 12.8), so the host driver is new enough.
If your VM used a Deep Learning VM image (for example the pytorch-2-9-cu129-ubuntu-2204-nvidia-580 family in the deeplearning-platform-release project), the nearest starting point is an official Runpod PyTorch image such as runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04. Install anything else your startup script added.
Pull private images from Artifact Registry
You can leave images in Artifact Registry and let Runpod pull them. Google documents two username and password methods for Docker: a service account key (username_json_key, or _json_key_base64 for a base64-encoded key) and an access token (username oauth2accesstoken). Access tokens last 60 minutes, so they don’t work for a stored credential. Use a service account key that can only read the repository.
1
Create a pull-only service account
Create a service account and grant it the Artifact Registry Reader role on the repository.
2
Create a key
Create a JSON key for the service account.Google describes service account keys as the least secure Docker authentication option because they’re long-lived. Scope this account to image pulls only, and rotate the key on a schedule.
3
Store the key as a Runpod registry credential
Pass the key file on standard input with the literal username The command prints the credential ID. The same credential can be created with
_json_key.POST /v2/registries or the runpod_registry Terraform resource.4
Reference the credential
Use the full image path and tag, and pass the credential ID when you create a Pod or template.
runpod-puller.json from your machine after you store it. If you’d rather not keep a Google key on Runpod, push the image to another registry or deploy from GitHub.
Translate a Vertex AI custom container
A Vertex AI custom container is an HTTP server that reads its port and routes fromAIP_* environment variables. This is a typical FastAPI server for Vertex AI.
main.py
AIP_HTTP_PORT, answers health checks on AIP_HEALTH_ROUTE, and serves the instances and parameters request shape on AIP_PREDICT_ROUTE.
You have two ways to run this on Runpod.
- Load-balancing endpoint (keep the server)
- Queue endpoint (handler)
Keep the FastAPI server. Read the port from The request and response bodies are unchanged, so callers only change the URL and the auth header. Loading the model in a background thread lets The image sets
PORT, add a /ping route that reports readiness, and use fixed paths.main.py
/ping return 204 during startup, which is how Runpod measures the cold start of a load-balancing worker.Package it with a Dockerfile.Dockerfile
PORT and PORT_HEALTH to the same value. Set both on the endpoint too and expose 8080/http, as shown in Step 6.AIP_STORAGE_URI, change it to read from a volume path (see Step 5) or download from Cloud Storage at startup with a service account key.
Translate a Cloud Run GPU service
A Cloud Run service already follows most of the load-balancing contract: it’s an HTTP server that listens on0.0.0.0 at the port in PORT. Make these changes:
- Add a health route. Serve
GET /ping, or setHEALTH_CHECK_PATHto a route you already have. Return204until the model is loaded. - Set the port explicitly. Set
PORTandPORT_HEALTHto your server’s port (Cloud Run’s default is8080) and expose it. - Replace Google identity.
K_SERVICE,K_REVISION, and the metadata server aren’t present. Code that fetches tokens frommetadata.google.internalor relies on the service’s attached service account needs a key from a Runpod secret. - Check request duration and size. Requests are limited to 5.5 minutes of processing and 30 MB. Move longer requests to a queue endpoint.
- Replace Cloud Storage volume mounts. A
cloud-storagevolume becomes a network or global volume at/runpod-volume, or explicit Cloud Storage client calls.
CLOUD_RUN_TASK_INDEX and CLOUD_RUN_TASK_COUNT to shard work. On Runpod, wrap the per-task work in a queue handler and submit one /run request per shard, passing the index in input.
Translate a Compute Engine startup script
A startup script runs as root on every boot of the VM. On a Pod, the equivalent is the container start command. Put one-time setup (package installs, driver-independent configuration) in the image, and put per-boot work in the start command. Official Runpod PyTorch images start SSH and JupyterLab from/start.sh. If you override the start command, call /start.sh at the end so those services still start.
Train with the Runpod SDK instead of a custom job
A Vertex AI custom job runs a container on a worker pool and exits. With the Runpod SDK,@app.task runs a Python function on an ephemeral Pod with the same lifecycle.
train.py
rp flash dev train.py. Checkpoints written to /data replace AIP_MODEL_DIR and AIP_CHECKPOINT_DIR, and /gcs/BUCKET paths become paths on the mounted volume. To keep your existing training container instead, run it as a Pod with your training command as the start command.
Step 5: Move your data and models
Decide first whether the data needs to move. In a hybrid setup, datasets and artifacts can stay in Cloud Storage and your Runpod workloads read them over the public API with a service account key or HMAC key stored as a Runpod secret. Move data when startup time matters (model weights) or when the workload reads it constantly (training data).
Create the volume in the data center where you’ll run compute. Check GPU stock there first, because everything that mounts a network volume runs in that data center.
- From inside a Pod
- Cloud Sync
- From your machine
Start a Pod with the volume attached, install the gcloud CLI in it, and pull straight from the bucket. This is the most direct path for large datasets because the data goes from Cloud Storage to the volume without passing through your machine.
gcloud storage rsync copies only what’s missing or changed, so you can rerun it after an interruption. Add --dry-run to preview. If you prefer rclone, rclone copy :gcs:BUCKET/models /workspace/models --gcs-service-account-file /tmp/key.json does the same job.https://s3api-DATACENTER.runpod.io/ with the volume ID as the bucket. Drive it with S3 clients such as the AWS CLI or Boto3. It isn’t a Cloud Storage API, so gcloud storage and the Cloud Storage client libraries can’t target it.
To fill a global volume, attach it to a Pod and copy into its mount path the same way. Global volumes have no S3-compatible API.
Keep reading from Cloud Storage
For data that stays in Cloud Storage, give the workload a key that can only read what it needs.- Create a service account with
roles/storage.objectVieweron the specific bucket. - Create a JSON key, or an HMAC key if your code uses an S3-style client against
https://storage.googleapis.com. - Store the key as a Runpod secret, for example
gcp_sa_key. - Map it to an environment variable on the Pod or endpoint:
GCP_SA_KEY={{ RUNPOD_SECRET_gcp_sa_key }}.
Step 6: Deploy on Runpod
Pod
This replaces a Compute Engine GPU VM. Here’s a typical VM with one L4, a Deep Learning VM image, and a startup script./workspace, so startup.sh must already be on it (copy it there in Step 5) or be baked into the image. The data center must be the one that holds the volume. In the console, the same flow is Pods > Deploy.
Wait for the Pod to be ready, then connect.
https://POD_ID-8888.proxy.runpod.net. The official PyTorch images start JupyterLab on port 8888 only when JUPYTER_PASSWORD is set, so add it to --env if you want JupyterLab on a Pod created with runpodctl. To create the same Pod over REST, send POST https://api.runpod.io/v2/pods with name, image, gpu (id, count, minCudaVersion), disk, ports, env, args, dataCenterIds, and mounts.network. See Create a Pod.
Load-balancing endpoint
This replaces a Cloud Run GPU service or a Vertex AI endpoint that serves short requests. Here’s the Cloud Run deployment you’re replacing.gcloud run deploy flags is direct: --gpu-type nvidia-l4 becomes the AMPERE_24 pool, --port becomes ports plus PORT and PORT_HEALTH, --concurrency becomes scaling.requestCount, and --min-instances and --max-instances become workers.min and workers.max. type and scaling are required, and a load-balancing endpoint only accepts REQUEST_COUNT scaling. There’s no --cpu or --memory equivalent because they come with the GPU.
The response includes the endpoint ID and requestUrls.base, which is https://ENDPOINT_ID.api.runpod.ai. Send a request once a worker is ready.
no workers available, retry after a few seconds, or poll /health until a worker reports ready.
Queue endpoint
This replaces Vertex AI endpoints with long-running requests, batch inference, Cloud Run jobs, and Batch jobs. Create a Serverless template for the handler image from Step 4, then an endpoint that uses it.--network-volume-id. It mounts at /runpod-volume on each worker.
Submit a job and read the result.
/run with a webhook URL and let Runpod call you when each job completes. See Send API requests.
Public Endpoint or vLLM endpoint
To replace calls to a Model Garden open model, check whether the model is in the Public Endpoints reference. If it is, there’s nothing to deploy. If it isn’t, deploy it on a vLLM endpoint from the Hub.https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1. See Get started with vLLM for gated models and tuning.
Instant Cluster
To replace a Slurm cluster built with Cluster Toolkit, create a Slurm cluster from the Instant Clusters page with your GPU type, node count, region, and an optional network volume in the same region. Slurm and munge are preconfigured, andsbatch, squeue, and sinfo work on the controller node. Managed Slurm clusters use official Runpod PyTorch images. Availability depends on the GPU type and region, so confirm it in the console before you schedule a cutover. See Slurm Clusters.
Step 7: Update the code that calls Google Cloud
Vertex AI online inference
Calls through the Vertex AI SDK authenticate with Google credentials and sendinstances.
gcloud ai endpoints predict and to REST calls against https://LOCATION_ID-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/endpoints/ENDPOINT_ID:predict.
On Runpod, the call is an HTTPS request with a Bearer API key.
- Load-balancing endpoint
- Queue endpoint
The body is unchanged because your server defines it.Add a retry for the first request after scale to zero, when no worker is ready yet.
Long-running and batch requests
Vertex AI batch inference takes aBatchPredictionJob with a Cloud Storage or BigQuery source. On Runpod, submit jobs to /run and collect results by polling or webhook.
IN_QUEUE, IN_PROGRESS, COMPLETED, FAILED, CANCELLED, and TIMED_OUT. Results from /run are kept for 30 minutes after completion, so write large outputs to Cloud Storage or a volume from inside the handler and return a reference.
Private Cloud Run services
A caller of a private Cloud Run service fetches an ID token for the service URL.https://ENDPOINT_ID.api.runpod.ai. Callers that still run on Google Cloud should read the Runpod API key from Secret Manager.
OpenAI-compatible calls
If you call open models through Google’s OpenAI-compatible endpoint, the client authenticates with a short-lived Google access token.https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1 and the model name the endpoint serves.
Control-plane automation
Replacegcloud calls and Google API client calls in scripts and CI with runpodctl, REST API v2, or the Runpod SDK. The CLI command map and API map list the equivalents. In CI, set RUNPOD_API_KEY as a secret in place of a service account key or Workload Identity Federation configuration.
Step 8: Verify, cut over, and shut down
1
Send a real request from outside
Call the Runpod endpoint from the same network location as your production callers, using production-shaped inputs. For Pods, connect over SSH and run the workload.
2
Compare outputs and latency
Send the same inputs to both platforms and compare the outputs. Measure cold-start and warm latency separately. On Runpod, the endpoint’s Metrics tab shows execution time, delay time, and cold-start percentiles. Tune the idle timeout, active workers, and FlashBoot before you shift traffic.
3
Run both in parallel
Keep the Google Cloud deployment serving while you route a small share of traffic to Runpod from your client or gateway. Watch error rates and logs with
runpodctl serverless logs ENDPOINT_ID.4
Shift traffic and set a cost guard
Move the rest of the traffic. Set
workers.max to the highest concurrency you intend to pay for, keep active workers at zero unless you need them, and enable low balance notifications. Runpod accounts also have a default hourly spend limit, described in Account spend limits.5
Shut down what you replaced on Google Cloud
Stopping a VM or scaling a service down doesn’t stop every charge. Work through the checklist below.
Google Cloud shutdown checklist
Delete only what you’ve replaced, and take a final backup of anything you might need.- Stopped VMs still bill for disks and IP addresses. Compute Engine bills attached disks and external IP addresses in every instance state, including
TERMINATED. Suspended VMs also bill for memory. - Disks can survive VM deletion. A disk that isn’t set to auto-delete stays after the VM is gone.
- A Vertex AI endpoint can’t be deleted until every model is undeployed from it. Uploaded models in Model Registry and their artifacts in Cloud Storage remain until you delete them.
- Cloud Run minimum instances bill while idle. Deleting the service removes them.
- Managed instance groups recreate VMs. Delete the group, not the instances.
- GKE node pools. Delete GPU node pools or the cluster if nothing else uses them.
- Reservations and committed use discounts. Reservations bill until deleted, and commitments run to the end of their term. Review them with your account team.
- Cloud Storage. Keep buckets your Runpod workloads still read, and delete the rest.
Migrate your Terraform
The Runpod Terraform provider 2.x (runpod/runpod) is built on REST API v2. It runs alongside hashicorp/google in one configuration, which fits a hybrid setup: Google Cloud keeps buckets, service accounts, and Artifact Registry, and Runpod runs the GPU compute.
Resource map
Run both providers side by side
Declare both providers. During the transition, the Google provider keeps managing the resources you haven’t moved, and it can create the pull-only service account that Runpod uses for Artifact Registry.providers.tf
registry.tf
google_service_account_key.private_key is the JSON key, base64 encoded, which is the format Artifact Registry expects with the _json_key_base64 username. The key is stored in Terraform state, so use an encrypted remote backend with restricted access. runpod_registry has no update operation, so rotating the key replaces the credential.
Compute Engine GPU instance to a Pod
This is a GPU VM with a data disk and a startup script.before.tf
guest_accelerator block with type and count. A G2 machine type carries its L4 GPU implicitly.
The Runpod version replaces the VM and data disk with a Pod and a network volume.
after.tf
machine_type became gpu_type_id and gpu_count, the boot image became a container image, google_compute_disk became runpod_network_volume, the attached service account became a key read from a Runpod secret, and metadata_startup_script became args. There’s no network interface or scheduling block. start_ssh needs an SSH key registered on your account. Changing gpu_type_id, gpu_count, cloud, min_cuda_version, or data_center_ids replaces the Pod, so keep anything you can’t recreate on the network volume. To stop the Pod without removing it from the configuration, set desired_state = "EXITED".
Cloud Run GPU service to a Serverless endpoint
This is a Cloud Run service with one L4 GPU.before.tf
after.tf
node_selector.accelerator became gpu_pools, max_instance_request_concurrency became scaling_value with scaling_type = "REQUEST_COUNT", scaling.max_instance_count became workers_max, ports.container_port became ports plus the PORT and PORT_HEALTH variables, and timeout is in milliseconds. endpoint_type can’t change after creation. For a queue endpoint, set endpoint_type = "QUEUE" and scaling_type = "QUEUE_DELAY", and drop ports.
Cut over in Terraform
- Add the Runpod resources and run
terraform apply. The Google resources are untouched. - Verify the Runpod endpoint or Pod, then update the outputs or variables your callers read (for example, switch a service URL output to
runpod_serverless_endpoint.embedder.request_urls.base). - Remove the Google resource blocks you replaced and apply again. If a resource has
deletion_protectionenabled, set it tofalseand apply before removing the block. - Keep the Google resources Runpod depends on: the service account, its key, the Artifact Registry repository, and any buckets.
terraform import with the Pod, endpoint, template, or volume ID. Be careful with Pods: the API doesn’t return template_id, start_ssh, start_jupyter, allowed_cuda_versions, min_cuda_version, min_ram_per_gpu, min_vcpu_count_per_gpu, or data_center_ids, so each one your configuration sets forces a replacement on the first apply after an import, which deletes the Pod and everything that isn’t on a network volume. List them under lifecycle { ignore_changes = [...] } before you apply. See the provider reference for every argument.
Reference
CLI command map
API map
Configuration field map
Environment variable map
Status map
Migrate with a coding agent
A coding agent can run this migration with the Runpod skills and MCP server.1
Install the Runpod skills and MCP server
- Claude Code
- Other agents
2
Give the agent this prompt
Run the agent in the repository that holds your application code and Terraform, with
gcloud authenticated.Troubleshooting
The image pull fails with an authentication error. The registry credential is wrong or the service account can’t read the repository. With--username "_json_key", the password must be the raw JSON key file. With _json_key_base64, it must be the whole file base64 encoded on one line. An oauth2accesstoken credential stops working after 60 minutes. Confirm the service account has roles/artifactregistry.reader on the repository and that the image path includes a tag.
A load-balancing worker shows as running but never becomes ready. The health check isn’t reaching your server. Vertex AI and Cloud Run images usually listen on 8080. Set PORT and PORT_HEALTH to that port, expose it in ports, and make sure GET /ping (or your HEALTH_CHECK_PATH) returns 200. Misconfigured workers return 502 and are stopped after 8 minutes.
Requests return {"error":"not allowed for QB API"}. You’re calling a path on https://ENDPOINT_ID.api.runpod.ai for an endpoint that was created as queue-based. The endpoint type is fixed at creation. Recreate it with "type": "LOAD_BALANCER".
Requests return no workers available. The endpoint scaled to zero and no worker was ready in time. Retry with a delay, raise the idle timeout, or set one active worker for latency-sensitive paths.
Google client libraries fail with a default credentials error. There’s no metadata server on Runpod, so Application Default Credentials finds nothing. Load a service account key from a Runpod secret, as shown in Step 5, or write the key to a file and set GOOGLE_APPLICATION_CREDENTIALS to its path at startup.
The container exits with a CUDA or driver error. The host’s driver is older than your image’s CUDA version. Set a CUDA floor (--min-cuda-version, gpu.minCudaVersion, or min_cuda_version) that matches the image. Cloud Run GPU instances use a recent driver, so images built against it may assume a newer CUDA than you had pinned elsewhere.
The handler receives instances as missing. Queue endpoints deliver the request body’s input object to the handler. Wrap the Vertex AI body: {"input": {"instances": [...], "parameters": {...}}}.
Long requests are cut off on a load-balancing endpoint. Processing is limited to 5.5 minutes per request. Move the workload to a queue endpoint and call /run.
The Pod creates in the wrong data center or fails with a capacity error. A network volume pins the Pod to its data center, and the GPU you asked for may not be in stock there. Check runpodctl datacenter list, pick a GPU type the data center has, or queue the Pod create through REST API v2.
Paths under /gcs/ or a Cloud Storage FUSE mount don’t exist. Vertex AI training mounts buckets under /gcs/. On Runpod, copy the data to a volume and update the paths, or read from Cloud Storage with the client library.
Next steps
Load-balancing endpoints
The full worker contract, limits, and health check behavior.
Terraform provider
Install the provider and manage Runpod as code.
Network volumes
Persistent storage for Pods and Serverless workers.
Runpod SDK
Define queue endpoints, tasks, and APIs in Python.