“The request was aborted because there was no available instance” means a Cloud Run request waited for an instance and none came in time. The fix is three numbers per service: concurrency, maximum instances and minimum instances. Startup time decides how much slack the other three give you.
What breaks at peak
Cloud Run logs the same message with two severities, and each points to a different cause:
The request was aborted because there was no available instance.
severity=WARNING, HTTP 429. The service hit its maximum instance count. Google’s description is that “Cloud Run cannot scale due to themax-instanceslimit you set during configuration.” The default maximum is 100 instances per revision.severity=ERROR, HTTP 500. Cloud Run couldn’t start instances fast enough for the traffic. Google lists a sudden traffic increase, long cold starts and slow or slowing requests as causes. A high error rate in your code is another.
The second one is the peak-traffic failure. A request that finds no free instance waits in a pending queue. Google gives the wait as up to “3.5 times average startup time of container instances of this service, or 10 seconds, whichever is greater.” If no instance is ready by then, Cloud Run drops it with the message above.
Three settings feed that race:
- Startup time. A new instance pulls the image, starts the container and runs your init code before it serves. Every second here is a second a queued request waits.
- Concurrency. Each instance takes up to this many requests at once. The gcloud CLI and Terraform give new services 80 per vCPU, and the console gives 80. Set it above what your code handles, and requests queue inside the container where Cloud Run can’t see them.
- The autoscaler’s targets. By default Cloud Run scales to keep CPU utilization and request concurrency at 60% of their limits. Scale-up starts once utilization leaves a 10% band around that target. A spike that arrives faster than instances start outruns it.
Traffic splits make it worse. Each revision that takes traffic gets its share of the maximum. A canary at 10% can hit its own limit and return 429 while the service as a whole has room.
The maximum also protects what sits behind the service. Each instance opens its own database pool. One hundred instances with a pool of ten is a thousand connections, and Postgres refuses logins long before that. The PgBouncer recipe covers the fix on that side.
Cloud Run cold start and concurrency config
Save the current service first:
gcloud run services describe api --region=europe-west1 --format=export > api-before.yaml
Then set all four values in one update:
gcloud run services update api --region=europe-west1 \
--cpu=2 --memory=1Gi \
--concurrency=40 \
--min=2 --max=60 \
--cpu-boost
--concurrency caps simultaneous requests per instance. I set it from a load test of one instance. Take the highest concurrency at which that instance still meets your p99 target, then go a little lower. Match it to the server inside the container:
- Node.js runs your handlers on one thread. For CPU-heavy handlers, give each instance 1 vCPU and keep concurrency modest. For I/O-heavy handlers, a high value works.
- Gunicorn serves
workers × threadsrequests at once. Set concurrency to that product, or Cloud Run sends requests that queue behind busy workers. - On the JVM, the servlet thread pool or the virtual thread executor sets the ceiling. Set concurrency at or below
server.tomcat.threads.max.
A concurrency of 1 suits code that can’t run two requests at once. Google warns it hurts scaling, because every concurrent request needs its own instance.
--max caps instances for the whole service, split across the revisions that take traffic. --max-instances sets it per revision instead. Size it from two directions and take the lower number:
- Peak demand. Peak concurrent requests equal peak requests per second times average latency at peak. Divide by concurrency, then by 0.6 for the autoscaler’s target. Add headroom for the error in your latency estimate.
- Downstream limits. Maximum instances times the pool size per instance must fit your database connection budget, along with every other client of that database.
I start from the downstream limit. If peak demand needs more instances than the database allows, the fix is a pooler or fewer connections per instance. A higher maximum that pushes the database over trades 429s for 500s.
--min keeps that many instances warm for the service, split across revisions like --max. --min-instances is the per-revision form. Warm instances serve the first seconds of a spike while new ones start. Set it to cover your normal baseline. Before a known peak, such as a launch or a sale, raise it ahead of time:
gcloud run services update api --region=europe-west1 --min=20
A Cloud Scheduler job can run that update before each daily peak and lower it after. Minimum instances bill while idle, so size them against the cost of failed requests.
--cpu-boost gives instances extra CPU while they start. New services get it by default. Older services may not have it, so pass the flag anyway.
Billing mode. With request-based billing, an instance loses its CPU between requests. Background work, such as flushing metrics or refreshing a cache, stalls until the next request. Instance-based billing (--no-cpu-throttling) keeps the CPU on. Google recommends a minimum of at least 1 for that background work to keep running.
Startup time. Shorter startup widens every margin above. Load config and open connections on first use, and keep global init to what every request needs. Ship a smaller image with fewer layers. Add a startup probe on a cheap endpoint, so Cloud Run sends traffic once the app is ready and not before.
Clients. Google’s own advice for these errors is exponential backoff and retries for requests the client can’t drop. The timeouts and retries recipe shows how to retry without turning a scaling lag into a retry storm. On GKE, the same scale-up race lives in the HPA, covered in HPA for peak traffic.
Test it on your system
Deploy the new settings to a staging service with the same image, CPU, memory and database tier as production. Test two shapes: the steady peak and the spike that arrives cold.
The spike matters most. Start from idle, or from your normal minimum, and jump to peak in seconds. This k6 script holds a fixed arrival rate, so slow responses don’t hide the failures:
import http from 'k6/http';
export const options = {
scenarios: {
spike: {
executor: 'ramping-arrival-rate',
startRate: 5, timeUnit: '1s',
preAllocatedVUs: 500, maxVUs: 4000,
stages: [
{ target: 5, duration: '2m' }, // quiet, instances scale in
{ target: 600, duration: '20s' }, // the spike, set to your peak rate
{ target: 600, duration: '10m' },
{ target: 5, duration: '1m' },
],
},
},
thresholds: {
http_req_failed: ['rate<0.001'],
http_req_duration: ['p(99)<800'],
},
};
export default function () {
http.get('https://api-staging-abc123-ew.a.run.app/v1/products');
}
Set the rate to your own peak and the thresholds to your own targets. Run it before and after the change. Then run it once with --min at zero, to see how bad a cold spike gets.
While it runs, watch three things:
gcloud logging read \
'resource.type="cloud_run_revision" AND resource.labels.service_name="api-staging" AND "no available instance"' \
--freshness=30m --format='value(timestamp,severity,httpRequest.status)' | sort | uniq -c | tail
- The log count above, split by severity.
- In Cloud Monitoring, Container instance count against your maximum, and Container startup latency.
- Your database connection count, which should stay inside its budget at the peak instance count.
Write your pass criteria down first. The change passes when no “no available instance” lines appear during the spike or the plateau. The p99 should meet your target from the first minute, and the instance count should stay below the maximum with headroom. If the 500 version still appears with instances well under the maximum, startup time is your bottleneck. Raise the minimum or shorten startup.
Keep it healthy
Cloud Run metrics live in Cloud Monitoring. Run stackdriver_exporter to read them into Prometheus, or use Managed Service for Prometheus, which queries them with PromQL under its own names. I wrote the rules below with stackdriver_exporter’s metric names.
Google’s request count metric excludes requests rejected at the maximum instance count. Count the abort message from the logs instead, with a log-based metric. Save this as run-no-instance.yaml:
name: run_no_instance
description: Cloud Run requests aborted with no available instance
filter: resource.type="cloud_run_revision" AND "The request was aborted because there was no available instance"
metricDescriptor:
metricKind: DELTA
valueType: INT64
labels:
- key: severity
valueType: STRING
labelExtractors:
severity: EXTRACT(severity)
gcloud logging metrics create run_no_instance --config-from-file=run-no-instance.yaml
stackdriver_exporter \
--google.project-ids=my-project \
--monitoring.metrics-prefixes=run.googleapis.com/container/instance_count \
--monitoring.metrics-prefixes=run.googleapis.com/container/startup_latencies \
--monitoring.metrics-prefixes=logging.googleapis.com/user/run_no_instance \
--monitoring.aggregate-deltas
The exporter adds the cloud_run_revision labels, such as service_name and revision_name, to each series. Cloud Monitoring publishes the data a few minutes late, so the alerts lag the first error by that much.
groups:
- name: cloud-run-scaling
rules:
- alert: CloudRunNoAvailableInstance
expr: |
sum by (project_id, service_name, severity) (
increase(stackdriver_cloud_run_revision_logging_googleapis_com_user_run_no_instance[10m])
) > 0
for: 5m
labels:
severity: critical
annotations:
summary: "Cloud Run {{ $labels.service_name }} is dropping requests with no available instance ({{ $labels.severity }})"
description: "WARNING means the max instance count was hit: raise --max if the database allows. ERROR means scale-up lagged: raise --min and check startup latency."
- alert: CloudRunInstancesNearMax
# 48 is 80% of --max=60; set it from your own maximum
expr: |
sum by (project_id, service_name) (stackdriver_cloud_run_revision_run_googleapis_com_container_instance_count) > 48
for: 10m
labels:
severity: warning
annotations:
summary: "Cloud Run {{ $labels.service_name }} is above 80% of its max instances"
description: "The next spike will return 429s. Check the database connection budget, then raise --max, or lower latency so each instance serves more."
- alert: CloudRunSlowStartup
# 5000 ms is an example; set it from your own startup baseline
expr: |
histogram_quantile(0.95,
sum by (project_id, service_name, le) (
rate(stackdriver_cloud_run_revision_run_googleapis_com_container_startup_latencies_bucket[30m])
)
) > 5000
for: 30m
labels:
severity: warning
annotations:
summary: "Cloud Run {{ $labels.service_name }} instances take over 5 s to start"
description: "Slow starts shrink the pending window during a spike. Check recent image or init changes, turn on --cpu-boost, or raise --min."
The instance count includes idle instances. To track busy ones, filter on state="active". Startup latency is in milliseconds.
Roll back
You saved the service to api-before.yaml. To restore the old settings, replace the service with it. Cloud Run deploys a new revision with the old values:
gcloud run services replace api-before.yaml --region=europe-west1
To revert one value, set it back with gcloud run services update, for example --concurrency=80. Service-level --min and --max apply without a new revision. --concurrency and the CPU settings deploy one.
If the new concurrency pushes p99 past your target or raises the error rate, roll back now. That means each instance gets more requests than it can serve. If you’re lowering minimum instances to save cost, do it after the day’s peak, so the next spike still finds warm instances.
More recipes
- Cloud NAT Port Exhaustion Alerts in Prometheus
Alert on Cloud NAT port usage per VM and on OUT_OF_RESOURCES dropped packets, so you see port exhaustion before outbound calls start timing out.
- Fix GCP Load Balancer 502s and backend_timeout
Size the backend service timeout to your slowest real request and hold backend keepalive at 620 seconds, so the load balancer stops answering 502.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.