es_rejected_execution_exception means a node’s thread pool queue was full, and circuit_breaking_exception means a request would have pushed the heap past a limit. Both come back as HTTP 429, and both are Elasticsearch protecting itself. You fix them by changing the load your client sends and the indices it lands on. Raising a queue or a breaker limit moves the failure somewhere worse.
What breaks at peak
Thread pools fill. Each node runs fixed-size thread pools, such as write for indexing and search for queries, and each pool has a bounded queue. Once the queue is full, the node rejects the next task. The reason names the pool and its queue capacity:
"type": "es_rejected_execution_exception",
"reason": "rejected execution of ... on EsThreadPoolExecutor[name = es-data-1/write, queue capacity = 10000, ...]"
Indexing pressure trips. Since 7.9, nodes also cap the memory held by in-flight indexing requests, through indexing_pressure.memory.limit, 10% of the heap by default. Past it, writes fail with the same type. The reason starts rejected execution of coordinating operation [...], or names a primary or replica operation.
Breakers trip. Circuit breakers estimate the memory a request needs and refuse it before the node runs out of heap. The parent breaker checks real heap use against indices.breaker.total.limit, 95% of the heap by default. Child breakers cap request structures such as aggregations at 60%, fielddata at 40% and in-flight request bodies at 100%. The reason shows which one tripped:
"type": "circuit_breaking_exception",
"reason": "[parent] Data too large, data for [<http_request>] would be [...], which is larger than the limit of [...], real usage: [...], new bytes reserved: [...], usages [...]",
"durability": "TRANSIENT"
A [fielddata] or [request] prefix names a child breaker instead. durability tells you whether a retry can help. TRANSIENT means other requests hold the memory and a retry may pass. PERMANENT means this request is too big on its own, so retrying it changes nothing.
Bulk failures hide. A _bulk request can return HTTP 200 with "errors": true, and a 429 on some of the items inside. A client that checks only the HTTP status drops those documents without a log line. At peak, that’s where data goes missing.
Users feel it two ways. Search rejections and breaker trips return errors to your users at once. Write rejections show up later, as missing or stale search results.
The fix for rejected execution and Data too large
Leave the queues alone. I keep thread_pool.write.queue_size and thread_pool.search.queue_size at their defaults. A bigger queue holds more requests in heap. It turns fast 429s into slow timeouts and moves the failure to the breakers. Recent 9.x releases already scale the write queue with the processor count.
Retry rejected items with backoff. Your client must read each item’s status in a bulk response and retry the 429s, with exponential backoff. The Python helper does this, but its max_retries defaults to 0:
import logging
import os
from elasticsearch import Elasticsearch
from elasticsearch.helpers import streaming_bulk
log = logging.getLogger("indexer")
es = Elasticsearch(
"https://es.internal:9200",
api_key=os.environ["ES_API_KEY"],
request_timeout=60,
max_retries=3,
retry_on_status=(502, 503, 504),
retry_on_timeout=True,
)
def actions(docs):
for doc in docs:
# A fixed _id makes a retried item overwrite itself instead of duplicating.
yield {"_index": "events", "_id": doc["id"], "_source": doc}
def index_all(docs):
failed = 0
for ok, item in streaming_bulk(
es,
actions(docs),
chunk_size=1000,
max_chunk_bytes=10 * 1024 * 1024,
max_retries=8,
initial_backoff=1,
max_backoff=60,
raise_on_error=False,
):
if not ok:
failed += 1
log.error("bulk item failed after retries: %s", item)
return failed
max_retries,initial_backoffandmax_backoffretry items that came back 429, doubling the wait each time. Without them, the helper reports rejected items and moves on.- A fixed
_idper document makes retries safe. With auto-generated IDs, a retry after a timeout can index the same document twice. max_chunk_bytescaps each bulk body by size. Document counts lie when document sizes vary, and a 100 MB body hits the in-flight breaker by itself.raise_on_error=Falselets you count and log failures instead of stopping on the first one.
The helper’s backoff has no jitter. If you run many indexers, start them at staggered times or add a random sleep between chunks, so their retries don’t land together. The timeouts and retries recipe covers jitter. Clients in other languages have the same pieces: per-item status checks, a retry limit and backoff.
Size bulk load to the cluster. Rejections depend on concurrent bulk requests across all your indexers, more than on the size of one. Size the total from the write threads: the write pool has one thread per allocated processor. Keep in-flight bulk requests per data node near that thread count, and let the load test move it. Start bulk bodies in the low megabytes and grow them while throughput still rises.
Spread shards across nodes. Read _cat/thread_pool per node. If one node rejects while the rest idle, it holds too many primaries of a hot index. Give that index enough primary shards to spread across the data nodes, and use index.routing.allocation.total_shards_per_node to stop them stacking. Each replica repeats every write on another node, so drop replicas to 0 during a one-off bulk load and restore them after.
Make searches cheaper. Search rejections at peak tend to come from fan-out. A query across 30 daily indices with 5 shards each queues 150 shard tasks. Cut the shard count with fewer, larger indices, and use filters on time fields so the pre-filter phase skips shards. max_concurrent_shard_requests on the search request limits how many shard tasks one request runs per node at a time.
Keep breakers from tripping. Each fix targets one breaker:
- Parent. Give the heap room. Recent versions size the heap from the node’s memory by default, so set
-Xmsand-Xmxonly if you have to. If you do, set them equal, at no more than half the container’s memory and under the compressed-oops cutoff near 32 GB. The JVM heap in containers recipe explains that cutoff. - Request. Cap aggregation size. A
termsaggregation with a hugesize, or nested aggregations on high-cardinality fields, build big structures in heap.search.max_bucketslimits bucket count per search. - Fielddata. Don’t sort or aggregate on
textfields. Map those fields askeywordor add akeywordsub-field. Fielddata ontextloads the whole field into heap. - In-flight requests. Cap bulk body size, as above.
Don’t raise breaker limits to make the errors stop. The breaker is the last thing between a heavy request and an OutOfMemoryError that takes the node down.
Managed Elasticsearch and OpenSearch. Elastic Cloud Hosted accepts only the user settings on its allowlist, so check whether your version lists thread pool settings. Either way, the client and index changes above apply, and a larger deployment adds threads and heap. Amazon OpenSearch Service doesn’t let you change thread pools. It publishes ThreadpoolWriteRejected, ThreadpoolSearchRejected and JVMMemoryPressure in CloudWatch. OpenSearch drops the es_ prefix, so search your logs for rejected_execution_exception. The executor name in its reason reads OpenSearchThreadPoolExecutor.
Test it on your system
Use a staging cluster with production’s node count, heap, shard layout and mappings. Index into a copy of a real index, with documents shaped like yours.
k6 can send bulk requests at a fixed rate, which holds the pressure steady while the cluster slows:
import http from "k6/http";
import { check } from "k6";
import { Counter } from "k6/metrics";
const ES = __ENV.ES_URL;
const AUTH = { headers: { "Content-Type": "application/x-ndjson", Authorization: `ApiKey ${__ENV.ES_API_KEY}` } };
const DOCS_PER_BULK = 1000;
export const options = {
scenarios: {
bulk: {
executor: "constant-arrival-rate",
rate: 40,
timeUnit: "1s",
duration: "20m",
preAllocatedVUs: 50,
maxVUs: 400,
},
},
thresholds: {
http_req_failed: ["rate<0.001"],
http_req_duration: ["p(99)<2000"],
bulk_item_429: ["count<1"],
},
};
const rejected = new Counter("bulk_item_429");
const randomString = (n) => Math.random().toString(36).slice(2).padEnd(n, "x").slice(0, n);
export default function () {
let body = "";
for (let i = 0; i < DOCS_PER_BULK; i++) {
body += JSON.stringify({ index: { _index: "events-loadtest" } }) + "\n";
body += JSON.stringify({ ts: new Date().toISOString(), user: randomString(8), msg: randomString(200) }) + "\n";
}
const res = http.post(`${ES}/_bulk`, body, AUTH);
check(res, { "status 200": (r) => r.status === 200 });
if (res.status === 200 && res.json("errors")) {
rejected.add(res.json("items").filter((it) => it.index.status === 429).length);
}
}
Set rate × DOCS_PER_BULK to your peak documents per second. The script counts item-level 429s in bulk_item_429, which http_req_failed would miss. Run your peak search traffic at the same time, through your app or a second scenario, since searches and writes share heap.
While it runs, poll the nodes:
watch -n 5 'curl -s -H "Authorization: ApiKey $ES_API_KEY" \
"$ES_URL/_cat/thread_pool/write,search?v&h=node_name,name,active,queue,rejected,completed"'
curl -s -H "Authorization: ApiKey $ES_API_KEY" \
"$ES_URL/_nodes/stats/breaker,indexing_pressure?filter_path=nodes.*.name,nodes.*.breakers.*.tripped,nodes.*.indexing_pressure.memory.total"
Write down your pass criteria first. A good set:
rejectedstays flat for both pools, or rises only while your client’s retries still succeed.- No breaker’s
trippedcount moves. - The script’s thresholds pass, including
bulk_item_429. _counton the index matches the documents your real indexer sent, so nothing went missing.- Heap after old-generation GC stays under your chosen ceiling, such as 75%.
Then raise rate in steps until rejections start, and note the rate. That’s your write ceiling for this layout.
Keep it healthy
Scrape every node with elasticsearch_exporter, started with --es.all so one exporter reports node stats for the whole cluster.
groups:
- name: elasticsearch-rejections
rules:
- alert: ElasticsearchSearchRejections
expr: increase(elasticsearch_thread_pool_rejected_count{type="search"}[5m]) > 0
for: 5m
labels:
severity: critical
annotations:
summary: "Node {{ $labels.name }} is rejecting searches"
description: "Users get 429 errors on search. Run _cat/thread_pool/search?v to see if one node takes the load, then cut shard fan-out or add data nodes."
- alert: ElasticsearchWriteRejections
expr: increase(elasticsearch_thread_pool_rejected_count{type="write"}[5m]) > 0
for: 10m
labels:
severity: warning
annotations:
summary: "Node {{ $labels.name }} is rejecting writes"
description: "Check that indexers retry 429 items with backoff, lower bulk concurrency, and check _cat/shards for hot primaries stacked on this node."
- alert: ElasticsearchParentBreakerTripped
expr: increase(elasticsearch_breakers_tripped{breaker="parent"}[5m]) > 0
for: 1m
labels:
severity: critical
annotations:
summary: "Parent circuit breaker tripped on {{ $labels.name }}"
description: "The heap is near its limit and requests fail with Data too large. Check heap use and GC, find heavy aggregations in the slow log, and cap bulk body size."
- alert: ElasticsearchChildBreakerTripped
expr: increase(elasticsearch_breakers_tripped{breaker!="parent"}[10m]) > 0
for: 1m
labels:
severity: warning
annotations:
summary: "The {{ $labels.breaker }} breaker tripped on {{ $labels.name }}"
description: "For fielddata, find aggregations or sorts on text fields. For request, cap aggregation size. For inflight_requests, cap bulk body size."
- alert: ElasticsearchHeapHigh
expr: |
elasticsearch_jvm_memory_used_bytes{area="heap"}
/ elasticsearch_jvm_memory_max_bytes{area="heap"} > 0.85
for: 15m
labels:
severity: warning
annotations:
summary: "Heap on {{ $labels.name }} above 85% for 15 minutes"
description: "The parent breaker trips near 95%. Look for fielddata use, large aggregations and too many shards per node before you add heap."
Heap swings with garbage collection, so a 15-minute for filters out the normal sawtooth. Also log item-level failures in every indexer, as in the Python example, and count them. The exporter can’t see documents your client gave up on.
Roll back
Save cluster settings before you change any:
curl -s -H "Authorization: ApiKey $ES_API_KEY" \
"$ES_URL/_cluster/settings?include_defaults=false&flat_settings=true" > es-settings-before.json
Dynamic settings such as search.max_buckets revert with a PUT _cluster/settings that sets them to null. Index settings such as replicas revert with PUT <index>/_settings. Restore replicas the moment a bulk load ends, since an index with no replicas loses data when a node fails. Client changes, such as bulk size, concurrency and retries, roll back with a deploy.
If a change raises rejections or breaker trips, roll it back now. A shard-count change needs a reindex or a rollover, so plan that for a quiet window.
More recipes
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.
- Fix Lambda TooManyRequestsException Rate Exceeded
Size Lambda concurrency from peak traffic and split it with reserved and provisioned concurrency, so critical functions stop throttling when traffic spikes.
- NAT Gateway ErrorPortAllocation and Port Exhaustion
Alert on ErrorPortAllocation and the signals that come before it, so you catch NAT gateway port exhaustion before outbound calls fail at peak.