The MongoDB WiredTiger cache size and the driver’s connection pool are two settings that ship with defaults built for a dedicated host. Run mongod in a container and scale your app to many pods, and both defaults turn against you at peak. You size each one from your own limits below, then load test the result.
What breaks at peak
The cache outgrows the container. WiredTiger’s default cache is 50% of (RAM minus 1 GB), with a floor of 256 MB. MongoDB’s docs warn that in a container with less memory than the host, WiredTiger may not account for the container’s limit. A pod with an 8 GiB limit on a 64 GiB node can get a cache sized from 64 GiB. The cache fills as traffic warms it, mongod passes its limit, and the kernel kills it with exit code 137. Kubernetes reports OOMKilled. A replica set then elects a new primary, and every client reconnects at once.
Check what your mongod sees before you change anything:
db.hostInfo().system.memLimitMB
db.serverStatus().wiredTiger.cache["maximum bytes configured"]
The first number should match your container limit. If it shows the node’s RAM, set the cache size yourself.
The cache fills and app threads evict. WiredTiger evicts pages in the background once the cache passes 80% full. At 95%, or when dirty data passes 20% of the cache, it makes application threads evict pages before they run their own operation. Query latency jumps across the board, even for queries that hit memory. A cache that’s too small for your working set gets there every peak.
The pool math. Each MongoClient keeps one pool per server it talks to. maxPoolSize defaults to 100 per pool. Your primary can receive pods × processes per pod × maxPoolSize connections, plus the driver’s monitoring connections. Each connection costs mongod a thread and memory. Past net.maxIncomingConnections, mongod logs this and drops the new connection:
Connection refused because there are too many open connections
The wait queue. A busy pool makes requests wait for a free connection. The wait ends in an error that depends on the driver:
MongoWaitQueueTimeoutError: Timed out while checking out a connection from connection pool
pymongo.errors.WaitQueueTimeoutError: Timed out while checking out a connection from connection pool. maxPoolSize: 100, timeout: 2.0
com.mongodb.MongoTimeoutException: Timed out after 120000 ms while waiting for a connection to server mongo-0.mongo:27017.
The Java driver waits up to two minutes by default. Your load balancer gives up long before that, so the user gets a gateway timeout while the request still holds a thread.
The MongoDB WiredTiger cache and maxPoolSize config
This example suits a replica set member in a container with an 8 GiB memory limit. Changed lines only, in mongod.conf:
storage:
wiredTiger:
engineConfig:
cacheSizeGB: 3.5
net:
maxIncomingConnections: 3000
Size the cache from the limit. Apply the default formula to the container limit: 0.5 × (8 - 1) = 3.5 GB. Leave the rest alone. mongod needs memory outside the cache for connections, sorts, aggregation stages and index builds, which maxIndexBuildMemoryUsageMegabytes caps on its own. The operating system’s file cache uses what’s left to hold compressed data files, so that memory works for you too. If you run thousands of connections, take the cache lower, since each connection’s thread and buffers come out of the same limit.
On MongoDB 8.2 and later, cacheSizePct sets the cache as a percentage of the RAM mongod detects. It saves editing the file when you resize a pod. Use it only if memLimitMB already shows the container limit, and set either it or cacheSizeGB, never both. MongoDB caps it at 80% and advises against going above the default.
Set the memory request equal to the limit on the mongod container. A pod with a lower request is first in line for eviction when the node runs short of memory. The OOMKilled recipe covers setting limits from real usage.
Cap server connections. net.maxIncomingConnections defaults to (RLIMIT_NOFILE / 2) × 0.8 on Linux, and MongoDB ignores a higher value. Set it to the connection budget you can afford in memory, and raise the file limit if the default comes out too low. A hard cap turns memory exhaustion into a clear error in the mongod log.
Size maxPoolSize from your pods. Connection string options apply to every pool the client opens:
mongodb://app:secret@mongo-0.mongo:27017,mongo-1.mongo:27017,mongo-2.mongo:27017/app?replicaSet=rs0&maxPoolSize=20&minPoolSize=2&maxConnecting=2&waitQueueTimeoutMS=2000&maxIdleTimeMS=60000
maxPoolSizecaps connections per server per client. Size it from the work one process does at peak: requests per second × operations per request × p99 operation time in seconds. Round up and add a little. Then check the product: peak pods × processes per pod ×maxPoolSizemust fit insidemaxIncomingConnections, with room for monitoring, backups and your own shell.minPoolSizekeeps a few connections open, so the first requests after an idle spell don’t pay for TLS handshakes and authentication.maxConnectingcaps how many connections a pool opens at once, 2 by default. A pod that starts cold at peak opens connections two at a time, and requests wait meanwhile. I raise it to 4 or so only when checkout waits show up after scale-out, since a higher value makes connection storms worse.waitQueueTimeoutMSbounds the wait for a free connection. Set it below your request timeout, so a saturated pool fails fast with the error above. The Java driver reads this option as itsmaxWaitTime.maxIdleTimeMScloses idle connections, which keeps the count down after a peak passes.
Create one client per process. MongoClient holds the pool, so create it once at startup and reuse it. A client per request opens a new pool every time. Serverless functions should create the client outside the handler so warm invocations reuse it.
Count processes, not pods. Gunicorn workers, Node.js cluster workers and Puma workers each hold their own client. Four workers per pod with maxPoolSize=20 can hold 80 connections per pod to the primary.
MongoDB Atlas. Atlas sizes the WiredTiger cache from the cluster tier, and you can’t set cacheSizeGB. If cache metrics show app-thread eviction at peak, the fix there is a bigger tier or a smaller working set. Each tier also has a connection limit, and Atlas alerts on connections as a percentage of it. The driver settings above apply unchanged. Amazon DocumentDB and Azure Cosmos DB for MongoDB speak the MongoDB protocol on their own engines. Only the driver half of this recipe applies there.
Test it on your system
Use a staging replica set with production’s container limits, data size and indexes. A cache test on a small dataset proves nothing, because everything fits. Load test through your app, since the pool lives in your app’s processes. Keep the pod count and worker count as they are in production.
Drive a fixed arrival rate with k6, so a slow database doesn’t slow the test down with it. The k6 breaking point recipe explains the executor:
import http from "k6/http";
import { check } from "k6";
export const options = {
scenarios: {
peak: {
executor: "constant-arrival-rate",
rate: 800,
timeUnit: "1s",
duration: "20m",
preAllocatedVUs: 400,
maxVUs: 2000,
},
},
thresholds: {
http_req_failed: ["rate<0.001"],
http_req_duration: ["p(99)<300"],
},
};
export default function () {
const id = Math.floor(Math.random() * 1000000);
const res = http.get(`https://staging.example.com/api/orders/${id}`);
check(res, { "status 200": (r) => r.status === 200 });
}
Set rate to your peak and the thresholds to your own latency budget. Spread the IDs across the whole dataset, or the test only touches a hot corner that fits in cache.
While it runs, watch mongod from mongosh:
// mongosh --quiet "mongodb://admin@mongo-0.mongo:27017/admin" watch-wt.js
while (true) {
const s = db.serverStatus();
const c = s.wiredTiger.cache;
const max = c["maximum bytes configured"];
print(
new Date().toISOString(),
"fill", (c["bytes currently in the cache"] / max).toFixed(3),
"dirty", (c["tracked dirty bytes in the cache"] / max).toFixed(3),
"appEvictUs", c["application thread time evicting (usecs)"],
"conns", s.connections.current, "/", s.connections.current + s.connections.available
);
sleep(5000);
}
Watch the pod’s container_memory_working_set_bytes against its limit, and your app’s logs for the three wait queue errors. In Node.js, the client emits pool events you can log:
client.on("connectionCheckOutFailed", (e) => console.warn("checkout failed", e.address, e.reason));
client.on("connectionPoolCleared", (e) => console.warn("pool cleared", e.address));
Decide on pass criteria before the run. Mine look like this, with numbers you choose:
mongodruns the whole test with no restart, and its working set stays under your chosen share of the limit.- Cache fill stays under 0.95 and dirty under 0.20, and
appEvictUsstays flat. - Connections on the primary stay under 80% of the total.
- No wait queue error appears, and k6’s thresholds pass.
If appEvictUs climbs while fill sits near 0.95, the working set doesn’t fit. A larger cache helps only if the container has room. Otherwise add memory or fix the queries that scan. If checkout errors appear while mongod has CPU to spare, the pool is too small. If they appear while mongod is saturated, a bigger pool makes it worse, so look at slow operations with db.currentOp() first.
Keep it healthy
Scrape mongod with Percona’s mongodb_exporter, started with --collector.diagnosticdata so it exports serverStatus. Run it as a sidecar, so your scrape config can add namespace and pod labels that match cAdvisor’s.
groups:
- name: mongodb-cache-and-connections
rules:
- alert: MongoWiredTigerCacheNearEvictionTrigger
expr: |
mongodb_ss_wt_cache_bytes_currently_in_the_cache
/ mongodb_ss_wt_cache_maximum_bytes_configured > 0.92
for: 10m
labels:
severity: warning
annotations:
summary: "WiredTiger cache on {{ $labels.instance }} is over 92% full"
description: "Application threads start evicting at 95%, and latency jumps. Check the working set against cacheSizeGB, look for collection scans in the slow query log, and add memory before the next peak."
- alert: MongoWiredTigerDirtyCacheHigh
expr: |
mongodb_ss_wt_cache_tracked_dirty_bytes_in_the_cache
/ mongodb_ss_wt_cache_maximum_bytes_configured > 0.15
for: 10m
labels:
severity: warning
annotations:
summary: "Dirty data on {{ $labels.instance }} above 15% of the cache"
description: "At 20% application threads must write pages out. Check disk write latency and IOPS limits on the volume, and look for bulk writes or index builds running at peak."
- alert: MongoConnectionsNearLimit
expr: |
sum without (conn_type) (mongodb_ss_connections{conn_type="current"})
/ sum without (conn_type) (mongodb_ss_connections{conn_type=~"current|available"})
> 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "{{ $labels.instance }} is using over 80% of its connections"
description: "New connections get refused at the limit. Multiply pods by workers by maxPoolSize, lower maxPoolSize or the worker count, and check for clients created per request."
- alert: MongoContainerNearMemoryLimit
expr: |
max by (namespace, pod, container) (container_memory_working_set_bytes{container="mongod"})
/ on (namespace, pod, container)
max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory", container="mongod"})
> 0.9
for: 10m
labels:
severity: warning
annotations:
summary: "mongod in {{ $labels.pod }} above 90% of its memory limit"
description: "Compare maximum bytes configured with the limit. Lower cacheSizeGB toward 0.5 x (limit - 1 GB), or raise the limit and the cache together."
- alert: MongoContainerOOMKilled
expr: |
increase(kube_pod_container_status_restarts_total{container="mongod"}[10m]) > 0
and on (namespace, pod, container)
kube_pod_container_status_last_terminated_reason{reason="OOMKilled", container="mongod"} == 1
for: 0m
labels:
severity: critical
annotations:
summary: "mongod in {{ $labels.pod }} was OOM-killed"
description: "Run db.hostInfo() and check memLimitMB. If it shows the node's RAM, set cacheSizeGB from the container limit. Then check the replica set elected a healthy primary."
Also alert on the app side. Count WaitQueueTimeoutError and MongoWaitQueueTimeoutError in your error tracker, and grep mongod logs for too many open connections. Any of them at peak means the pool math broke.
Roll back
Save the running config first:
db.adminCommand({ getCmdLineOpts: 1 }).parsed
db.serverStatus().wiredTiger.cache["maximum bytes configured"]
Keep a copy of the old mongod.conf and the old connection string. To roll back the cache, restore the old file and restart members one at a time, secondaries first. Then step down the primary with rs.stepDown() and restart it. MongoDB’s docs advise against resizing the cache at runtime through wiredTigerEngineRuntimeConfig, so I use restarts.
To roll back the pool, redeploy the app with the old connection string. Check the product of pods, workers and the old maxPoolSize against maxIncomingConnections first, or you swap wait queue errors for refused connections. If mongod is getting OOM-killed or refusing connections, roll back now. Otherwise roll the restarts through off-peak, since each primary step-down triggers an election and a short write pause.
More recipes
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.
- Fix Lambda TooManyRequestsException Rate Exceeded
Size Lambda concurrency from peak traffic and split it with reserved and provisioned concurrency, so critical functions stop throttling when traffic spikes.
- NAT Gateway ErrorPortAllocation and Port Exhaustion
Alert on ErrorPortAllocation and the signals that come before it, so you catch NAT gateway port exhaustion before outbound calls fail at peak.