The uncomfortable truth

You ship a “working” agent, and the story usually rhymes like this.

• Demo: one user, one agent; silky smooth.
• Staging: a handful of users, some timeouts, still manageable.
• Production: serious fan-out, and suddenly it feels like a different product entirely.

Same prompts, same tools, same business logic, but now you’re out of DB connections, your HTTP pools are starved, rate limits light up, retries stack into a thundering herd, and the whole thing eats itself. The agent logic didn’t change; what changed is everything around the model calls.

Under real scale the work shifts toward infrastructure, not cleverer prompts in isolation. OpenAI serves ChatGPT to 800M weekly users on a single-primary PostgreSQL fronting millions of QPS across ~50 read replicas. Stripe defends its payments API with a four-layer rate limiter that rejects millions of misbehaving requests a month, the same pattern you need around an LLM provider key. Discord scaled 5M+ concurrent WebSocket connections on Elixir/BEAM, a different runtime expressing the same lesson your Python async stack relies on: cheap concurrent units, not threads, are how you fan out across thousands of network calls. The common thread is a production toolkit you would not skip on any critical path: queues, pools, backoff, rate budgets, and observability. Treat agent infrastructure with the same rigor as any other production system.

Why async actually buys you concurrency

async multiplexes many paused I/O waits on one thread but it does not hide blocking calls or long CPU work on that same thread. Under load that distinction is what decides whether your loop keeps moving. Python’s event loop is basically a single-threaded scheduler built from a ready queue of callbacks, an OS-level I/O multiplexer that tells you which sockets woke up, and a time-sorted heap for “run this later”. It loops forever, draining ready work, asking the OS what is readable or writable, then scheduling the next callbacks.

When a coroutine hits await, it doesn’t park a thread; it registers interest with the kernel and yields control back to the loop. That is why a single process can have tens of thousands of in-flight LLM calls. Each one is a cheap coroutine waiting on network I/O, not an 8MB stack thread doing nothing.

The gotcha is almost boring, but costly. Blocking work still blocks the loop. Because everything shares one thread, any stretch of synchronous I/O or non-yielding work can freeze the whole scheduler. We’ve run into production incidents where sync HTTP clients sat inside async handlers, blocking database drivers held the loop, or code waited synchronously instead of yielding so other work could run.

Common traps in agent-shaped code include the following.

• Synchronous HTTP clients inside async functions
• Synchronous database drivers
• CPU-heavy JSON parsing on massive model responses
• File I/O without async wrappers

The rule stays simple. In an async context, I/O should yield. If it doesn’t, the “100k concurrent agents” story collapses into one thing at a time.

HTTP under real load

Every LLM call is just HTTP with extra vibes. At scale, timeouts, resets, 429s, and flaky 5xxs are not edge cases; they happen all the time. You want pooling, layered timeouts, retries that don't make things worse, and a circuit breaker so you’re not hammering a dead endpoint on principle.

Opening a new connection repeats DNS lookup, the TCP handshake, and TLS negotiation. That setup alone routinely costs tens to a few hundred milliseconds before you send application data on a connection you have not reused yet. Stack thousands of those setups and you burn minutes on overhead alone. Pooling amortizes that setup across many requests and the critical alignment detail is matching your pool's per-host connection limit to whatever concurrency cap you think you're enforcing with semaphores. If fifty coroutines can hit the LLM but the pool only owns ten sockets, forty of them are just waiting on each other.

Different failures deserve different clocks. Layer them so each is tighter than the next: connect ≤ TTFT ≤ inter-chunk ≤ total. Connect timeouts should fail fast on a dead route, but not so tight that they false-positive on a normal TLS handshake. TTFT (time-to-first-token) covers “request accepted, model has not produced a token yet” and needs to absorb provider queueing and reasoning that model warm-up without cutting off legitimate slow starts. Inter-chunk read timeouts need slack so tool-calling and reasoning pauses inside an otherwise valid stream don’t look like a stall. In practice, timestamp each chunk and fail the stream early if the gap exceeds a per-chunk ceiling, which catches “accepted, started, then dropped mid-generation” failures faster than a wall-clock total alone. The total cap is the hard backstop, sized from product UX rather than a stock default. Anything that wants to run beyond your product’s tolerance belongs in a durable workflow, not one HTTP call.

Retries only work if you classify errors first. Permanent failures (400, 401, 403, 404) should not loop because the request is wrong or unauthorized. Rate limits (429) need retries that respect Retry-After, not blind hammering. Server overload (502/503/504) needs exponential backoff so you aren’t pile-driving an already sick dependency. Transient blips (resets, timeouts) get the usual backoff, plus jitter, because synchronized retries are how a thousand agents become a thundering herd.

Circuit breakers are the production version of “stop calling this for a bit.” When a dependency's error rate crosses a threshold, the breaker opens and fails fast locally instead of wasting slots on an endpoint that is already unhealthy. After a cooldown, it moves to half-open and lets a small amount of probe traffic through. If the probes succeed, it closes again. If they keep failing, it opens again. Scope breakers per dependency, provider, model, or region so one flapping endpoint does not disable the fallbacks meant to save you.

PostgreSQL: the part people undertune

There is a reason the "just use Postgres" meme has stayed popular :). A well-tuned PostgreSQL instance can carry far more load than we can expect, and its feature set covers enough ground that adding extra moving parts is rarely the first answer. Every extra hop is another network round-trip, another on-call surface, and another place consistency can quietly drift from the source of truth. Add a moving part when measurements demand it, not before.

Reads stack in three layers, fastest first. Hits in Postgres shared_buffers are usually cheapest. A miss may still be satisfied from the OS page cache, with no physical disk read, but Postgres still needs to copy that page into shared_buffers. When both caches miss, you fall through to storage, which is far slower. On a dedicated host, shared_buffers around a quarter of RAM is a common starting point so the kernel still has room for page cache and filesystem work. In addition, Effective_cache_size is not allocated memory; it is a planner hint for how much cache Postgres can reasonably expect across shared_buffers and the OS page cache, often somewhere around half to three quarters of RAM. In production, watch buffer hit ratio, query plans, pool waits, lock waits, and p95/p99 latency. If those look healthy, many reads are already coming from memory one way or another, which is the bar Redis has to beat. On read-heavy hosts, let spare RAM stay useful as page cache instead of reflexively adding another cache tier.

Postgres Caching Layers

Redis: add it for a reason, not by reflex

Hot take, but I stand by it. Redis is another network hop, another cache invalidation story, and another on-call surface area. If Postgres is already hot in memory and returning what you need, "faster in isolation" isn't automatically faster end-to-end. Measure cache hit ratio and tail latency first. A lot of "we need Redis" is really "we haven't looked at indexes and pool sizing yet."

Redis earns its keep when you need something like the following.

• True sub-millisecond latency requirements Postgres can’t meet after honest tuning
• Specialized structures such as TTL-heavy rate limits, pub/sub, sorted sets
• Distributed locks across many app instances
• Ephemeral session state you can afford to lose on restart

For a lot of agent stacks, that list is shorter than people expect, which is good news if you like sleeping through the night.

Queues: Turn Spikes Into Backlog

The pattern is old. Without a queue, a traffic spike means your API tries to run a thousand agents at once, each holding DB slots, HTTP pool capacity, and RAM, and the stack falls over in a correlated splat. With a queue, submission decouples from execution, so spikes mostly grow backlog depth, which is a knob you can monitor, cap, and shed. Workers pull at a sustainable rate, and retries and dead-letter behavior live in the broker instead of half of SQS reimplemented in app code.

If depth is unbounded you eventually run out of memory or disk. The quieter failure is staleness: the worker reaches the job after the user has already timed out or given up. Backpressure is how you refuse to absorb endless demand: reject or slow enqueue past a depth threshold, shed low-priority work to protect the hot path, or return an explicit busy / retry-later so callers backoff instead of joining a minutes-long pipe. The right mix depends on the product; for user-facing agents, a fast “please try again” is often better than landing in a five-minute backlog.

Prefetch is your distributed semaphore. Ten unacked messages per worker across fifty workers is five hundred in-flight agent runs, regardless of how deep the queue is. Line that number up with provider rate limits and your PgBouncer pool on purpose. Pair it with queue-depth autoscaling and the system self-regulates. Backlog grows, replicas add (up to a cap), work drains, replicas ease off.

Consumers should be idempotent. Exactly-once delivery is not something a broker can magically guarantee end-to-end; failures can happen after work completes but before the ack lands. Treat at-least-once delivery as normal, then enforce exactly-once effects in application logic with dedupe keys, outbox patterns, and state checks. Record a stable task id when side effects complete, then skip if it has been processed before.

Postgres-as-a-queue (SELECT … FOR UPDATE SKIP LOCKED) is a solid option well past “demo” scale. Postgres can carry serious throughput if you operate it well. I have seen teams stay comfortable on the order of 100k events per second before a dedicated broker is even on the table. The catch is discipline: the queue table is one of your hottest writers, so dead tuples accumulate fast. Tune autovacuum aggressively and prune finished rows on a tight schedule, or bloat catches up and surfaces as long stalls and random-looking crashes under load.

Autoscaling with queue depth

KEDA-style scaling on queue depth is the straightforward mental model. Backlog grows, replicas add, backlog falls, replicas trim. Make scale-down lazier than scale-up. When depth spikes, users are waiting, so add capacity immediately. When the queue drains, don’t rip capacity away instantly or you’ll thrash on the next wiggle. A short stabilization window before scale-down is cheap insurance.

Once you write down the guardrails, the math gets boring in a good way. If you target fifty pending messages per replica, cap replicas at a hundred, and each worker prefetches ten jobs, your worst-case in-flight work is on the order of replicas × prefetch, or a thousand concurrent agent executions in that toy example, bounded by the pools and rate limits you sized upstream.

Observability: traces, logs, and metrics

Async fan-out turns “grep three surfaces” into the default debugging path. OpenTelemetry is the usual answer — traces, metrics, and logs as one model, emitted from ingress through workers and LLM clients with context propagated end-to-end. Reuse the same trace IDs in your structured logs so a user-visible timeout collapses to one worker line. Keep metrics RED-style on hot paths plus queue depth, pools, breaker state, and provider 429s. The LGTM stack (Loki + Tempo + Grafana) gives you LogQL and TraceQL out of the box. If you run LangGraph, LangSmith adds a nice graph-native trace and eval UI on top, which we use alot to traceback failures and different error types.

Start with structured logs, request IDs, basic metrics, and traces. That baseline is cheap enough to add early and saves painful guessing once work fans out across APIs, queues, workers, and LLM calls. Add tenant, cost, and cardinality controls when noisy neighbors, bills, or telemetry volume become real problems. The goal is enough signal to answer “what broke and why,” not instrumentation on every line.

Putting it together

The shape that keeps showing up starts with an API that accepts work (sometimes shedding load). A queue decouples submission from execution, workers scale with backlog and respect prefetch as a global throttle, HTTP clients are pooled and defensive, traces and logs carry shared context end-to-end, Postgres holds state behind PgBouncer with replicas or partitions when needed, and extra infra only appears when measurements demand it.

Each layer targets a different failure mode.

• Structured logs and traces tied to the same request IDs
• Pooling (HTTP + DB) fights connection exhaustion
• Breakers and classified retries stop cascades and thundering herds
• Prefetch caps runaway concurrency
• DLQs keep poison work from infinite loops
• Partitioning keeps retention from becoming a vacuum crisis
• Autoscaling matches spend and capacity to real queue pressure

In my experience, reliable stacks rarely win on the cleverest prompts alone. They treat the system around the LLM calls like any other production surface. Start from the Postgres deployment you actually tune. Understand your event loop. Harden your HTTP clients and circuit breakers. Propagate trace context through workers and provider calls. Pool everything that can be pooled. Add Redis only when measurement forces the issue. Put a queue in front of heavy work. Let autoscalers react to depth instead of vibes.

If you are building this kind of system, the lesson stays simple. The model is the interesting part, but the infra is what decides whether users get an answer or a timeout.