▸ Blog series
Serving AI in Production
GPU-aware routing, scaling triggers that actually track LLM load, and the rate limits that stop one agent loop from burning your margin.
- 01
GPU-aware load balancing for AI inference
Round-robin breaks when LLM requests span 50 to 50,000 tokens. GPU-aware load balancing with prefill/decode disaggregation and the metrics that cut P99.
Architecture · 8 min read - 02
Smart auto-scaling for modern AI apps
CPU auto-scaling fails GPU workloads. Why queue depth, KV-cache pressure, and TTFT beat CPU as triggers, plus the KEDA patterns to scale AI in time.
Architecture · 6 min read - 03
Rate limiting: protecting your AI wallet
One runaway agent loop can mean a $5,000 LLM bill. Why request-per-second limits lie, and how hierarchical token-bucket limits protect your margins.
Architecture · 7 min read