▸ Tag · #llm-inference
LLM inference.
Serving models efficiently: prefill and decode, KV cache, GPU-aware load balancing, vLLM, and the latency budget behind time-to-first-token.
-
AITraining vs inference: scaling AI systems
How training and inference differ in compute, cost, and hardware, and how to architect each phase so your AI app stays fast and affordable at scale.
Read post →
-
ArchitectureSmart auto-scaling for modern AI apps
CPU auto-scaling fails GPU workloads. Why queue depth, KV-cache pressure, and TTFT beat CPU as triggers, plus the KEDA patterns to scale AI in time.
Read post →
-
ArchitectureGPU-aware load balancing for AI inference
Round-robin breaks when LLM requests span 50 to 50,000 tokens. GPU-aware load balancing with prefill/decode disaggregation and the metrics that cut P99.
Read post →