Make Each Token Count: Compression, Caching, and Retrieval for Production LLM Context

Every agentic AI system today is running the equivalent of an uncompressed MapReduce shuffle: bloated contexts, redundant retrievals, and cache misses that silently multiply cost and latency. As agents move from demos to production, tokens, not GPUs, become the dominant cost and latency bottleneck. Yet most teams treat context as a string they keep appending to, rather than a managed resource with a cost model.

This talk maps the classic distributed-systems playbook: compression, caching, tiering, observability, onto the token layer.

I'll cover three patterns proven in production:

1. Learned prompt compression — training a small encoder (ModernBERT-based, dual-head) to decide what to keep, drop, or summarize, and why task-aware compression beats heuristic truncation
2. Prefix-cache-aware context alignment — restructuring prompts so provider-side KV caches actually hit, turning small prompt-ordering changes into measurable latency and cost wins
3. Compress-cache-retrieve pipelines that treat conversation history and tool outputs as tiered storage

I'll share real production numbers from enterprise deployments and our open-source work in this space — token reduction rates, cache hit improvements, quality-preservation benchmarks — plus the failure modes we hit: when compression breaks tool-calling, why naive truncation destroys agent trajectories, and how to measure "quality loss" when there's no ground truth.

Drawing on 15 years building infrastructure at Netflix scale (Netflix Drive, Axion), I'll argue that context engineering is becoming its own infrastructure discipline and give you the mental model to own it.

Main Takeaways

• Tokens are the new bandwidth. Context is a managed resource with a cost model; treat it like one — compress, cache, tier, and observe it the way you would network I/O or memory.
• Compression is a quality problem, not just a size problem. Tool schemas, code, and conversation history each fail differently under naive truncation. Learned, task-aware compression beats heuristics — and you must benchmark quality preservation, not just token counts.
• Provider-side prefix caches are free money most teams leave on the table. Stable prefixes and deterministic prompt ordering can dramatically change KV cache hit rates — no model changes required.
• Observability for context is a blind spot. Per-request token accounting, cache-hit telemetry, and quality regression detection are prerequisites for optimization, not nice-to-haves.