Open Source Agentic Inference: Unlocking the Harness with vLLM

Coding agents are turning inference into a continuous workload. A single developer task can trigger dozens of model calls, tool executions, and an expanding context window. Across an engineering organization, those interactions make inference cost, data control, and provider dependence architectural concerns.

Enterprises want to run open-weight models on infrastructure they control while keeping the coding tools their developers already use. But the harness depends on more than model quality. It expects a server that understands tool lifecycles, manages conversation state, and streams a coherent response across multiple inference rounds. The API contract increasingly defines whether a model can participate in the developer workflow at all.

This talk explores how Agentic API, vLLM, and llm-d address that challenge across agentic orchestration, model execution, and distributed inference serving. We’ll follow a coding task from the harness through model generation, tool execution, and continuation, showing what it takes to support the developer experience as more execution moves onto the server. We’ll examine the consequential design choices: where tools run, who owns context, how concurrent work reaches the client in order, and how the serving infrastructure supports these workloads at enterprise scale.

We’ll connect those decisions to enterprise economics: how repeated inference and context growth drive consumption, where shared serving infrastructure can improve utilization, and why cost per completed task matters more than token price alone. The goal is to give enterprises control over the models and infrastructure behind their coding agents while preserving the workflow developers rely on.