Agents That Improve Over Time: Lessons from Shipping an Optimizer

Improving an agent today usually means a manual loop: observe failures, hand-edit the agent's configuration, then re-run evals to validate the fix. This works for one agent, but becomes untenable for a fleet. Agent optimization replaces that loop with search: treat everything swappable about your agent as a config, and search that config space against an eval metric, so your agents keep improving after launch instead of quietly stagnating.

This talk draws on experience shipping the Agent Optimizer in Microsoft Foundry Agent Service and on head-to-head measurements of the major optimization methods.

In this talk I will cover:

• The three layers of an agent you can optimize, and why most teams should invest in the exposed configuration (instructions, tools, skills) rather than the harness or the model

• How optimization algorithms like MIPRO and GEPA work: one base loop with three knobs, and what head-to-head measurements show about when each is worth it

• Where the money goes in an optimization run, and why evaluation, not the optimizer, dominates the cost

• Why eval-creation experience and job latency, not algorithm sophistication, decide whether teams adopt optimization

• How to run optimization safely: side-effect-free test environments for candidate rollouts, human review of proposed candidates, and staged rollout of winners