Supercomputers and AI Agents for Self-Remediation at Azure High Performance Computing

A year ago, operating our GPU fleet scaled O(n): more clusters meant more humans running more runbooks. Today a control plane manages AI-supercomputer GPU capacity across US data centers with near-constant human effort. This talk covers how we got there: the control-plane architecture that turned per-node servicing into fleet-level load balancing and process management, and the self-remediation AI agents that now handle routine failures. 

Key Takeaways: 

1. Architecture patterns that move fleet operations from O(n) human effort to O(1): control planes, load balancing, process management over runbooks. 
2. A production guardrail design for autonomous remediation agents: closed action sets, evidence grounding, graduated autonomy, escalation asymmetry. 
3. How to measure whether an ops agent deserves more autonomy, and when it should hand off to a human.