Be Your Toughest Customer: Shipping Collaborative Agents Twice, to Our Own Org and to Lawyers

Harvey has two demanding customers: lawyers, and ourselves. Before our agents touched a legal matter, we shipped them internally as Spectre, our cloud agent platform, now past 7,000 merged pull requests across every repository. This talk covers five problems every production agent platform hits, each solved twice: durable runs that survive dying workers, meeting users inside Slack for engineers and inside Word and Outlook for lawyers, per-user agent identity after a shared service account destroyed attribution, information boundaries that map one to one onto ethical walls and approval trails, and context engineering after restoring everything made agents measurably worse. Failures get equal billing with the wins, and every metric comes with its denominator.

Key Takeaways:

1. Make your own org depend on an agent platform first: every customer-side problem (durability, identity, boundaries, evals, context) surfaces internally months earlier, where failure is cheap.
2. Collaboration and boundaries are the same feature: per-user identity, scoped credentials, and audit trails are what make shared agent runs safe enough for a profession where information barriers carry legal weight.
3. Trust is evidence, not demos: honest denominators, decision logs for the failures that produce no signal, and eval gates so nothing ships on vibes.