See all guests

Guest Spotlight: Marco Casalaina

Marco Casalaina
Microsoft | Exec

About Marco Casalaina

Marco Casalaina is VP of Products for Core AI at Microsoft and a self-described AI futurist. He works on the evolution from chatbots to open-world agents, focusing on evals, go-live metrics, and Copilot velocity, the measurement discipline that determines whether increasingly autonomous AI products are truly ready to ship at scale.

Notable Quotes

"We discovered that about 80% of accesses to our documentation were coming from coding agents."

"We are now entering the era of open-world agents—agents that are capable of doing whatever."

"The thing you need to do now is master the art of evaluation."

Marco Casalaina
Microsoft | VP of Products, Core AI

From chatbots to open-world agents at Microsoft: evals, go-live metrics, and copilot velocity

Go to S1 | E4

Subscribe to the podcast

Takeaways

All Takeaways

Guardrail open‑world agents: add out‑of‑domain evals and explicit capability limits; set acceptable error rates based on the stakes of your use case.

Go to S1 | E4

Write for agents as readers: tighten documentation, ship vetted code samples, and monitor bot traffic patterns.

Go to S1 | E4

Build self‑verification into workflows: pair agents with automated testing (e.g., browser runners) and iterate to thresholds, not perfection.

Go to S1 | E4

Productize adaptability with model routing to match tasks to the right model family as capabilities shift.

Go to S1 | E4

Evolve evals for agents: track tool call accuracy/success and task completion/adherence; A/B test models and strategies.

Go to S1 | E4

Measure value by go‑lives and real usage (token volume), not time in portals or playgrounds.

Go to S1 | E4
The experimentation edge podcast logo with a picture of host Ashley Stirrup