Notable Quotes
"I use one simple rule: blast radius times reversibility. Big changes demand relentless testing; small, reversible ones don't—ship fast, watch the monitors, and fix it forward."
"High activity doesn't equal high quality. If users keep retrying, they're stuck. I look at success ratio—fewer clicks to the right outcome. Efficiency is the real marker of quality."
"Optimizing one metric ruins the others. We built a composite score—cost per quality inference (CPQI)—stitching cost, latency, and accuracy. It forces alignment and keeps us honest: fast isn't always good."
Takeaways
Build composite metrics (e.g., CPQI) to align finance, engineering, and data science around shared outcomes.
Treat ML features as living systems: feature-flag rollouts, realistic staging, drift monitoring, and LLM-as-judge evaluations—and be willing to kill “wins” that erode trust.
