Notable Quotes
"It reads the objective, it proposes a strategy, and then it writes an actual ML model to implement that strategy. It scores it and then it goes again. It does this about 300 times."
"The worker that writes strategies has no access to the validation set. Only the evaluator sees it. Anything that can see the test set will eventually find an answer. It cheats, essentially."
"Both of them lost revenue with statistical significance. Hiding the module from customers who had already engaged with it cost us with no offsetting gain anywhere else. We assumed low engagement meant low intent, but it didn't."
"The objection to all of this comes down to trust at the end of the day. And you don't talk anybody out of that without strong evidence. You show them six months of the machine's decisions beating the human-based decisions."
"The two losing parlay tests are worth more to me than some of the wins. When you have a loss that becomes a coded constraint for the next experiment, that can pay dividends for a long time."
Takeaways
Autonomy is earned in shadow mode, where the machine's complete package is recorded against every human decision until six months of evidence settles the trust question.
Losing tests become coded constraints: two losing popular parlays tests turned into a 2.5x penalty weight and a hard floor in the next experiment's envelope.
One churn run cut 750 features to 20, lifted F1 by 6% and tested live across three arms of 47,000 users, worth roughly a million dollars over baseline.
