"You can design a very creative and very good mechanism for players, but you don't really know what setup works best for the masses."
"If we make the experience of the game better, business outcomes come naturally. It's not our first goal."
"If you see the data contradicts with your hypothesis, that's actually the best part of the experiment, where you learn something completely new."
"Game players go to the game to have fun. If we are testing them by giving some people a disadvantage, it's not a good experience, because their main motive is just to have fun."
"Basically everyone can run and analyze experiments on their own now. But it also creates a new challenge: we don't know the quality of those AI-driven analyses."
Takeaways
AI-powered self-serve analysis means everyone can now run and analyze experiments, so the next challenge is making the quality of AI analysis consistent across the whole company.
Promise fairness, not just transparency; players who get the worse variant always receive a make-up event later, because game players come to have fun, not to be disadvantaged.
Announce experiments to users before they run; Supercell's community managers tell players what is being tested and why, which turns a skeptical community into a research partner.
