Transcript

The Experimentation Edge - Luis Trindade

===

Ashley Stirrup: [00:00:00] Hello, and welcome to today's episode. Today we have Luis Trindade, Principal Product Manager of Experimentation at Farfetch. Luis, welcome to the show.

Luis Trindade: Thank you for inviting me and having me here. It's a pleasure, looking forward to it.

Ashley Stirrup: And you've been at Farfetch a long time. Maybe we start off with you just telling us a little bit about the company, and then we can talk about your background.

Luis Trindade: Of course, happy to do it. I joined the company 12 years ago already, so it feels like an eternity. In the middle of that, lots of ups and downs. I basically joined the company when we were starting to expand towards our tech hub outside of the Porto area, into Lisbon. So we created a new tech hub here, and I came in to help build the product team over there. For those who don't know Farfetch, we are one of the biggest global markets for luxury. Our goal is [00:01:00] exactly that: to make sure that we offer a connection between the supply that exists all over the world and the clients that are all over the world. We create that connection especially for unique brands that people cannot access in their local shops, and that's our main driver. And we have been growing a lot. Lots of ups and downs. Three years ago we got acquired by Coupang.

We have been focusing the business on the marketplace itself. We had lots of business explorations before that in the offline world, store-of-the-future concepts and the like. But nowadays we have been focusing on what matters more, which is the marketplace, and making sure that we create an experience for all the clients, but in a way that is structured and [00:02:00] scalable. Part of my role is making sure we do it sustainably, taking advantage of all the data and insights we can generate.

Ashley Stirrup: Got it. And how big was the company when you joined?

Luis Trindade: Quite small. I was employee number 700. So from somebody that came from a startup world where we were like five doing everything, it was actually quite a big jump. But compared with what we went to, we were up to 7,000 people worldwide. Nowadays we are a bit smaller. It was quite a journey from starting at a relatively small but already established company with lots of room to grow. One of the biggest challenges that we started facing with that, because in that hyper-growth stage we saw an opportunity to ask: how could we make sure that everybody, especially when we were creating this new tech [00:03:00] hub, how can we make sure that everybody's taking the learnings, using the same methodologies, using the same approaches, learning from each other, making sure that we have this sustainable growth instead of just adding and adding more people doing things. And that's where the role of this experimentation area came in, which was supportive of product-led growth. I always connect experimentation and product management very closely. Sometimes I even challenge if it's not the same. And that's how we also took that advantage. Humbly, I would say something that helped that growth was those early decisions of having such a strong experimentation-led organization.

Ashley Stirrup: And so when you joined, did you start off in experimentation?

Luis Trindade: Not directly. I started in a different area, [00:04:00] which was the data products area. In reality it was created as a proposal when I joined. And after that, we saw that we had lots of different areas spread across the company. I mentioned Farfetch at some point had the goal of being a platform of marketplaces, meaning that other players could build their entire solution on top of us. And one of the requirements was to make sure that we had all the different modules of our offer productized. One of them, and probably the most relevant here, was all the recommendations engine, the search engines that we were building. And at some point, we saw the opportunity of taking them, instead of just being another asset that we had there, to elevate it into a proper data product with their own evolution model, and making sure that they grew as a potentially different sub-product that we could offer to [00:05:00] our clients. That's where I started at the company, leading the growth of that area. And then the jump to experimentation was the next step, after two or three years.

Ashley Stirrup: And when you say marketplace, could you say a little bit more about that? Are you a little bit like an eBay for fashion products?

Luis Trindade: You can call it the Amazon for fashion. It's an interesting one, because Amazon has been trying to enter this luxury space. So not just fashion, luxury fashion in particular. Amazon has been trying to enter in this space for a long time. And even with their power, they haven't cracked it.

I think the secret sauce, if there is any, was all about these relationships between the different boutiques, the brands that are still very old school and oriented to traditional retail. [00:06:00] They really wanted somebody who understood their businesses, and how to propose to them a different way to engage with these new markets. That's how basically Farfetch started in the early beginnings, and we keep evolving it in that sense. We tend to not have basically any type of stock on our hands, making sure that we are all virtual in terms of connecting the supply that exists from the different suppliers, but making sure that we deliver it in the faster way. Faster being the most recent launch that we did, which is Farfetch First. It's a big program here in Europe. Next-day delivery, which you also have with Amazon, feels natural. Nowadays, we all see that as a commodity. In the luxury industry it's not there yet, [00:07:00] and we are offering it right now here in Europe.

So basically in entire Europe, we can deliver in the next day most of the items that we can offer. And that's a big win for the value proposition that we are offering as a marketplace right now.

Ashley Stirrup: That's pretty incredible, because your sellers are shipping direct to your buyers. Is that correct?

Luis Trindade: In some cases. In others, we have a central warehouse to support those operations.

Ashley Stirrup: Got it. Interesting. And tell us a little bit about experimentation at Farfetch. How did that start and how has it evolved?

Luis Trindade: I think it started with a bit of background on myself, because I entered this area of experimentation and product development almost by chance. Things started to make sense in the days when product management was not even called that.

So yes, I'm that old. But [00:08:00] it was interesting how I always tried to apply these principles even when I was working and launching many startups, validating their ideas, testing those ideas very quickly, testing and learning as fast as we could. That was the motto for any startup, but in that case it was definitely the need. So I always understood how to connect and validate an idea, and how to validate it against real customers, making sure that we were pursuing the right things and the right ideas and evaluating them in the right perspective. I think that mix of understanding the different perspectives as one is what makes experimentation a thing. So when I joined Farfetch we were in that hyper-growth stage, onboarding lots of people as a company. It was the moment where we were also splitting for the first time our tech hub outside of Porto, [00:09:00] from the Guimaraes area to Lisbon, to take advantage of gathering more qualified talent. And that was a moment where, when you start doing that, processes that were very lean, in the sense that you would just lean over to a colleague's desk and understand how they were doing that stuff, or the other way around. When you do it that way, you need to start having some processes that helped the teams to grow in that sustainable manner, and making sure that knowledge was not siloed and that it was shared across the company. That's where, after joining two or three years and with the experience of what we were doing in the data products area, we saw a huge opportunity to put in place an organizational shift, with experimentation becoming a center of excellence. [00:10:00] So we were not as we had been at the beginning, where you have the engineering team that runs their own tests with their own tool that they actually implemented. The marketing team would use an external vendor because that's the one that they can get access to. Some other team didn't even know those capabilities existed.

So how could we make sure that we talked the same language all as a company? And instead of that typical model where you gather everybody who has that knowledge and then wait for somebody to ask you to execute for them and deliver just the results. I never believed in that, and it was very interesting how the community and industry evolved towards understanding that was not the right idea. And we actually started very early on that process, which was interesting. But the leadership also believed in that view and gave me the sponsorship to put it in place, which was a driver to where we are nowadays, [00:11:00] which is having this center of excellence, quite small - it actually reduced over time while keeping usage at levels higher than ever before. We also chose that the small maintenance team is focusing especially on the engineering side and data science side, and now refocusing all our efforts on practices. Because to be good you need to practice, practice and repeat. And that's probably where I spend most of my time nowadays. It's helping the different product teams, and that's also a difference. In most companies all the experimentation comes from marketing. It's all about CRO, all about optimizing the campaigns. [00:12:00] But where I saw the huge potential, Farfetch being a tech company and a product company by definition, was actually to be the enablers on the product side, on the engineering side, and making sure we use all those practices and we removed any blockers for the entire team to do experimentation. Which was interesting also in our approach in terms of build versus buy. So we were using external tools to support the marketing team and the CRO perspective. On the engineering side, we built our own, Fabs. We call it Fabs, the Farfetch A/B testing system. All internal: the split engine, the setup engine, the stats engine, everything built in-house. And it was all about the engineers, making sure we were offering those [00:13:00] capabilities. At some point we integrated this external vendor, mainly used by marketing, but making sure there was connecting glue between all of it. We also implemented our tracking layer entirely.

And remember when I mentioned we were at some point even exploring the offline world, making sure that the boutiques were part of the store of the future. So we had to create a system that was not just tracking events on a website, but through the entire journey of a customer, including when they were, for example, entering a boutique and to be recognized there, making sure that the entire experience was interconnected. We call it omni-tracking, and it was a cornerstone to where we are nowadays. Even with that external vendor, we had a stage where we were in a hybrid approach, basically with our own setup plus this [00:14:00] external vendor. But then we realized that, especially given our maturity level as a company, and since we are a tech-heavy company, it didn't make sense to have this external vendor. At some point we realized that between the negative performance impact, the lack of control, and all the rework we were already doing to get the data back to us for our own analysis and deep dives. That is one of the key value propositions of Fabs: doing all that automatic deep diving on the results instead of just looking at a certain metric and win or lose. That doesn't say anything. All of that led us to say, "Let's focus internally on having just Fabs powering the entire solution." And that was when we started evolving Fabs to the version it is nowadays. We call it Fabs [00:15:00] 2.0, which extended Fabs from just an A/B testing tool mainly for experimentation on the product side, but also to enable the different areas of the business to do all sorts of experiments. So we started by creating an ecosystem of modules. A key one was the feature toggle. We created a feature toggling system that is deeply interconnected as part of the experimentation platform. Nowadays it is the only entry point at Farfetch to run experiments.

Everything needs to go through our feature toggling system. It's much more than the typical feature toggle with key values just on or off. It's deeply interconnected with our segmentation service, our benefits service, our user systems. So we are able nowadays to define who sees what and when, and connect that to a randomized split. [00:16:00]

So it's not just a typical engineering solution for risk mitigation or controlling deployments. It's much more than that. At the same time we started connecting with our CMS, our recommendation system and our messaging systems, which allowed the marketing and content teams to do those types of experiments without relying on an external tool, and at the same time having the full control over what they were testing, when and what. Because as many of you know, especially with tools based on JavaScript and code injection, they break a lot.

They create inconsistent results. So having all of that centralized, that's where we got into, and that's where we are right now.

Ashley Stirrup: One question I wanted to ask you was that it sounds like early on marketing was a silo. [00:17:00] They were using their own tool. You were doing things internally. As you brought the marketing and product teams together, did that kind of unlock some either new use cases or new ways of learning, because people were working together more?

Luis Trindade: Totally. Even if organizationally we are still separate - marketing, the commercial teams, content teams - in the ways of working it made total sense. For that, we didn't just build the software. It was not just Fabs. It was much more than that. It was all about building the other tools, the ceremonies that actually helped people to start working more closely together, having common objectives like let's get together as a group, defining together what are the next best hypothesis to be tested, evaluate, challenge the actual hypothesis, the definition of success of them. That was [00:18:00] a key moment, because we started creating this loop of knowledge that whenever a new joiner or somebody that was quite senior but not so aware of the processes, sometimes they simply entered those sessions just listening, and they were already learning and upskilling their knowledge to be able to even already providing some inputs to the solutions that were being discussed or the hypothesis being discussed, but at the same time already thinking, "How could I create those hypotheses myself?"

So that's where 80% of my time on experimentation is mainly about that, about enabling, making sure that all those ceremonies keep rolling, and stay engaged, creating a knowledge base for all the experiments and learnings in a central manner, using a single template that actually is used by everybody on the [00:19:00] company to define a hypothesis. All of that nowadays, it feels like, okay, it's part of the playbook, everybody's doing it. Not everybody, which is the reality, but at the same time, it's what made us use that common language, making sure that we were doing things in a much more concise way instead of having silos of knowledge.

And for example, another ceremony that we also implemented, so besides these weekly experimentation clinics - I call them that because they're basically a group therapy session - we also have the monthly test and learn session where we actually share all the knowledge around the company, and those are open to everybody from the junior developer up to C-levels, where they listen, but not just listen, they can get and create a discussion, sometimes not as deep as those clinics, but sometimes it triggers really interesting [00:20:00] outcomes. We go there sharing not just a reporting session where we go say, "Okay, we launched this test. This is a winner, loser, X amount of potential money that we got from it." That's where it started and quickly evolved to be a session where we were sharing basically most of only the learnings. That was another thing that we also changed, was not talking about winning or losing the experiment. We rebranded it, even in our tooling: are we able to learn from this experiment, yes or no? A failure is actually a test that was badly set up: wrong metrics, sampling biases. That was a failure test. All the other tests are opportunities to learn. In most cases, on some we proved the hypothesis and on others we didn't, but in both situations we learn. And that was also a trigger for how people were [00:21:00] seeing experimentation, not as "let's not be afraid of failing because I'm not going to be able to prove my hypothesis", but as a way to say, "Okay, let me try this thing. If I learn that this is not the way to go, perfect. Let me proudly say to everybody that we just avoided spending more money on something that was not worth going towards."

And that shift was great, because we don't use any winning rate. So it's a common metric that everybody tends to use. "Oh, what is your winning rate?" We don't even track it. We have it there on a dashboard, but it doesn't matter.

Ashley Stirrup: Yeah.

Luis Trindade: It's all about the learning rate. And for me, my goal is always to have a learning rate of 100%, meaning that we don't have failed tests. Of course we do have failed tests, but that's the goal. We should always be going towards the zero [00:22:00] failed rate.

Ashley Stirrup: I love that, having a goal of a learning rate versus a winning rate. And you're doing a fair number of tests - you scale things up and down depending on whether it's a high season or not. But when it's a low season, you're doing a couple hundred experiments a month.

Luis Trindade: A lot of learnings.

Ashley Stirrup: So that's an opportunity for a lot of learning.

Luis Trindade: And the good thing I found is that it's mainly about product learnings, not just about the content, which are the ones we run during the peak seasons, because that's when we have the volume for those small tests. But lots of learnings come from the low season.

Ashley Stirrup: That makes a lot of sense. Can you tell us about an experiment that you ran where you had a lot of learning?

Luis Trindade: Let me use one from where I actually started here at Farfetch, which was the recommendations area, the data products area. It's an old one, but I think it [00:23:00] was one of the most impactful in terms of learnings. So as I mentioned, we started building our own recommendation engine at some point in time. We were using a third party, probably the world leader in recommendation engines, one that many of you will know, but we quickly started to feel that it was not the right thing. Also, it was not strategically good for us. Like I said, we were building a platform for creating new marketplaces on top of us, so removing all those dependencies in terms of external vendors was a plus. But at the same time, we felt mainly that there was an opportunity to actually take advantage of the entire lake of information that was sitting out there and that we were not being able to feed into this external vendor. So we started creating this engine, we call it Inspire still nowadays, and surprise, at the beginning, it was completely losing against the world leader of recommendations. It was [00:24:00] a great exercise also to understand that not always it's all about failing fast, in the sense that, okay, failing and just dropping that idea entirely. There was a strategic decision to explore that, and it was more than just a couple of failed iterations. There was something that pushed us further, and that's also important many times. But what was important was to understand very quickly in which direction should we continue investing our dollars. Every tiny dollar that we invested in that engine, we had to justify, because at the same time we were still spending on the third-party vendor. It's always tricky to explain that to the business. But we kept running very quick, strongly directional experiments. [00:25:00] Not all of them A/B tests. That's also a big learning. I always say we all have a tool belt of experimental tools. All of them are valid. We just need to understand their different capabilities and limitations, and use them at the right moment to get the fastest learning we can. So we mixed quasi experiments and qualitative insights, all in the sense of gathering and creating stronger and stronger insights to tell us if we were going in the right direction, or if not, quickly stop and move to the next one, but always keeping the same direction of the vision. It took almost two years, this long experiment. So it was hundreds of iterations on that experiment. But at [00:26:00] around half of that time, we saw the tide shifting. That was the moment we believed we were proving the hypothesis and could change from one service to the other. We took that decision, invested more, and started reducing the investment in the third-party vendor, and we started to phase it out and actually evolve the product that is nowadays being used for everything on Farfetch, basically. And so that tells a story of resilience, but always with a vision and a direction, which is very important.

Strategy is key when we are doing experimentation, but at the same time, we need to do it in multiple and small, quick learning iterations to tell us if we should keep investing on that direction or not, but always finding the path towards our vision. And it was also an example [00:27:00] of building something genuinely hard. Many times we test the blue button versus the red button. In this case we were testing an actual recommendation engine, and a lot of learnings came from it. We were testing at the platform level, tracking challenges, making sure that we were proving the value on different levels. So lots of learning opportunities came from there, which many people at the beginning would have said we should not go down.

But it proved out and it was really a great learning.

Ashley Stirrup: So to summarize what I think I'm hearing you say: you took on a pretty ambitious task and you wanted to leverage lots and lots of data to do personalized recommendations. And you just had to keep iterating over and over again to see what was working, and you really had to track a lot of different metrics at the same time in order to get to the outcome, which sounds like it was a [00:28:00] pretty strategic win for the company.

Luis Trindade: It was, in many ways. Besides proving that we now have an internal solution that delivers more value than the third-party vendor. The cost, without going into details, compensated a lot, even with the full maintenance of the system.

But again, it's part of our DNA here, and part of our strategy to have the tech team to support it. So I would not recommend doing this for a company where having a tech team is not core.

But the main learnings were in the process the company went through, and this was done in the early stages of the experimentation center of excellence. So it was a huge case of driving awareness on how to do things and how they could do the same approach on other areas.

Ashley Stirrup: You built a lot of your own best practices around how to do [00:29:00] experimentation as you went on that journey. And when you have a new product manager join or maybe somebody who's less experimentation-oriented, how do you coach that product manager on how to think about experimentation and when to apply it and things like that?

Luis Trindade: Quick answer: I throw them to the lions. Jokes aside, it's actually what I tend to do. I'm assuming it's a product person who already has some product background, but each one of us has a different background in how we handle experimentation. Some are more tactical.

They run lots of A/B tests but follow those practices very blindly. I'm a believer that experimentation is more holistic as a way of thinking and not necessarily a tool that you can just use. We all have a big tool belt that we should be able to choose from. So whenever we onboard somebody [00:30:00] new, we have regular onboarding sessions. So we have lots of training material that we have been gathering along these times, and we keep refreshing it. That's the beginning, but it doesn't stop there.

Because you go to a lesson and you hear about it, especially at the beginning when you're a new joiner, you are overwhelmed with information. So those are just notes for you to be able to - how I see it is just notes so that whenever I hear about that thing, that's where I should go to look for it. But the throwing-to-the-lions joke is the reality, because as quickly as they onboard, the first question I challenge them with is, "In a couple of weeks, bring me what is your key strategy for your area as a product manager. You will be leading a certain area. What are your strategy metrics to measure success for that area? [00:31:00] How are you going to interconnect with your colleagues?" And then asking, "What is your first hypothesis that makes sense for you to improve? Because as an external person, you will certainly have experience using our tools. So what is your first hypothesis on how to improve it somehow?" And that first experiment is what I mean about throwing them to the lions. I challenge them immediately to start looking for it and attending all those regular weekly sessions with all the other PMs. At the beginning, of course, being a fly on the wall, just listening and observing. That becomes an ongoing training.

Even the more senior and the more junior persons attend those sessions, and that's what creates this cadence of knowledge between all of them, and what helps them onboard and become part of those [00:32:00] ceremonies and using the right tools from the right people.

Ashley Stirrup: So interesting to hear you talk about that. It's almost the group teaching itself and learning and building new skills. A new person comes along, you throw them in the deep end and say, "Come up with an experiment," and then they're learning by doing and learning by listening to everybody else's experiments at the same time.

Luis Trindade: Totally.

Ashley Stirrup: So how do you see experimentation evolving at Farfetch?

Luis Trindade: I think it's an interesting question because I don't see it just applying to Farfetch. We can even explore this question and the next follow-up. But how I see experimentation evolving in the world, I think we are in a phase of lots of changes.

AI is part of all of our lives nowadays, and of course it's involved in parts of experimentation, and many of my colleagues, even in the industry, [00:33:00] we have lots of chats about it: AI can automatically create hypotheses, render tests very quickly, even use synthetic users to validate a hypothesis without putting it live. So there is a bit of a feeling that doom is coming for experimentation. And at the same time, there is also the economic perspective, which is of course, all companies are shrinking and holding themselves because of all these wars and all this uncertainty. And of course, the first place any business decision maker cuts resources is anything a bit more, let's call it, experimental. Even if, in my personal opinion, it's core for a tech company to keep that light on. In our case, even with us shrinking our capabilities to evolve the platform, [00:34:00] I don't see a problem with it. We are able, even with a very small team, to keep evolving it steadily, and the number of experiments keeps growing, as does their complexity and the platform. In terms of the ceremonies, it's just a question of adapting them to the new reality. Challenging and presenting different tools to those product managers: now you are able to do the entire product discovery yourself instead of taking lots of time from a research team.

Or to complement what they are planning, and take advantage of the research team for deeper dives instead of the generic things that are now on your plate directly - creating pilots and validating many of those things quickly, almost by yourself, without spending engineering resources, and focusing those resources on scaling the ones that work. [00:35:00] So testing all those opportunities - that's what changes in terms of mindset, and that's what we are taking advantage of. So we have a more iterative approach in many internal processes. Where we previously had to put lots of human resources into executing something very quickly to take advantage of a business opportunity, nowadays we can automate most of those processes very quickly and efficiently.

So that's great. Let's move those engineers to what actually drives more value. It's never about reducing human resources. It's all about making sure that we put them working on what drives more value.

That's where I'm seeing the industry going. And here at Farfetch, that's where we are putting our efforts: if we had an entire team of customer support just [00:36:00] doing translations, let's put them on something more valuable, which is answering the actual clients' needs, and the translation is automatically handled with a tool. Those are the types of things where I'm seeing it evolve, but always with this experimental mindset: any evolution process should always be driven by a hypothesis. If we formulate it first as a hypothesis and then validate it, we have much stronger insights, and from there conclusions we can turn into decisions for the company. And even on those opportunities where we want to revamp a business area because we feel there is an automation opportunity, let's put it as an experiment. Let's define it as an experiment. Let's track it, all together with the right definition of success, [00:37:00] making sure that we look at all the metrics from a holistic perspective, not just winning rate. And that's what we are seeing, and that's where I'm seeing the future here at Farfetch, but overall in the industry of experimentation.

AI is here as a huge opportunity and we should be taking advantage of it.

Ashley Stirrup: I think you've touched on a lot of really important points, because for sure AI is going to empower people to do a lot more. But how do you accelerate learning? A lot of the cultural things are going to become even more important. People are going to be able to do more things on their own.

Obviously AI will be able to automate a lot of things, but the learning part - the cultural part, the sharing, the jointly looking at things - is actually going to become more and more important.

Luis Trindade: Even more so, because with AI nowadays I don't have a fully automated process in my case. And I feel that if I look to any of my colleagues, especially on the product area, [00:38:00] each one has their own setup of tools, processes, skills and gems.

And one of the things I see as a great opportunity is taking advantage of that and creating corporate knowledge. How can we make sure those tools are used in a way that doesn't create the silos that exist nowadays? The context of everything I'm doing is just on my account, or on my PC. So we definitely need to invest more in having that corporate knowledge.

Ashley Stirrup: I think that's a really important point. You hear that a lot of engineers are finding their jobs a little more lonely, because now they're managing 10 agents instead of working with their colleagues. There is potential for creating a lot of silos, so creating that central repository matters.

Luis Trindade: The cultural part needs to be the glue for all of them, [00:39:00] more than ever.

Ashley Stirrup: Luis, thank you so much for joining today's episode. You covered a lot of great ground. It sounds like you've got a fabulous business and you're doing a lot to innovate there. So it was really exciting having you on the show. Thank you so much.

Luis Trindade: Thank you so much for having me. It was a pleasure and super fun.