Notable Quotes

"Sometimes if you ask a user what they want, what they tell you they want is not actually what they want."

"I heavily believe that failed tests are just as good and we should iterate or not."

"This was actually a bit of a disaster. It was not a good user experience."

"I think teams only need one or two big wins a quarter. It's just not sustainable to have a million hits."

"The tests that are less risky, for sure, have an AI run the analysis, have an AI run the test brief. But then I might question why are we running it then?"

Transcript

The Experimentation Edge - Lina Blackman ===

Ashley Stirrup: Welcome to the Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Stirrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions. This show is sponsored by GrowthBook, the open source experimentation platform leader. Now let's jump in and get started with our next guest Hello, and welcome to today's podcast. I'm excited to have Lina Blackman, head of product analytics from Squarespace on. Welcome, Lena

Lina Blackman: Thanks so much, Ashley. S- super excited to be here

Ashley Stirrup: So Lina brings a tremendous amount of great experience with us. She's been head of product analytics at Squarespace for over five years. And Lina, maybe you could walk us through your background a little [00:01:00] bit.

Lina Blackman: Yeah, sure. So I've been at Squarespace about five years, always in product analytics. Prior to Squarespace, I was doing digital analytics at iHeartRadio, so mostly mobile app, but got some exposure to some of the TV and Alexa devices too, which is a lot of fun. Streaming data is super fun. And then prior to that, I was at Bloomingdale's.com when e-commerce was just getting hot. And that's where I learned experimentation

Ashley Stirrup: Wow. Sounds great. Before we talk about experimentation at Squarespace, maybe you could talk a little bit about kind of some of the priorities you have there and just gener- j- a little bit of an overview of the Squarespace business.

Lina Blackman: Sure. So Squarespace is a website builder platform. That's what we're primarily known for, but we also have scheduling product, domains product, suite of commerce product and functionality. The website CMS is core though, so it's user coming in, trying to build up a website. They can choose [00:02:00] from templates that we have, or they can build from scratch using AI

Ashley Stirrup: And how many customers do you have approximately?

Lina Blackman: We have around three million website customers

Ashley Stirrup: So that's a lot of volume, a lot of opportunity for testing, plenty of power

Lina Blackman: For sure, yeah. We have some experiments are powered, some are underpowered, depending on the slice

Ashley Stirrup: Yes, of course. Could you tell us a little bit about experimentation there? Is it a more centralized model or a decentralized model?

Lina Blackman: So the way my team is structured is we have embedded analysts into each of the product areas. So I would say it's pretty decentralized because we view our analysts as the partner in crime or spy, analytics spy within each of the product teams. At the central level, at the analytics level, we own the experimentation program. So the repo with all the A/B tests results that are consolidated, the templates for A/B test briefs, that framework is centralized, but the analysts are owning each of their tests [00:03:00] with their product teams individually.

Ashley Stirrup: Got it. And how does your team decide what, where to focus and where the biggest opportunities are?

Lina Blackman: I would say it's a little mixed by team and business line, but for the most part, we have company goals that we go towards. They're all driven by our user problems and revenue targets, of course, and then they kinda trickle down to teams. So specific teams will own, I view it as kinda like a North Star framework or a metrics ladder leading up into all of those goals and user problems. But depending on which team you are, you own a part of that user journey, and your key goal should be optimizing that specific part of the user journey

Ashley Stirrup: Makes sense. And roughly how many experiments are you running per year?

Lina Blackman: We have around 150 to 200. It's a little bit of a moving target

Ashley Stirrup: Got it. And any areas in particular that your team's focusing on?

Lina Blackman: I think what areas are we not focusing on? Some of the areas that have been hottest in - for testing in the [00:04:00] last year is our onboarding experience, so when a new user joins, how to get, they get through that flow. We're a little newer to mobile app AB testing, which is really exciting, and the implementation of that is just a little more challenging, so that's been fun. And checkout and the pricing page have been big in the last couple years

Ashley Stirrup: Got it. And - I wouldn't think of mobile as being an important surface for a website building tool.

Lina Blackman: Yeah. It's interesting 'cause the business has gone back and forth over that over the years too. What we find is our power users like our mobile experience. And it's just a more simple UI to manage your business, so we know it's a key part in the, in our small business owners

Ashley Stirrup: Could you tell us about an where you had a lot of learnings?

Lina Blackman: So many to choose from, it's hard. But there's one that I think walks you through our core experience pretty well. So when a user joins Squarespace, they can land on squarespace.com. They can start a trial. They can either select a [00:05:00] template, or they can go through our Blueprint, Blueprint AI builder experience. Prior to all of that, what really set the stage for figuring out that we should focus on this flow was we just launched a blank template. So a user can select a beautiful designer template, or they could choose a blank white screen and just start adding things from scratch. This was actually a bit of a disaster. It was not a good user experience. We saw early signals that we were increasing the number of users, like, coming in to our CMS and all of that, but they didn't end up converting. And when we started to dig into that, why, we realized that we have two very distinct user types. Like, one that is more technical and wants the customization, and then a whole lot of users who actually need more guided a guided way to design their templates. These are, you know, non-tech s- they don't have time to be digging [00:06:00] into the HTML and custom code of their website. They just wanna select a few things and customize. , But it kinda helped us segment our users between learners and builders or tech savvy versus not. And it was a start of some of the... You'll see it now live on the site, but Blueprint AI Builder that was ki- helped the team think through all of the problems and user flows and segments that ultimately came out to a better experience, Blueprint.

Ashley Stirrup: Got it. And so did you end up looking for ways to segment the user experience so that the people that the blank template was a good fit for, that was an option, and the other people, you were steering them another way?

Lina Blackman: that design was killed forever, and I think the Which is good, 'cause I think failed experiments are just as good 'cause we were able to iterate off of it, and the iteration is really versions of blueprint or better versions of aided onboarding rather than a blank screen. I think that was too risky for us[00:07:00]

Ashley Stirrup: Got it. Got it. Makes sense. Interesting. Yeah. I think there's so many examples of where you really try to-- wanna try to understand the user experience, and then sometimes your users will say, "Oh, just give me the blank thing," or just, you know... I was, just re-recently recorded an episode with Signet Jewelers, and for them it was like, "Oh, show me all the rings," right? But, like, all the rings is not what you wanna see at Signet Jewelers. If you're shopping for engagement, you're shopping for an engagement ring, and sometimes you have to do-- you guide people in ways that, that they don't actually tell you they wanna go, if that makes sense.

Lina Blackman: Sometimes if you ask a user what they want, what they tell you they want is not actually what they want

Ashley Stirrup: Exactly. You said that a lot better than I did. So yeah, in general, how do you approach experiments that lose? How do you try to extract as much learning as possible from them?

Lina Blackman: I heavily believe that failed tests failed tests are just as good are just as good and we should iterate or not. When we do have a test where none of the variants win of course, [00:08:00] the teams get, you know, they're emotionally bought into the experience, and they're rooting for the experience. Even my analysts, who are the spies and supposed to be agnostic to the product team. The two questions I would ask is, one, are there granular segments where this experience actually worked, or, you know, is there some nuance? That's the fun analytics deep dive part of it. Two is, should we continue investing in this idea, which I think is a really fun business problem because there's a million things that a product team can be building. I think experimentation's just a good way to focus and revisit all of the user problems that you could be solving and figuring out which one's most be impactful. And then it also just helps build your knowledge, too, right? Not that you should always use historical results as, , the final say, but it builds your knowledge on this worked for the user, th- these kind of experiences don't work for the user. , So adding to that library and knowledge.

Ashley Stirrup: Yeah, I think that makes a lot of sense. Yeah it's funny [00:09:00] 'cause of course we all want nothing but winners all the time, but if that's what we got, we wouldn't need experimentation. We also wouldn't be learning that much about the user journey. I think the losses is where you learn what are people really trying to do, and is this thing I just inserted into the experience actually supporting that goal, or is it taking them in a path that's not as helpful?

Lina Blackman: Absolutely. Yeah. I think teams only need one or two big wins a quarter or whatever. It's just not, it's not sustainable to have a million, million hits

Ashley Stirrup: you try to work with product managers? I would imagine that depending on the product, some of them are gonna na- more naturally lend to experimentation 'cause you're trying to optimize an onboarding flow. And then some of them, they might be like, "I just wanna ship features and I don't need to do any of this A/B testing." And how do you help somebody like that think through how to prioritize experimentation relative to all their other priorities?

Lina Blackman: I think all of our business lines and product teams are [00:10:00] on different points of the maturity curve for experimentation. For those that are earlier on one, testing is not just about the test itself, it's also just the toolkit that comes with it. We've built analytics tooling around it. We have a very clean brief that helps us align on goals. We have the technical infra behind all of it to launch tests in a very controlled way. So - that's like the first thing I offer. Like, we have a beautiful toolkit around it. Even if you're doing a rollout, just try out the testing framework first, because it actually may end up saving you time or the headache of going back to a launch that you wanted to measure via pre/post or something like that. The second thing is also showing the wins, like being very open about, "Here's our experimentation program. Here's what's worked, what hasn't worked," , so that teams can learn from other teams. And then maybe the third thing might just be [00:11:00] sometimes it's an analyst going in and just fighting. and being really embedded in the team and understanding deeply the problems that team feels they are facing and the roadmap and all that, and just coming in with the right context at the right time. Like, "Hey, this is a good opportunity for an A/B test." And then once they get into the rhythm of one or two tests it just becomes more natural, and then we can, If they're earlier on the maturity curve for experimentation, we always try to err on the side of over-testing too, just so they get in that rhythm and build that habit with the teams.

Ashley Stirrup: That's a really great answer. I love it. So are there ways that you're trying to package up those learnings, especially helping other teams learn from each other?

Lina Blackman: Yeah, we have a few ways. One, we have a consolidated AB test log where all briefs and all results are linked. And we're constantly reminding analysts and teams, "Please keep those updated because it's so [00:12:00] helpful to be able to look back." So that's public, and anyone at the company can access it. The others, we have insights rituals where we do try to make it a point for analysts and even product managers to present results to a large format meeting regularly. And it's presenting the result, but it's also asking what did we learn about the user that could be applicable to other areas. We do that whether it's large format that is more like product wide but also smaller teams by business lines, like just talking and discussing the results and creating that buzz around the AB test results.

Ashley Stirrup: Yeah,

Lina Blackman: And if- If I were better, I would consolidate those results and present them more, but, that train falls behind a little bit

Ashley Stirrup: Yeah, of course. That gets to be a tricky thing. And so do you find that the teams that are most active in A/B testing are generating insight that the other teams can leverage?

Lina Blackman: I would like that to be more. There's there have been instances over the years of [00:13:00] teams connecting the dots, for sure. But the analysts, I view them as a kinda like glue team where they help connect the dots w- if the product owners aren't doing that. And it's hard sometimes because, uh, sometimes a team is, is like a year ahead of another team in their growth funnel. So it makes a lot of sense that not everything can be one-to-one, and sometimes it's just very specific to the product or business line that you're working on.

Ashley Stirrup: Yeah. And how do you think about kinda north star metrics and, if you listen to Ronny Kohavi, he talks a lot about the importance of getting everybody aligned around an overall evaluation criteria. That can be really hard depending on what you're trying to optimize and I'm not quite sure how this step in the process aligns with my north star metric."

Lina Blackman: We're big on the North Star metric. I don't think we have it down perfect. Why I love it is it's not the metric itself for me, it's about the KPI tree that happens. So [00:14:00] it's a simplified way of understanding revenue inputs tied to the user. And we launch this every year with our teams because we get it. Not everyone is as into the numbers as, as we are. so giving teams some guidance, like these are the three levers that you should be working towards, whether it's revenue or subscriptions or customers or, or upselling customers, whatever it is. It's just that we have three very simple ones, and teams know what they are, and then they know how they might ladder up. Um, because the job of the analyst might be connecting alongside the product manager, connecting that product area , that they own, those user flows connected to that metrics ladder that we release

Ashley Stirrup: Yeah. And I'm guessing you have a lot of metrics that you look at for each experiment. Do you kinda help people with a decision framework as they look at, "Okay this metric's up, that one's down. What do I do?"

Lina Blackman: Yeah, for sure. Part of our AB test brief is is a decision matrix [00:15:00] where an analyst and a product manager will fill in the two metrics. One might be the key metric, the other might be the counter-metric that you don't wanna hurt. And then within that matrix, it's a go forward or roll back decision or analyze further.

Ashley Stirrup: Do you find yourself sometimes saying, "Okay, let's replicate this test 'cause, maybe it's in the 5%, not the 95"? If you have a 95% confidence interval, your 5% of your tests are gonna be false positives. So every once in a while you need to go back and maybe this one's in the false positive bucket."

Lina Blackman: Yeah. So w-we definitely debate our levels of certainty all the time. I would say for tests that are our core funnel, like subscriptions, pricing tests, we accept a certain level of uncertainty. We're tighter, though, on bounds. We wanna make sure it's within the [00:16:00] 5% or whatever it is. For tests that are more engagement, though, we accept less certainty, so the bounds for that can be a lot wider, and we try to not have teams spin on that because you c- you can never be sure, and you may run into false positives or anything like that. But it should really only be if it is something that affects bookings or bottom line that we need more precision

Ashley Stirrup: Got it. That makes a lot of sense. And as we wrap up where do you see experimentation going at Squarespace?

Lina Blackman: Some of the things I'm excited for most how AI changes the AB tests workflow, of course, we're thinking of that, and we're already, experimenting a little bit. I'm curious from how it speeds up things, but also . what we'll realize in the workflow we want to do manually because it is a good forcing function for the teams. , So I think it's gonna be a, you know, like a back and forth on that and exper-experimenting within the [00:17:00] workflow. The other thing is as we expand, business lines, for example, like our scheduling business is a little bit newer to some of us, getting experimentation programs spun up for some of those business lines that are just newer along the maturity curve and taking learnings from the other teams. But AI, I'm sure everyone is saying AI, but that-that's top of mind for us for sure.

Ashley Stirrup: Yeah. Yeah, that makes a lot of sense. Are you seeing the product teams moving faster because of AI, and has that affected how you look at experimentation?

Lina Blackman: I still think it varies by team a lot. There are some teams who are definitely moving a lot faster, and there are a lot of tiger teams that are spun up now that are experimenting with AI in everything in workflow. So a little-- Some of that is wait and see and some teams are just a little behind still. We're-- I think we're seeing it's definitely gonna be faster. I think for experimentation, we gotta figure [00:18:00] out which parts of the workflow we can be more hands-off with. Like the less-- The r- tests that are less risky, for sure, have an AI run the analysis, have an AI run the test brief. But then I might question why are we running it then? So I think it's gonna be a back-and-forth conversation on what we wanna let go of and what it can optimize, and maybe it's the mundane parts of the analyst workflow too, like the tracking or assignment setup

Ashley Stirrup: Yeah. Yeah, for sure with AI at a minimum you wanna do do, a lot of do no harm testing and then maybe you put your emphasis in terms of like hypothesis definition and real thought on experiment design, you put that into the bigger bets.

Lina Blackman: Yeah, exactly. Exactly. There's still-- I think we'll still have to approach some parts of the process manually because we need to be thinking methodically through each of the steps

Ashley Stirrup: Yeah, that, that makes a lot of sense. I think we've all [00:19:00] used AI to do something, and then you go back and you look at what it wrote and you're like, "What? Wait a minute, what is it saying here?"

Lina Blackman: Hey

Ashley Stirrup: So like forcing yourself to go through the thought process maybe as you collaborate with AI or even after the fact I think that's important. , One thing we're seeing is that, some of our customers are going very heavy in AI-powered apps. So not just coding with AI, but building things on top of LLMs. And that creates some different dynamics. Sometimes it means you need new metrics to track in order to do experimentation. Are you working with LLM-powered apps and are you dealing with any of those challenges?

Lina Blackman: Internal apps or like product facing?

Ashley Stirrup: I was more thinking product facing, yeah

Lina Blackman: Product facing. We-- Yes, there are a number of initiatives planned that are getting to the... E-everybody's releasing some chatbot or something like that, right what's fascinating from the data side for that is, one, the tracking of that, like the prompt [00:20:00] quality and evaluation and all of that. The other piece is if these features are just gonna launch, for experimentation specifically, we might not... It might be more of an optimization experiment than do we roll this out or not? We know we wanna roll this out because that's what the industry is doing. But we could definitely leverage experimentation for optimizing the experience or the entry points

Ashley Stirrup: Yeah. The good news is that you're doing something relatively simple say for example, Typeform they built an AI agent to help people build customer surveys. So they're able to measure how fast are they building the survey without AI and how fast are they doing it with AI, and, how often do they actually send out the survey. So there's some very clear metrics in those cases that you can then say, "This-- did this prompt help them get there better or did this system prompt?" In the case of Khan Academy they ended up saying we're trying to help students [00:21:00] learn," and they invested a lot in creating a new metric around cognitive engagement and just, okay, is the student asking questions where they're really trying to learn or are they just trying to get through this course as quickly as possible? And so then they were able to optimize on this engagement metric. And so that's what I think's super interesting is, so often with a chatbot, did the person leave with a good experience or not, and how do you measure that? And so that I find that a fascinating topic within experimentation.

Lina Blackman: Yeah, for sure. , There's like engagement, there's quality of the output, and then there's long-term implications too. Do they come back and use that? Did they get value out of it, or were they just experimenting to mess around with the chatbot or whatever it

Ashley Stirrup: That's right. That's right. Yeah. Yeah, and that's where I think data teams can be really valuable to the rest of the org, is that sometimes an engineer will say I used another LLM to grade it," and it's like, well, maybe you didn't quite design that the [00:22:00] right way. So I think there's a lot of opportunity for collaboration there.

Lina Blackman: Yeah, for sure

Ashley Stirrup: Yeah. Lina, thank you so much for today's episode. I think you brought a lot of great concepts and I feel like I just learned a ton, so I really appreciate you coming on the show.

Lina Blackman: Thanks so much, Ashley. This was a lot of fun

Ashley Stirrup: Thank you. Bye-bye

Lina Blackman: All right, have a good one