Transcript

Andy (00:00) The thing I often say is, identify the riskiest assumption you want to test, not the most comfortable one, because I think it's instinctive to go test the safe thing because we can test it quickly. We can get a cleaner metric from it.

Andy (00:13) how quickly and cheaply can you test the riskiest assumption because that's the thing that's load bearing. That's what the bet ultimately rests on. And so everything can follow that.

Ashley Stirrup (00:55) Hello and welcome to today's episode. I'm excited to have Andy Denning, Director of Product at CV Library. Welcome to the show, Andy.

Andy (01:03) Great to be here. Thanks for having me.

Ashley Stirrup (01:06) let's start by you telling us a little bit about CV Library. It's a pretty interesting company.

Andy (01:11) Yeah, absolutely. CV library's been around now for about 25 years. It's predominantly based in the UK. Fundamentally, it's a marketplace for job seekers. So it operates both sides of the marketplace. It has a space for job seekers to go look for jobs, learn more about companies and apply. And then on the flip side, it has a space for our partners who are recruiters, agencies.

Andy (01:36) or direct employers to come look for candidates, but also post jobs. And then in the last few years, obviously, we've made a push as well to try and enter the US market and take what we have and offer it into the market there.

Ashley Stirrup (01:49) Yeah, super interesting. can you tell us a little bit about your role?

Andy (01:53) Yeah, so in the product role, a lot of what I do is what you might expect from a product function in that we own the product strategy. We work with the parts of the business across sales and marketing to really understand our customers on both sides of that marketplace and just build a better product for them. It gets them closer to the job seeker that provides the right context and information for job seekers to have confidence in what they're doing.

Andy (02:17) And then obviously more so in the last few years, we've been pushing into how we can utilize AI with things like better job matching when you sign up to the platform, right down to things like understanding where individuals might be authorized to work in a particular country, state, or a certain qualification as well, which is really just taking a lot of the pain and hassle out of the job seeking experience. So everything that we do is really understanding the problem.

Andy (02:43) gathering the evidence to know that we have enough to move forward and then testing, learning and iterating about the solutions that we're delivering, learning about them, feeding back into that loop.

Ashley Stirrup (02:52) Yeah, and how important has A/B testing been within that kind of mix of things you're doing?

Andy (02:58) It's been really critical because I think one of the things that I find really, important is, I guess, moving away from a mechanism and methodology that historically the business has looked at how many tests can we run. It's got a massive appetite for learning and wanting to test. But obviously with the way that the industry's changed, with things like AI, the ability to run tests at pace at a high volume has really

Andy (03:24) being compressed. And I think one of the most important things for us as a business has been really about understanding which decisions that we want to make, which things are worth testing, which things are worth learning from and where we can make decisions that we don't need to test or we can just go ahead and build things. So that's been a learning for us as a business. We punch above our weight in the industry.

Andy (03:47) in that we operate around number three in the market, but we have a much smaller organization, a much smaller footprint than a lot of our competitors. And that means the decisions that we make are probably even more important because of our resources that we have. So there's a massive appetite to want to test and learn and improve what we're offering. But the decisions behind it for me are the thing that's been the most important learning as we've gone.

Andy (04:10) and where we're actually making decisions about what we're testing and being more thoughtful about it.

Ashley Stirrup (04:15) Yeah, so it sounds like you're trying to implement strategies that help you learn more from the tests you choose to run.

Andy (04:21) Absolutely. I think what I've seen more generally in the industry is a bit of an index now towards test velocity. More experiments means more learning. But obviously the risks with that is that underpowered tests, badly chosen tests actually can manufacture a false confidence in things which often can be worse than actually running nothing. So for me, a lot of what I've thought about

Andy (04:45) is how experimentation is providing an element of organizational honesty for what we do, especially in smaller organizations. You have a lot of opinions and voices in the room. And I think experimentation outside of tooling actually forces people to say what they believe before the data and holds them to it and holds the business to the decisions that it wants to make. And it's a really powerful way, especially in fast moving, scaling, smaller organizations.

Andy (05:12) to move away from the decision by opinion, the decision by seniority. So I think that's one of the things that we've put in as well as just our pure capability to actually run experiments and the tooling move from things like amplitude through to post-hoc. And then I think as well, the feedback loop is the biggest piece for us. One of the things we found when we started experimenting was

Andy (05:33) Our capability to make a decision was fine. Our capability to set up and run an experiment was fine. But where we lacked some robustness, for want of a better word, was actually taking the learning from an experiment and doing something with it rather than it just being a bit of a tick box exercise.

Ashley Stirrup (05:49) Yeah, yeah. Super interesting. and how long have you been doing A/B testing?

Andy (05:55) weirdly enough, when I started in product, I actually started in Deloitte in the consultancy world when they were setting up their digital practice in the UK here. I'm not going to say how many years ago, was a number of them. And I got involved in a prototyping software called IRISE, which I don't think exists anymore, but it was a of

Andy (06:17) when things like Balsamic was coming out and

Andy (06:20) how those things evolved. And so I actually started at that point using that prototyping software to A/B test. That was very rudimentary at the time. It was putting different prototypes and different interactions in front of some of the clients that we were working with. So I started AV testing many, many years ago and probably didn't quite realize what it was. I think it was more as I moved into spaces like odds checker as we grew that business and beyond that I started actually

Andy (06:47) Using tooling and things like Optimizely at the time so it was probably around that time that I really started using A/B testing properly so maybe maybe the last maybe eight eight years eight 10 years or so.

Ashley Stirrup (06:58) Interesting. And how is experimentation structured at CV library? Do you have like a central team, central data scientists or data scientists kind of put into each product team? How does that work?

Andy (07:10) So

Andy (07:11) set up's probably a little bit more traditional in that we do have a data team that work and they will run a number of experiments themselves. But we're a little bit more value stream aligned, think the tribe type of model. So we'll have a cross functional team basically with product through to engineering and design. And they'll own the experiments that they want to run to test their hypotheses in those spaces.

Andy (07:38) And then they'll reach out for support from the data team where it might be about checking that, for example, they've understood the right baseline rate, that they've got a segment size that's appropriate based on things like weekly traffic. So they'll lean on the data team where needed, but we've tried to ensure that they have as few dependencies as possible. And obviously now in things like post-hoc, it will give you a huge amount of help.

Andy (08:01) with the tooling and sample sizing and significance and things like that. it's largely a concept to cache type of model in that

Ashley Stirrup (08:08) Yeah.

Andy (08:09) the team can run their experiment end to end.

Ashley Stirrup (08:13) Yeah, got it. I think

Ashley Stirrup (08:14) One of the most important parts of any good experimentation program is the kind of culture building and in particular I see two areas where you know, people can really make an impact. One is

Andy (08:24) Mm.

Ashley Stirrup (08:25) on the experiment design review side and kind of doing that in a more public way. And another is

Andy (08:31) Hmm

Ashley Stirrup (08:31) sharing results and you know, making sure that other teams benefit from the learnings, you know, one team gets a learning, how do you make sure that other teams benefit from that? Are you doing anything

Andy (08:41) Mm-hmm.

Ashley Stirrup (08:41) in those kinds of areas?

Andy (08:43) We have been, I'd absolutely echo your point. I think it's universal in a business that those learnings are really critical because I've seen examples where you share those learnings outside of just the product or through to engineering, sales and marketing. And the number of eyes that lights up the number of ears that prick when they hear a certain piece of information that either is just something that they didn't understand or know about.

Andy (09:08) or in some cases actually creates an opportunity to go after an additional piece of value, change something for the better, because it brings the information into a room where it hadn't previously existed. I think it's something that I know I can do better on because a lot of it is more of a, there's a before and after, we're going to run an experiment and then here are the results. And I think there's a space in the middle where

Andy (09:32) the design of the experiment can be improved based on better collaboration versus sharing an experiment outcome, say, with the sales team and it being a bit of a done and dusted exercise. Something

Ashley Stirrup (09:43) Yeah.

Andy (09:43) I used to do many years ago and I've tried to keep doing in any of the roles I've done is popcorn sessions where we will take the session recordings from experiments that we've run. And in these cases, they're sometimes moderated or unmoderated tests.

Andy (09:58) And we'll put everyone in the room just to watch these recordings because it creates a closeness to the customer. But it is exactly as you said, it's an opportunity for people to see what tests are being run and what we're learning from it. But I do think the collaboration on how we design them and having a wider set of first-class inputs to that from marketing, sales, operations, whoever it might be.

Andy (10:22) is actually a really great opportunity to get better refined on the question that we're trying to answer in that that experiment.

Ashley Stirrup (10:31) Yeah, yeah. Also just okay, I wanna move this metric, whatever that metric is, but I know

Andy (10:38) Mm-hmm.

Ashley Stirrup (10:38) that this experiment's gonna start by moving this other metric, and so let's make sure

Andy (10:41) Mm-hmm.

Ashley Stirrup (10:42) we have a little bit of a chain, right? So that you understand the user flow and maybe it moved metric one and two, but not three. And so then that tells you kind of where to iterate, things like that. So yeah.

Andy (10:53) Yeah, I think that's actually really pertinent, especially when you look at regulated or compliance heavy industries, not only that who's in the room when you have that conversation, but those supplementary, whether they're secondary metrics or guardrail metrics,

Andy (11:07) defining those upfront, because baking in what else you might impact and where you have to a threshold on something that you don't want to impact.

Andy (11:17) in a negative way or where you have to break it down by segment rather than just the aggregate. Actually, it's something that can be easily missed when you get a bit blinkered on. I've got an experiment and I've got a particular metric I want to prove or disprove a hypothesis on. And you don't see the downstream blast radius of the things that you might be doing. So A, having those people in the room without it being

Andy (11:40) by consensus, think is important. And I think more so in regulated

Ashley Stirrup (11:42) Yeah.

Andy (11:43) spaces where you just have to be conscious of rules and guidelines that you just cannot influence in a negative way and how you have to bake those into the experiments as well.

Ashley Stirrup (11:53) Yeah. And has your needs around experimentation changed as you've tried to expand more in the US market?

Andy (12:00) Interesting question. other than just the fact that we now have a different market with a variance of product in terms of the experience, what it looks like, what it feels like based on a different needs statement. I don't think our core needs in terms of experimentation have changed, but it just brings that additional complexity of

Ashley Stirrup (12:22) Yeah.

Andy (12:23) when When we run an experiment, we need to understand, we trying to prove something in the UK, the US, both? What does that mean? What does that change in terms of the segments we might be targeting, the traffic we expect to see coming through an experiment, how long we need to run it, being able to break it down by the different audiences. So the need's still there in that we want to validate things as part of releasing them. I think the biggest change is recognizing that

Andy (12:51) not everything that we do is going to land in both markets. And so we will test things separately by market and then try and roll it out in the in the additional market if successful

Ashley Stirrup (13:02) Yeah.

Andy (13:02) to see whether it also works there. So that market variance is probably the biggest change in what we've needed to accommodate.

Ashley Stirrup (13:10) Yeah, yeah, that all makes sense. you could see that you might have more first time users in the US since you're expanding there and that

Andy (13:16) Mm-hmm.

Ashley Stirrup (13:17) you might have more power, you know, i in in the UK, things like that. So

Andy (13:22) Exactly. That's a really great point. We've often struggled in the US because of the throughput of traffic that we have on our platform, whereas in the UK, it's never been a problem. So we've had to design things slightly differently based on where we're starting the test.

Ashley Stirrup (13:38) Yeah, yeah, super interesting. Do you have an example of an experiment you've run where you've had a lot of learnings, maybe something that was really surprising?

Andy (13:45) Yeah, recently we've been running a lot of tests on our registration and first apply flow specifically in the US because it has a different commercial model than the UK. And so the way that we've approached how we monetize has needed to be different. And while there's been a lot of assumptions in the behavioral patterns of job seekers is the same.

Andy (14:07) they're all looking for jobs. It was something that we proved to be relatively untrue to some regard and the things that were common didn't really surprise us, but we didn't wait as heavily as we could. So to give you an example, one of the challenges we have is that we're a little known brand in the US. We don't have really any brand attribution or traction with the consumer base yet.

Andy (14:30) And so when we first started pushing into the market, we didn't see great conversion or traction in the job seeker base. And one of our assumptions that we wanted to test was whether this was down to the fact that they didn't know us as a brand or one of the problems being that in the US market, what you see is a lot of job applications that will make candidates jump through a few hoops before they actually get to the application.

Andy (14:55) We took that latter part as what we thought to be the problem based on our own just manual review of the journeys and what we could see in the data. Basically the problem being they have to jump through loads of sites to get here and they're not convinced that they're actually applying to who they're applying to when they land. For example, they might be applying to Dollar General say, and it's not clear that they are. They think they're just going to be asked to register to another site and bumped on to another landing page.

Andy (15:20) And so we ran an experiment with our headless CMS to basically brand the experience more heavily towards the brand of the actual recruiter in that instance, thinking that would actually change the dynamic and they'd be more likely to apply. So we created kind of a branded, Reg and apply journey, tested it on a few brands relatively quickly, probably ran the test within about a week.

Andy (15:42) But I actually found that it had little to no and in some cases a negative impact, but enough within variance suggests basically didn't have a positive impact by adding that

Ashley Stirrup (15:50) Yeah.

Andy (15:50) additional branding. What we subsequently found, which was I guess our pivot hypothesis was that we needed to push more into the supplementary information on a job listing, i.e. hasn't got a salary, does it have information about the culture of the business, the benefits?

Andy (16:08) that it offers. So alongside running our tests, we were also running some surveys, which helped us with that. But it was interesting because everything that we'd seen suggested that the lack of visibility about the brand, the lack of trust in this is what I'm actually applying to, was a key reason as to why job seekers were not applying. But having run the test, we saw little to no change in the metrics to suggest that actually doing that had any positive impact.

Ashley Stirrup (16:46) Yeah, super interesting. that's a great example where, intuitively both of those things, I care more if I feel better about the brand I'm applying to. And of course I also want to know more about the job. and

Andy (16:57) Yeah.

Ashley Stirrup (16:58) it's interesting how you kinda went on that journey to, you know, learn which one mattered more and, how it impacted the whole process. Yeah.

Andy (17:04) Yeah,

Andy (17:05) it's interesting you said that because I think it comes back to maybe the piece that for me is important, which is that judgment layer, which is, it worth running the test? For me,

Ashley Stirrup (17:13) Mm-hmm.

Andy (17:14) I'd look at that and say, is the outcome uncertain, say with that branding and in retrospect, we probably already knew adding the branding is just a good thing anyway, but will it actually

Ashley Stirrup (17:24) Yeah.

Andy (17:25) make a difference? So when I look back, the judgment layer before that, might have changed our decision to say,

Andy (17:31) do the branding anyway, it's just a positive step, even if it has little to no impact, we need to get to the point at which putting things like the salary

Andy (17:40) and benefits and so forth are visible and available. For us, it was about the speed to learning and the branding was a much, much quicker test to run. But looking back over at

Ashley Stirrup (17:49) Yeah.

Andy (17:50) that judgment layer piece probably should have lent us towards it's a wasted test because we probably going to do it anyway.

Ashley Stirrup (17:57) Yeah. Although I'd argue that there's some great examples. Ronnie Kohavi's super well known in the experimentation

Andy (18:03) Hmm.

Ashley Stirrup (18:03) industry and he has some great stories from Microsoft. one of the ones I love is where

Andy (18:07) Mm-hmm.

Ashley Stirrup (18:08) they actually tested slowing down the Bing search experience just a little bit to see the revenue impact. And it

Andy (18:13) Yeah.

Ashley Stirrup (18:15) basically taught them that wow, speed is super important and we should go like significantly increase the size of the performance engineering team.

Andy (18:24) Yeah, yeah.

Ashley Stirrup (18:24) And I could see the same thing here that like you're learning about okay, how much should we portray the brand and when should we pr portray the

Andy (18:32) Mm-hmm.

Ashley Stirrup (18:32) brand? You can still learn those things, even though yes, of course

Andy (18:34) Yeah.

Ashley Stirrup (18:34) you wanna do more, but how much more? Like if it had turned into a huge winner, you probably

Andy (18:39) Yeah.

Ashley Stirrup (18:39) would have said, Okay, where else can we do this? Or how else do we lead with our brands more, right? So

Andy (18:45) Yeah, no, totally fair. I think that just points to always the value in having a mechanism to run the test.

Ashley Stirrup (19:03) let's say you started working with a new product manager and they had

Andy (19:06) Mm-hmm.

Ashley Stirrup (19:06) a feature they were all excited about and were just ready to prove it was gonna be a winner.

Andy (19:10) Mm-hmm.

Ashley Stirrup (19:11) And you wanted to help guide them to think about, well, not every test is a winner and maybe there's gonna be opportunity to iterate. Let's make sure that we design this test so if it doesn't work, we still get a lot of learning and we know whether to keep iterating or move on to the next feature and how to iterate, stuff like that. Do you have any advice on how to think about experiment design?

Andy (19:32) I think that the first thing outside of those judgment pieces that I mentioned, is it definitely something we should test on the fact that it's a proper decision to be made? Would we do it anyway? Is narrowing in on what your riskiest assumption is to define the test. So often when

Andy (19:49) we'll take something off of the roadmap. There's an initiative and it can be broken down in lots of different ways. And especially as businesses are trying to move at pace. The

Andy (19:57) thing I often say is, identify the riskiest assumption you want to test, not the most comfortable one, because I think it's instinctive to go test the safe thing because we can test it quickly. We can get a cleaner metric from it. But one of the things I'll always tell my teams from a product perspective is,

Andy (20:16) is how quickly and cheaply can you test the riskiest assumption because that's the thing that's load bearing. That's what the bet ultimately rests on. And so everything that can follow that.

Andy (20:25) So the first thing I'd be asking is, have you identified what your riskiest assumption is? And do you know the quickest and cheapest possible way to test it? I think having got to that point, one thing that outside of the basics of

Andy (20:39) running some of those back of envelope type of checks, i.e. do you have the existing baseline in place? i.e. have you got something to compare it to? Once you have actually run the test, do you have a view of how long it's going to take? Because you could argue if you run the numbers in terms of your traffic and significance and it says, great, this test is going to take six weeks to run, there could be an argument of stop, pick a different method, find a different way to run this test.

Andy (21:05) because are you going to get the latitude to actually be able to run a six week test?

Ashley Stirrup (21:10) Yeah.

Andy (21:11) I think then also just especially an environment which is probably different for different people is I've operated in direct to consumer versus B2B environments and just being aware of how those different environments change the test that you're going to run.

Andy (21:27) Obviously in direct to consumer, you probably got high volume. You've got traffic to run the test. You've got the volume to get clean metrics. Obviously in a more B2B environment, you often don't necessarily have that volume available. And so it's really about what your population can support and are you going to try and do it at the aggregate or are you going to do it at the account level within it? So there's maybe a piece of advice there about understanding

Andy (21:52) what environment you sit in rather than just all tests are equal in terms of their approach. Can you run the numbers upfront to understand how long this test might take because that's going to buy you the latitude to run it if it takes a shorter amount of time versus it has to be a really high quality, well-documented and justified case if it's going to run for weeks and weeks

Andy (22:13) And I think always have a pivot point and a plan B, right, of

Andy (22:16) If those things don't work out, how would we still go test this? Let's not just not go test it. Let's find a different swap that we can we can take to actually go still still test this.

Ashley Stirrup (22:27) Yeah. Yeah, I really liked your point about the the kind of the riskiest metric. I

Ashley Stirrup (22:31) had a guest on a little while ago and he talked about for for the higher risk tests that they were doing, doing a pre mortem, okay, let's pretend this failed and

Andy (22:39) Mm-hmm. Yeah.

Ashley Stirrup (22:41) you know, what do we do next type of thing and what are all the different ways this could fail? And that that's

Andy (22:43) Mm-hmm. Mm-hmm. Mm-hmm.

Ashley Stirrup (22:46) a little bit along the same lines. So

Andy (22:49) Absolutely. I mean, you're like with any artifact that we create before, all the basics need to be in place. And the thing I'd add maybe to how all this failed is those non-goals. What are the things we explicitly don't want to try and test as part of

Ashley Stirrup (23:03) Yeah.

Andy (23:03) it? Because that's where the drift turns and you do end up with the experiments that grow the arms and legs and they become over complicated.

Ashley Stirrup (23:11) Yeah. Also just the trade-offs. a great example was a Philadelphia Inquirer and they have a a

Andy (23:16) Hmm

Ashley Stirrup (23:17) team that wants to sell ads and then they have a product team that wants to create a great online experience for customers and sometimes the ad team wants more ads than the content team wants to have on a individual page.

Andy (23:27) Yeah.

Ashley Stirrup (23:29) And so yeah, experimentation can be a great way for everybody to come together and say, Okay, well, if we get this much more advertising revenue and we still see engagement at this level, then it's a winner and it's a way to kind of create a common framework for evaluating whether an experiment's truly a winner or not. Yeah,

Andy (23:47) Yeah, absolutely. Absolutely.

Ashley Stirrup (23:49) yeah. And as you look forward for CV library, what are the biggest opportunities you think in terms of where it might evolve?

Andy (23:57) Mm-hmm

Andy (23:59) it's hard to ignore the AI piece around this.

Andy (24:02) I think most product teams, most organizations, most engineering teams are in some form of AI reshape for what they do. Either they've started from AI or they're trying to find a way to either bolt it in or ideally kind of reset and reshape off the back of AI.

Andy (24:21) as much as it unlocks a huge amount more generally, but specifically with things like testing in terms of how many variants you can generate and the economics of it, right? And that you can create almost endless variants incredibly quickly at an almost zero cost. The piece that just more generally I see is that piece around judgment that says,

Andy (24:44) The bottleneck in that world no longer is can we identify experiments? Can we create experiments? But it's about the quality of the experiments that we're running and have we got enough traffic to run and get quality outputs from those experiments. So for me, think the shift, like with a lot of things now with AI specifically in the product world is no longer about

Andy (25:06) the throughput, right? You hear about businesses in the Amazon world of they're running hundreds of experiments a day. And the reality is the 99 % of us are not in that type of environment. But

Ashley Stirrup (25:16) Yeah.

Andy (25:16) AI allows us to think we can be. And I think it's that point about AI makes us think that we can now run a huge volume and a huge throughput of experiments. But the question, the hypothesis, the problem that we're trying to

Andy (25:34) to solve and to your point how we make the trade-offs on which ones we do and don't take forward. That for me is the really important part because it's very, very easy now to flood a backlog, to flood a roadmap with just low conviction tests because we can run them.

Ashley Stirrup (25:53) Yeah.

Andy (25:54) So for me, the discipline and guess the piece I'm going to be interested to see for us and just more generally.

Andy (26:00) is how teams get better at understanding the hypothesis, understanding the problem, the evidence they have to support that hypothesis and the way they want to test it and feed that back in versus can they actually just run the test and can they run it quickly and can they create lots of variants? For me, that's a way to kind of focus on a volume challenge over quality.

Andy (26:22) challenge.

Ashley Stirrup (26:22) Yeah.

Andy (26:23) So I think for me the ability to know the hypothesis, be confident about learning from it, matters far more now than it ever has.

Ashley Stirrup (26:32) Yeah, I think that is so dead on. you really want to think about it not from a how many experiments am I running, but how much my learning perspective. And I do think you

Andy (26:37) Exactly.

Ashley Stirrup (26:41) want to take a portfolio approach because you could always have a surprise, right? I had a guest

Andy (26:45) Mm.

Ashley Stirrup (26:45) on and he was talking about how they had a prompt and it had seven steps and they actually had a typo

Andy (26:50) Mm-hmm.

Ashley Stirrup (26:51) in it where they had two steps four, and then step six said go back to step four in the prompt.

Ashley Stirrup (26:57) And so well, obviously we should fix that, right?

Ashley Stirrup (27:00) But they they just had a philosophy of testing everything. And so they fixed the prompt and it actually performed worse. so

Andy (27:07) Really?

Ashley Stirrup (27:08) obviously a quirk of AI and something to keep an eye on. But

Andy (27:10) Yeah.

Ashley Stirrup (27:12) you know, you do get surprising results. And so maybe,

Andy (27:14) Yeah. Yeah.

Ashley Stirrup (27:16) you know, try to run as many experiments as you can, but have low friction experiments for a lot of things where it's really kind almost more do no harm type of testing. And then you have

Andy (27:25) Mm-hmm. Mm-hmm.

Ashley Stirrup (27:26) the ones where you're like, no, I really want to think about the hypothesis.

Ashley Stirrup (27:28) Hypothesis and experiment design, and all that is where I want to try to maximize learning. So you get a little bit of the best of both worlds there.

Andy (27:36) Yeah, absolutely. And the way that a lot of those things get commoditized is, especially with AI is, where do you spend that effort? Where do you spend that human judgment? It's on those riskier things. It's on those things that there is more learning versus using what is now commoditized to be able to push on those say lower risk type of experiments. It's always

Ashley Stirrup (27:58) Yeah.

Andy (27:59) now a challenge of where does the human judgment get spent? Where do we think a little bit more?

Andy (28:04) Deliberately about what we do versus where can we just push things through quickly. And I think it comes back for me to that riskiest assumption. That's where we spend the time. The other things yet we can move at pace. There's lower risk. So we can push things through quicker at a higher pace and learn from them after the fact versus let's go spend the time on the riskier assumptions.

Ashley Stirrup (28:25) Yeah.

Ashley Stirrup (28:26) So I think, the the whole job seeking slash recruiting industry. if you think about industries and where you'd stack rank them in terms of opportunity for disruption from AI, I would guess

Andy (28:36) Hmm

Ashley Stirrup (28:37) it's pretty high here, right? nobody likes looking for a job. You end up applying for all these jobs you never hear back. As a

Andy (28:39) Absolutely. Yeah. Yeah.

Ashley Stirrup (28:44) hiring manager, you review all these resumes that are just terrible fit, like, you know, why

Andy (28:47) Mm-hmm. Mm-hmm.

Ashley Stirrup (28:49) why is the system showing me these?

Andy (28:51) Yeah.

Ashley Stirrup (28:52) are there certain things that

Ashley Stirrup (28:53) You're you're thinking about a at CV library around AI?

Andy (28:57) Yeah, it's interesting you say that because I think you're absolutely spot on. in fact, that AI has already done that to some extent. Everyone just will probably have interacted with the Indeeds and the Linkedins of the world. And the problem with AI now is what it's done is it's allowed a flood of applications through various tools

Ashley Stirrup (29:15) Yes.

Andy (29:16) that allow you to just apply. It's an apply tsunami, someone maybe a couple of weeks said. And that puts recruiters and employees in a very difficult position because

Andy (29:26) They've got a thousand applications and there's no way they're going to be able to get through them. And it creates friction on both sides because then it doesn't feel like as a job seeker, you're going to be able to be heard in

Ashley Stirrup (29:38) Yes.

Andy (29:38) that amount of noise. And then it's a waste of time, but it's also frustrating because you don't hear back. But on the flip side, how do I get through that volume? How do I find a good candidate? So I think that in this industry, that problem does need to be solved.

Andy (29:53) but what it's done is it's put pressure on the pipeline further down the stream and that's in the decision. And so I think

Ashley Stirrup (29:59) Yes.

Andy (30:01) what you're gonna find is it's the decision making process that's more important. There are gonna be ways to solve and you can already see it with the way some of the ATSs are able to now use AI to identify the candidates in that volume, but it's gonna be the decision making process of.

Andy (30:16) we've got a role, but have we really understood what that role needs? So how

Ashley Stirrup (30:20) Yeah.

Andy (30:21) assessment is done on the role requirements and then how assessment is done on the individuals to find the better matches in there so that employers and hiring managers can better justify and defend the decisions that they make. Because they're not going to struggle with volume. What they're

Ashley Stirrup (30:40) Right.

Andy (30:40) going to struggle with is finding the right person for the right role and have they

Andy (30:44) understood their role really about what it needs

Ashley Stirrup (30:47) Yeah.

Andy (30:47) and have they understood the candidate. And that's where there's a lot of companies now spinning up in that kind of assessment

Ashley Stirrup (30:53) Yes.

Andy (30:53) space that help with that, whether it's using AI from an interview perspective or the tests that do psychometric type testing that people can go do through to interviewing hiring managers to pull out all the things that it thinks the role needs and looking at skills assessment.

Andy (31:09) That for me is, I think, to be the space that

Ashley Stirrup (31:13) Yeah.

Andy (31:14) is starting to show signs of growing and changing.

Ashley Stirrup (31:19) Yeah, that makes a lot of sense. long ago I worked at Teleo, which is an applicant tracking system. And

Andy (31:24) Yes, yeah, yeah.

Ashley Stirrup (31:25) it's really interesting, you know, 'cause a small company, they're mostly just trying to find good candidates and hire somebody, but the big companies they need to be able to really prove that they have an unbiased process and you know, there's

Andy (31:36) Mm-hmm.

Ashley Stirrup (31:36) just a lot more requirements on that. So you can't just say, Well, AI picked these candidates. It's like, Well, was that really the best candidates I should have talked to?

Andy (31:43) Yeah.

Ashley Stirrup (31:44) Was there bias?

Ashley Stirrup (31:45) that so yeah super interesting dynamics there yeah

Andy (31:47) Yeah, absolutely. Yeah.

Ashley Stirrup (31:49) well Andy thank you so much for coming on the show I I think this is such a fascinating industry and really appreciated I always enjoy having product managers on I feel like you bring like a different perspective than you know kind of the data science

Andy (32:01) you

Ashley Stirrup (32:02) side of things right

Andy (32:03) glad that you have some products in here. And look, I've really enjoyed being here and really appreciate you guys having me on.

Ashley Stirrup (32:11) Thank you so much.

Andy (32:13) Appreciate that. Thank you.