Matt Mulvey: Transforming Customer Experience at British Airways

Episode 016 - Matt Mulvey - Transforming Customer Experience at British Airways

Gavin Bryant  00:03

Hello and welcome to the Experimentation Masters Podcast. Today I would like to welcome Matt Mulvey to the show. Matt is currently a digital experimentation specialist at British Airways. The UKs flagship airline, with revenues of approximately 3.5 billion euro. Previously Matt worked in senior conversion rate optimisation roles at Woolworths New Zealand's largest supermarket chain. He has studied computer science and information systems and has extensive experience driving experimentation program strategy, execution, and cultural transformation. Welcome to the show Matt. 

 

Matthew Mulvey  00:45

Oh, thank you very much. No, I'm happy to be here.

 

Gavin Bryant  

For our listeners, Matt has recently relocated from New Zealand to the UK to take up his new role at British Airways. We were chatting off air (waxing lyrical) about our mutual love of the live music scene. 

 

When younger, I lived and worked in London for 5 years. The thing that I miss most about my time in London, apart from being able to travel the world at will, is the live music scene. Every week I’d be out seeing world-class bands, excitedly buying the Time Out magazine each week to see which acts would be coming to London soon. 

 

My Top 3 music memories from London – (Firstly). seeing Radiohead in a tiny venue with only 300-400 others (Secondly). Going to Glastonbury Festival on my own (a whole other story) and (Thirdly). Being one of the 250,000 people that descended to Brighton for the Fat Boy Slim Big Beach Boutique 2,  and not being able to get a train back to London until 6am.

 

But, hey, that’s enough of the nostalgia from me, we’re here to talk about experimentation.

 

Gavin Bryant  00:48

Let's dive right in, Matt. If you could please provide our audience with an overview of your journey and your background with experimentation optimization?

 

Matthew Mulvey  00:59

Yeah, so I think like, pretty much everyone in the industry, I didn't start out intending to be in experimentation. It's not something you really hear about at a high school careers fair, right. So, I actually through uni, and even when I started work, I started out as a software developer. So that was kind of my initial path. And I really thought I'd stick with that for the rest of my career. But as I went through my first kind of couple of roles in the graduate program that I joined Woolworths and found myself being more and more drawn to the data side of software development, so database management, and all of those kinds of sub disciplines. And that led me eventually to more kind of data science or data analytics roles. 

 

And eventually, I joined our digital analytics team, which was also managing things like tracking along with digital analytics and experimentation, which, at the time, they were kind of doing as a little bit of a side project. But over the time that I was in the team, we really advanced in our maturity in that space, and it became more of a fully-fledged function. And then there was a point where I was kind of considering my career paths. And I was thinking about new roles, or what I wanted from my existing role. And I realized that I definitely wanted a role that included experimentation. 

 

And after that, I thought to myself, Oh, I could make this my entire role if I find the right position, which hadn't really occurred to me before, because this had kind of just been something I did on the side, which was really fun, and definitely the part of my work that I enjoyed the most. But yeah, it hadn't occurred to me that it was a full-time gig. And so, I took that thought to my manager, and very graciously, she was able to transition me into a kind of permanent experimentation role which I think I just fell in love with the function of experimentation within businesses. And that's kind of led to me to my transition to British Airways as I moved over to the UK.

 

Gavin Bryant  03:18

Let's talk a little bit more about your experience at Woolworths. You got to jump on board early then and to really work in the scaling and the growing of a function in a business. Can you describe that journey for us please?

 

Matthew Mulvey  03:35

Yeah, like I said, when I first joined the team, we did have some process, we were running some tests, and we did have some champions within the business. But I think there was very little kind of full-time dedicated support, and there was kind of pockets of adoption rather than a widespread culture. 

 

So, I think over the years that I was there, and I certainly hope, since I've left, we really saw it grow. And it was kind of almost an evolutionary growth, where we would work with one product owner and get them really on board. And they would mention it to another product owner that they were working with. And so, we get in with that team and slowly build up their maturity, or a product owner would move to a new function, so they would take them by in to their new role, but their replacement and their existing role would also get up to speed with experimentation. 

 

So, it was almost like a spread throughout the business. As I'm sure these cultural movements often develop. So that was great to see. And by the time I left, we got pretty significant buy in from quite a few teams who are getting teams up and running their own experiments with very little oversight from us as a center of excellence. So that was really great to see. And we had some really strong buy in from senior leadership as well, which I think was really key to driving that.

 

Gavin Bryant  05:08

So, thinking about some of those early adopters at Woolworths what were some of those pockets that were early on the train?

 

Matthew Mulvey  05:18

I think the number one was, we had a senior manager within the digital department who eventually became head of digital, who had previously done experimentation in one of his former roles. So, he was a true believer in the cause. Having done it himself, and was always pushing the product managers or product owners underneath him to, should we be testing that, what can we do to run more tests, kind of driving it from a top down level, which I think is really helpful for experimentation, if you don't have the senior leadership support, it's very hard to build that culture, but then especially if you're looking to grow your team and get more resource, that's going to be basically impossible, if you don't have a champion that that kind of management level. 

 

So that was pretty much invaluable. And then there were a couple of product owners who had had done experimentation in previous roles and have brought that desire over with them. And so those were kind of where we got a foothold that led us to run a few tests, and almost like case studies and showcase the results we driven to spread that message to the rest of the business. So, I think those initial footholds were really invaluable for us.

 

Gavin Bryant  06:48

Matt, what you’ve just described there is like the perfect environment for experimentation to flourish. On one hand, we have a highly motivated leader with previous experimentation experience, providing strong top-down leadership support and program sponsorship.

 

On the other hand, we have this situation at the Team / Team Member level where experimentation is naturally and organically spreading, a little like diffusion via osmosis. As Product teams experience success with experimentation, they share their learnings and benefits, creating interest and motivation from nascent teams, working in a virtuous cycle.

 

And, when these two Top Down and Bottom Up forces come together collectively, it creates the perfect environment for experimentation to grow and scale.

 

Matthew Mulvey  07:16

Yeah, that's absolutely correct. And I think that is key, because I did see specific cases where there was top-down desire, but you know, say the actual developer or the actual analyst wasn't really interested. And they kind of felt like they're being forced to do something when they didn't want to. And that was really difficult. But similarly, I've been in teams where the developers or even the product owners really want to test something. And the overall business objective is just get it out as fast as you can, and they basically can't justify testing it in that case. So, I think you do really need the like, you said that the top-down business support, but also you need to inspire that interest from the people that are gonna be working with us day to day. So yeah, it does need to come from both sides.

 

Gavin Bryant  08:07

Yeah, it's really interesting that there can be sort of negative push on two fronts there that a desire for speed and execution and implement all costs. And also there can be some motivational issues at a team level where there isn't buy in -- Yeah, an interesting case study that one. So, what were some of the key learnings that you took from that initial experience at Woolworths?

 

Matthew Mulvey  08:38

I think that the main things I learned, I would say were around kind of the stakeholder management domain. So, like, we're saying, if people feel like experimentation is a chore, or just something they need to complete on the checklist, they might do it, but it's probably going to be low quality, and they're not going to be very invested. So, one thing I learned is you need to make people excited about it, and they need to understand the value that it's providing to them.       

 

So, if a developer is building something, it is easier for them to just, you know, merge it into production and launch it and not have to worry about a two-week run and then hearing what the results are going to be. And likewise, if you have a designer or a product owner who's got an idea for a feature, it can be a bit threatening to say, hey, cool idea. Let's see whether it actually works. And if it doesn't, I'm gonna basically recommend that it go in the bin, essentially. 

 

So that can be quite threatening. So, you do need to be very mindful about how you communicate what you're doing. And that's something I didn't initially realize because I was bought into the concept of experimentation. And I was fully on board with that. And it took me a while to realize Oh, not everybody is going to be at that level of understanding and maturity. 

 

So, you need to tailor your communication or the narrative of how you're presenting your work to the, I guess, the mindset that your audience has, which I think is a really great lesson that I also learned from my time in Digital Analytics, which is you are communicating for your audience and not for yourself, which I think is an easy place to land in.

 

Gavin Bryant  10:29

You find that over time that embedding experimentation into the deployment and development process it just became a habitual way of working was that something that the business transition to over time?

 

Matthew Mulvey  10:44

Yeah, we're definitely making great progress. And that's something that really surprised me when I joined British Airways, I'll be on a call about a feature with product teams development teams. And they were just kind of say, offhand-- Oh, and of course, we need to test this. And it would just be like a five second comment in a meeting. And no one else would say anything. And I was like, Oh, okay. They just want to test it with, you know, no debate, no discussion, I don't need to convince anyone. They were all just kind of like, oh, yeah, we need to test it, because it's a big change. And it was almost like second nature for them at that point. But of course, we would test it, why would we not test it. 

 

So that was an almost interesting gearshift when I joined British Airways to see that they really had that culture embedded, at least on the teams that I was working in. And it was a nice change, to not have to kind of, you know, go through that sales pitch of experimentation and work on that internal marketing, and kind of see that the job has already been done in that sense. So that was really nice.

 

Gavin Bryant  11:55

I think that's a really good point that a different level of maturity where experimentation is the path and experimentation is the operating model. And I don't know about you but being the pied piper of experimentation and experimentation salesperson it can be a hard graft and hard work and it does take a lot of energy, especially for programs that are lower in level of maturity. Hey, just before we dive on and discuss British Airways more deeply and specifically a quick philosophical question for you. What's your strongest held belief about experimentation now?

 

Matthew Mulvey  12:36

Oh, my strongest held belief-- Well, I-- Okay, this might be a little bit controversial in the experimentation community. But I do have a mindset that I think business value and speed to action needs to come above the scientific process. And so, what I mean by that is, I've talked to a few businesses over the years. And I remember there was one business that was asking me some questions about how we were running experimentation, because they kind of saw me as running a more mature program. 

 

And so, they're looking for some advice, and they came to me and said, Hey, we see that you guys have got a ton of traffic, and you're running some really great tests, and you're reporting these results with 95% plus confidence as kind of this, the standard, we're not sure we can get to that level. And one thing that I said to them was, if your business was making the correct decision, say 80% of the time, you would still be crushing your vertical in your market. 

 

So, if you need to lower your confidence level to get a result, or if that's the best result you can get with the data that you've got, that is still incredibly valuable for your business. I think, I really believe in a hierarchy of evidence, as a tool to support decisions and an experiment that's even at 80% confidence is very, very high on that hierarchy of evidence, when it's being compared to the opinion of Dave from marketing or some VP who has just had some idea out of out of the back of his mind. So, I think there is real value in maintaining the kind of scientific rigor of an experimentation program, but you know, when you're up against the wall, and you have to make the best of the data that you've got available. I think it's more important that the business is able to act and is not you know, suffering from analysis paralysis, because I think there is real value and kind of speed of decision making so that's kind of my potentially a hot take and could lead to some problems if you go a bit overboard with it. That's what I believe.

 

Gavin Bryant  14:51

Yeah, it's a real balancing act, isn't it? And I think one of the things that we need to be mindful of, you know, I think there's two tiers of organizations that experiment, there's a top 1%, who will performing very academic controlled experiments, who have enormous traffic and can conduct more academic experimentation. And there's a lot of businesses who are still really struggling and finding their way and may not have that same level of traffic. 

 

So, I think that there needs to be a consideration between the upside and downside of performing those experiments with a lower level of power and being able to gather some data relative to no data, but also being mindful of the consequences of potential false positive risks.

 

Matthew Mulvey  15:51

Yeah, absolutely. And I think it speaks to a more general theme where if you're following experimentation influencers on LinkedIn, you can kind of see hundreds and hundreds of best practices and ways you should be running experimentation. I think if I was starting a new experimentation program, and I'm a Junior CRO, that can be really intimidating. And I could almost think, oh, I don't think I can achieve that I shouldn't run any tests, I think it's much more important that it's much better to run experimentation at a very basic level than to not run it at all. 

 

So, if you can only test on a couple of 1000 people and you're getting to 75%, 80% confidence, if your tests are not backed and research, that's still better than doing nothing. So that's, I guess my overall message is doing something is better than doing nothing. And doing something well is better than doing something badly. 

 

Gavin Byrant  16:48

Yes, doing something is better than guessing.

 

Matthew Mulvey  16:52

Yeah, exactly. Like I said, the alternative and the hierarchy of evidence is some person's opinion. So, let's try and get away from that.

 

Gavin Bryant  17:02

Amen, brother. Okay, let's shift gears now and let's dive into the British Airways experimentation journey. And I'm really excited to learn more about the transformation program that British Airways a little while ago now, they announced a very big transformation program spending more than 7 billion pounds over a number of years to really work through the entire value chain in the business end to end. So just looking to learn a little bit more at a higher level around the transformation program and how that impacts you in your work.

 

 

Matthew Mulvey  17:45

Yeah, so I think the overall theme, at least my impression, from the transformation is, we have this position as a brand that we're a premium airline. So, we see ourselves competing with someone like Air France, for example, rather than, someone like EasyJet, or the more budget low-cost airlines. And I think the sentiment that we're getting from customers, and some of the macro research we've done seems to suggest that in some cases, we're not quite showing up to that positioning, so that the whole transformation program, at least from my understanding is, how can we fulfill that promise that we're making to customers. 

 

So, moving the delivery of the service to the level that customers would expect, if we're claiming to be at a certain level of quality. So that's kind of the broader theme. There is quite a bit of investment from that in the digital estate. 

 

So, I was just talking to a friend recently, and I told him, I worked on the website for British Airways. And he said to me, oh, man, last time, I tried to make a booking, it didn't work. The time before that I tried to make a booking, it also didn't work. It was kind of just saying, Man, your website is terrible. And I kind of just had to say, Yeah, I know, I'm working on it, I'm trying. Which is a little bit embarrassing, but I think is a good opportunity for us to kind of, like you said, can transform. You know how it is we're delivering the service. 

 

The first concern of that being, can customers actually do what they're trying to do without technical challenges. I think that's kind of a foundational level of challenge for us. But yeah, it's really great to say that the business is so committed to improving the experience for our customers because obviously you're someone who works on a product day to day. If you hear testimonials like that, you do want to action them. So, it's great to see that there's very, very top-level business support behind that at such a large scale.

 

Gavin Bryant  20:09

So, your focus at the moment is largely around website and app or are you spreading intent more broader than that?

 

Matthew Mulvey  20:21

Mostly website and at the moment yeah, there is a stream of experimentation have always wanted to get into I haven't had the chance in either British Airways or Woolworths, which is how does your experimentation touch on the physical delivery of the product. So that was obviously grocery stores when I was working for Woolworths. But that could be anything from you know, checking in at the airport, dropping off your bags to the actual end flight service, or collecting your bags at the other end at British Airways. I think there is some incredible kind of opportunity there, I've seen great case studies of some direct customer company that was experimenting with its packaging, for example. But it's so much harder to run a test that touches on the physical world, than a website test. 

 

So, it's not something we've dived into yet. But I'm still holding out hope we do have really great relationships with teams that do touch on the operational and customer side of the business. So, I'm hoping we can get into that domain at some point in the future. But for now, working on the website, is the key focus for sure.

 

Gavin Bryant  21:39

It would seem that the opportunities for experimentation right throughout the customer lifecycle journey end to end are absolutely enormous. And yeah, to your point, there's everything from website and app through to arrival and check in and then boarding and post flight as well. So, it would be an amazing experience to cross all those touch points of the journey to improve it.

 

Matthew Mulvey  22:16

Yeah, I know there are teams that are doing experimentation in those areas. Like I had a case study from our sustainability team, where they were testing various different forms of packaging whether it's the utensils on your flight, or even the packaging for the meals themselves. But it doesn't seem like there's an overarching kind of experimentation leader who can coordinate between the physical side and the digital experimentation side. So, I do see glimmers of hope and pockets of experimentation work throughout the business, but yet to be unified, which is a little bit of a shame.

 

Gavin Bryant  23:02

One of the things that I was interested in understanding from you was the customer that you'd mentioned that there's potentially some customer pain around the website, what do they expect and how is that changing over time that in doing a little bit of research for this chat today, I'd read a few times in communications from the airline industry that's currently a renaissance around innovation more broadly across the industry. So just interested to understand from a customer perspective, what are those triggers or pains that need to be resolved from your end?

 

Matthew Mulvey  23:45

Yeah, totally. So, I think like I said if you have issues with site stability, or load times, those kind of underpin everything else, if those aren't right, then they need to be fixed. And I think we are doing a decent job of that at British Airways. And there's definitely a large focus on improving where we're not. But the broader industry trends are really interesting, because I think everything is almost framed in a context of COVID and the pandemic, so it's almost like BC and AD but it's before COVID. So, the trends that I've seen are that the industry obviously reset a bit during COVID, when everybody had to shut down and there was very little air travel. And so that renaissance that you talked about is, I guess the spring from that winter, where the industry is starting to open up, we're seeing an increased uptake of travel. And I think we've actually seen a little bit of a-- I guess, an overcorrection, where there was a lot of latent pent-up demand that has now been released. 

 

So, people who'd want to travel over those years have now got the opportunity to so there's probably more people traveling then might have otherwise, if they hadn't been restricted in that way. And I think that does reflect in team structures. So, I wasn't at British Airways during COVID. But I heard from friends who worked for other airlines that teams really shrunk during COVID, obviously, because there was very little revenue coming in. And as the revenue returns and the customers return, that also means the teams grow again, which means fresh blood in the business, there's more capacity, which means we're in a place where we can actually start working on some of those initiatives to delight customers, surprise customers, we're not just trying to survive anymore, which is great to see.

 

Gavin Bryant  25:53

Good point. So, taking the conversation back more specifically to experimentation and optimization how do you organize for experimentation at BA?

 

Matthew Mulvey  26:05

Yeah, so currently, we've got a center of excellence model, where we've got a core team that oversees the digital experimentation program, primarily made up of analysts or specialists. And from there, we kind of spread out throughout the business throughout the various tribes and product teams. And the goal there is relatively as embedded as we can. 

 

So, I'll contrast this a little bit to some of the ways we were working at Woolworths. We were a little bit less specialized at Woolworths. And what that meant was an experiment from a certain team couldn't be picked up by any of the analysts in the squad or the specialists in the squad. And that was great, because it means the work is very modular, and very transferable. But I will say what I've noticed, and the more embedded model a BA is, you do build much deeper relationships. So, for instance, any of the squads that I work with at the moment, all of their tests are going to come to me, they're never going to get to anyone else unless I'm on leave, for instance, because I'm kind of their key point of contact. And that also frees me up to get more involved with them. 

 

So, I can go to say, their sprint reviews, or you know, their retros. And I can almost function like I'm not gonna say a fully-fledged member of the squad, because I'm not there full time. But I can at least be a floating resource that they know, they can call on anytime. And they don't have to wonder, Oh, who should I talk to you from the experimentation team. It's always just mad, because, you know, I'm responsible for the area. So, I do really like that function, I guess it's almost a hub and spoke kind of model where we've got a central team, but we spread out amongst the business.

 

Gavin Bryant  28:01

Yeah, interesting that, you know, when you're spending more time within a specific domain that you can then start to develop a much deeper domain expertise in that particular area, I guess it's a little bit similar to a SaaS model where you would have squads and teams’ architecture around acquisition for example, or retention. So, it just allows you a much deeper focus to solve customer problems more effectively.

 

Matthew Mulvey  28:30

Yeah, exactly. And it was kind of funny, I was covering for one of my coworkers who's on annual leave recently. And I looked at the dashboard that he had for a certain experiment, and I just looked at it and I thought, I don't know what any of these acronyms mean, like these, these are all metrics that are, you know, completely foreign to me, I have no idea what any of this is. And I think it just shows like a great testament to like you said, I had become very focused on the KPIs for my squads, the metrics for my squads, and obviously, how those ladder up to, you know, total business objectives and goals. 

 

But I think the trade off of that is you don't have the view of the full business because, you know, there are things that other teams are doing that you you're not working on, which I think is kind of unavoidable. You know, you can't be across the entire work of the whole business at any level of significant detail. But it was an interesting finding when I just looked at this dashboard, and it looks completely foreign to the ones that I was familiar with.

 

Gavin Bryant  29:35

One of the things that I want to ask you about that I stumbled across was the central analytics team. And their mantra was something that really stuck out to me. The company's success is equal to the sum of its decisions, which I thought it rounds in very nicely with us in experimentation is the central analytics team and the innovation team in the data scientists in that function is that something that you tend to touch on and work closely with as well or they're more a separate function.

 

Matthew Mulvey  30:09

Is that the central analytics team at BA? To be honest, I haven't heard that the terminology central analytics team. But there are a few analytics teams that we do work with. So, I'm sure it's related to or tied to one of them. But that's something that I've really appreciated about being in a larger business. So, at Woolworths, like I said, the team that I was in kind of owned digital analytics and experimentation. So, I was sitting shoulder to shoulder with the analytics team. 

 

But that also meant that a lot of that responsibility fell on me as an analyst. What I've noticed British Airways is because it's a larger company, we do specialize a bit more, which means that I can kind of hand over some of those analytics responsibilities to people who do it full time, and to be honest, do a much better job than I do. 

 

So, that's been a great thing. But like you said, I think the point about the business being a sum of its decisions, really comes back to the principle that I mentioned earlier, where if you can make decisions quickly, and you're making something with some degree of confidence, that's more important than being paralyzed, because you're waiting to get to 95%. You'll increase that sum and therefore the value of the business by making decisions in a timely manner, add to the maximum level of confidence that you have available at that time.

 

Gavin Bryant  30:13

Yeah, it really allows you to create and deliver value for customer and business much faster, doesn't it?

 

Matthew Mulvey  31:49

Yeah, exactly. Because, like you said, that there is a real cost to delaying decisions. It's not just a theoretical kind of strategic initiative. Delaying an experiment rollout by a week could mean, you know, X amount of dollars that the business has forgotten, or it could mean, all the customers that booked during that week don't have this shiny new feature that you're gonna give to them. So, I think you do need to be mindful of that.

 

Gavin Bryant  32:16

Yeah, that's a really good point. There's always opportunity costs to all the decisions that we're making and actions that we're taking. 

 

Matthew Mulvey  32:23

Yeah.

 

Gavin Bryant  32:26

So, let's think about BA a very big business, a lot of different teams and a lot of complexity in the business working through a large transformation program at the moment. What are some of the challenges that you're facing into with experimentation at the moment?

 

Matthew Mulvey  32:46

So, I think, one of the biggest things for us is, as we get into new ways of working, or as we work with new teams, I'm a little bit afraid that we will lose some of the built-up culture that we have already. And I think that's kind of a probably a natural human concern, when it comes to change. I'm trying my best to fight that because I think, as an experimenter, you really do have to embrace change. And often, the work that you're doing is kind of the measurement of some of the biggest changes within the business. 

 

So yeah, what I think about a lot is, how can we work with teams that are working in different ways, or teams that haven't worked with us before, and bring them up to the level of maturity of the teams that we've been working with for years, as quickly as possible. And I don't think you can really force that because like we were saying earlier, I don't want to just go up to these teams and say, Hey, you can't release anything until you've tested it, here's a new way of doing things. And I don't want to force it on them. 

 

So, it's very much a kind of internal marketing piece. It's a coaching piece, taking them on the journey of demonstrating the value that experimentation provides, and then convincing them that, hey, given that value, this is the easiest way to achieve that, which is the experimentation process that we have. So, I think that's probably the biggest thing as the business changes around us, trying to retain the gains that we've made over those last few years. But I think it is a great opportunity as well, because working with new teams and new people means that you can find new ways of working that are potentially better than your existing ones. So, I'm kind of embracing the change from a growth mindset on that front.

 

Gavin Bryant  34:50

Yeah, the experimentation program is always being experimented on too, isn't it?

 

Matthew Mulvey  34:56

Yeah, well that's the thing, right? We may get small sample sizes in terms of the number of actual experiments that we're running. But we're always open for new suggestions. It would be quite hypocritical of us if we said, hey, this is the way we run experimentation and when we're not willing to test any new ones. So that's a great reminder for anyone who's kind of feeling a bit too protective of the current process, I think.

 

Gavin Bryant  35:24

Yeah, I wrote an article on this recently around the experimentation tax and the key call out there was that the job of the experimentation team is never done, it's a destination that you never reach.

 

Matthew Mulvey  35:37

Yeah, well, that's something that I remember the surprise me when I first joined Woolworths, and I was very, very new to the kind of web development business, I thought to myself, Hey, are we just going to build all these features on our roadmap? And then we're finished building the website? Like, is there gonna be a point where the website is done? And the team’s kind of just said to me, Matt, if that was the case, we would have been done 10 years ago, you know? But no, definitely not the case in process. I can't remember who this quote is from but somebody said there's no such thing as a finished product. It's basically always evolving so.

 

Gavin Bryant  36:16

I guess it gets back a little bit to that notion of Product Market Fit, doesn't it? That we find product market fit at a point in time but product market fit is never stagnant that as customer motivations, needs desires, and market conditions change, we need to then recalibrate and keep finding Product Market Fit perpetually over time. So, I think that's an interesting point to remember with the development process.

 

Matthew Mulvey  36:47

Yeah, exactly, you can have product market fit. And then when the market moves, you are no longer fitting the new market. So, it's always kind of you are chasing your customers to be where they are currently at. And you never quite catch them. It's almost like an asymptote from mathematics. 

 

Gavin Bryant  37:05

It's always a quest for relevance, isn't it? And that one of the things that I want to reverse up on there for a moment was at your time at Woolworths, you mentioned that a really effective way to be able to onboard new teams and new people was to use shining lights from within the business to communicate and demonstrate the benefits of experimentation coupled with some really strong top-down leadership support. Within BA where there are a nascent teams and teams that may have had little to no exposure with experimentation is that still your go to strategy or are there some other strategies that you found effective to?

 

Matthew Mulvey  37:55

Yeah, that lead has been a really big help, we do have some really strong buy in among the management team here at BA as well. I think that there are some of the strategies that that I've seen that what they were currently kind of thinking about. So, things like an poll to predict experiment results. And at the end of every quarter, you announced who was the best at predicting the results, that's an interesting and fun way to build engagement that I've seen, suddenly, you can have a competition for submitting experiment ideas, and whoever's idea gets prioritized the highest and has the biggest impact wins, some kind of novelty prize at the at the end of the quarter. 

 

So, I think those are an aspect to how can you make experimentation fun, rather than a chore? But yeah, the key strategy for me, was back then and remains showing the business and customer impact. So, if you can show a case study of an experiment that's generated six or seven figures of revenue for the business, and it took two weeks to build, it's pretty hard to argue with the ROI of that. And similarly, if you say, hey, the business was interested in rolling this feature out, we tested that and we would have lost 10 million pounds a year if we'd rolled it out, then that's kind of, I guess, more-- One speaks to greed and one speaks to fear. So, you can pull on both of those, I guess core human psychological drivers to make your case for experimentation.

 

Gavin Bryant  39:38

And sprinkled in there a little bit of fun to gamify experimentation is always good to-- Hey, just out of curiosity, predicting the experiment or guessing the test. What's the leader in that competition? How accurate can they predict test?

 

Matthew Mulvey  39:57

So, I should not count this. But we were doing this a lot, BA, Woolworths, and for one quarter, I was the winner. But I was ineligible to receive the prize because I had access to all the test data. So, I was discouraged. I still count that as a win, because I didn't take, I made all my predictions before the test had launched. So, I was still counting myself as the winner. Even though officially I was not eligible, which made me very sad. I was very unhappy about that.

 

Gavin Bryant  40:30

So, what was your hit rate out of 10?

 

Matthew Mulvey  40:36

It was not great. To be honest, I think it was about maybe 45%. And that was that was roughly choosing when loss or inconclusive, so you would expect about 33% chance at random. And I was about 45. So very, very slightly above the random rate, but the sample size was also very small. So that may have just been an outlier.

 

Gavin Bryant  41:05

Just shows that human beings are terrible at being able to predict the future.

 

Matthew Mulvey  41:03

Yeah, well, these are tests that I was running, I had come up with some of these ideas. And even I just was basically as good as random chance at guessing the outcomes, which was a bit embarrassing, but oh, well.

 

Gavin Bryant  41:24

So, thinking about some advice to our audience and our listeners that you have been really lucky to be able to work through the line in increasing maturity across two different experimentation programs and optimization programs what would be your top two or three pieces of advice from our listeners to consider we're also on this journey?

 

Matthew Mulvey  41:49

Yeah, so I'd say number one is, I think, especially if you're new to experimentation, or you're a new CRO, it's easy to get downhearted about where you are, you know, you look at booking.com you look at Facebook, you even look at something like British Airways, which is not in that 1%, but has still got a reasonably mature program. And I think you could easily feel quite discouraged. What I will say is, if you're testing anything, you're doing a lot better than the vast majority of businesses out there, which are testing nothing. So, if you're doing any tests, you're already doing great. I think the other thing, like you said is, if there's no such thing as a finished product, in terms of a website, there's no such thing as a finished experimentation program. 

 

So, I think, when I first got into the professional world, I was always looking to finish things, almost like an assignment, a university or the way that you're taught in school. So, it always made me feel a little bit anxious that, hey, I've been working on this for a year now, or I've been working on this for six months. And we've still got so much, we basically have an infinite backlog that that extends into the future. That almost made me feel a bit uncomfortable, because I felt like oh, have we even really made any progress, we had this unfinished product, now we've still got an unfinished product in terms of our kind of experimentation process. It's never going to be finished, it's never going to be finished. 

 

So, it's like you said, it's more about the journey than the destination. You make it better every step of the way. But you're never going to get to a point where you can say, hey, that's done, I can leave the business now. And nobody needs to worry about experimentation anymore, because it's going to take care of itself from now into perpetuity. So, enjoy the ride and don't get too hung up on when it's going to be finished, I would say.

 

Gavin Bryant  43:43

I was thinking about this recently myself like a metaphor for experimentation and I was thinking that it's a bit like a fire that initially you need a lot of kindling in to get the get the fire going and you start to fan the flames and eventually that fire starts to burn bright. However, at any stage if you stop adding wood to the fire to keep it burning that it will peter out and die. So, it's this constant cycle of invigoration or rejuvenation where you need to keep fanning the flames, and adding fuel to the fire to keep that fire burning bright.

 

Matthew Mulvey  44:24

Yeah, it's funny, you mentioned that because there was a product owner that I was working with in the past, and we had a great relationship. We're running some tests with them. And I thought, Oh, this is great. He's really bought into experimentation. And then I think it was about three to six months later, I was thinking to myself, and I realized, Hey, I haven't spoken to that guy for a while, like, I wonder what he's been doing. And I checked in on them and they just been you know, continuing his work still releasing features, but he just hadn't been testing any of them. And I thought, shit, I had kind of taken it as a given that once he starts running tests, he's gonna keep running them. But like you said, you need to keep feeding that fire, you need to keep doing the internal marketing of the experimentation program. You need to keep checking in with people. You can't kind of like you said, rest on your laurels. So that was a lesson I learned the hard way unfortunately.

 

Gavin Bryant  45:13

Was there any feedback from the PM as to why they stopped experimenting?

 

Matthew Mulvey  45:20

No, I think it was a terrible situation because he hadn't made an active decision. I hadn't made an active decision. I had just gotten busy with other stuff. And he was busy doing his releases. So, neither of us had thought to reach out to the other about doing tests. So, it was a kind of a tragedy of coincidence, I guess.

 

Gavin Bryant  45:43

Interesting point. let's close out with our fast for closing questions - number one what are you learning about that we should know about?

 

Matthew Mulvey  45:56

Oh, one podcast that I've been listening to recently is, so I'm very interested in behavioral science, behavioral economics. I've been learning a lot about the replication crisis and economic science, basically saying that a lot of the published papers these days, can't be replicated, or they're victims of P hacking where you select 20 variables, and one of them is going to increase significantly by the nature of random noise. So, I've been thinking about how to apply that to digital experimentation. But it's an evolving thought. I haven't gotten anything interesting. That's come out of that yet.

 

Gavin Bryant  46:38

Any interesting specific podcasts in that space that you think it's worth listening to?

 

Matthew Mulvey  46:43

I would always recommend Freakonomics Radio. So, the cult, pop psychology book Freakonomics [phonetic 46:50], they've got a podcast, which is on every week, and they get into some really fascinating research. I also know that I have seen Ronny Kohavi kind of ex Amazon Experimentation guy on LinkedIn has also talked about, if you see surprising experiment results, your first step should be to try and replicate them, which I think is a great position to take.

 

Gavin Bryant  47:15

Yeah, good rule of thumb. Number two, what's been your most challenging personal struggle with experimentation?

 

Matthew Mulvey  47:24

I am gonna take it slightly out of experimentation and bring it to the work, work life more generally, I really struggle with receiving feedback. And I think this is kind of a human nature thing. But it's very hard for me not to take it as criticism or to take it personally, like, so that's something I've really been trying to work on is, if someone criticizes your work, they're not criticizing you, the work you do, and the person that you are, are two separate things. And so, it kind of framing it in that way has really helped me to say, Oh, someone isn't happy with this report that I made, or they given me feedback on the last six months of my work, that's feedback on the actions that I've taken. And the decisions that I've made, but that's not me, as a person, I shouldn't feel bad about myself as a person, I shouldn't feel personally attacked. I should take it as a learning opportunity. So that's probably something that I have struggled with in professional capacity.

 

Gavin Bryant  48:27

To flip that one around has it changed the way that you give feedback to others?

 

Matthew Mulvey  48:33

Yeah, I think I was a terrible giver of feedback. When I first started working. I was, I tend to be quite a people pleaser, I like to get along with everyone. So, I would give the kind of feedback where I'd say, hey, you know, you've been doing really well. And I've really enjoyed working with you. And people say, Matt, do you have anything I could improve on? And I was like, oh, no, not really great. And I think that would be really frustrating if someone was seeking to improve because I'm not giving them anything they could improve on because I don't want to hurt their feelings or I don't want to be confrontational or I don't want to be rude but I think I have now kind of built up that courage to say hey really enjoy working with you and we're doing great. I would enjoy even more, if you could try to think about it. So, try to think about why.

 

But really taking care to still frame that as not a personal criticism, but you know, criticism of the process or a specific piece of work.

 

Gavin Bryant  49:37

Feedback is an interesting one …I’ve always been very mindful about giving feedback to people. I always try to take the time to ensure that feedback is relevant and meaningful, so that people can integrate the feedback to develop, grow and improve. Rightly or wrongly, I’ve been a believer that “no feedback can often be better than providing bad feedback” which is maybe flawed thinking. But providing thoughtful, carefully considered, and timely feedback is super important to me. Providing bad feedback can be soul destroying for the receiver. Providing feedback always makes me nervous … I always want to try and provide high-quality feedback. 

 

Matthew Mulvey  50:09

Yeah, definitely

 

Gavin Bryant 50:14

Okay, number three, what's the biggest misconception people have about experimentation?

 

Matthew Mulvey  50:18

I think it might not be the biggest, but one that always kind of annoys me about is when you talk to someone who hasn't done experimentation before, say, a development team and this and you describe the process to them, and they say, oh, that's going to add two weeks to our release cycle, or that's going to make everything take longer. I think there was grain of truth to that, it probably does add some time. But if you look at some of the online stats that say that 90% of experiments are either flat or losing, are you really willing to waste 90% of your time your web development effort to build things that have no impact or actively harm the business? So yeah, I think that kind of misconception that it's just an extra chore on the checklist and that it's going to slow everything down by quite a significant degree is a bit ill founded. My philosophy has always been if the experimentation program slows your release cycle down by too much, then you need to improve the experimentation program make it faster.

 

Gavin Bryant  51:22

Yeah, that was one of my questions that I was just thinking about then that how do you address that objection, which I think you've just hit it on the head there is to try and make the experimentation processes friction free as possible. It's interesting that point there that people immediately jump to more time and that's understandable. They're thinking about what's the impact to me but I think inherently and you hit on this that you know, a lot of what we try we know it won't work that you know, somewhere it needs to factor in what's the cost of learning and the cost of learning maybe a little bit more time?

 

Matthew Mulvey  52:05

Exactly, and I think the cost of learning is different depending on when you do the learning. So, learning after you're released to production, and after you've done an entire site redesign is incredibly expensive. Learning after you've done a five-minute mock up and say the visual editor of Adobe target, or Optimizely, is much cheaper. So, it's like you said it's minimizing that cost of learning is one of the functions of experimentation.

 

Gavin Bryant  52:35

Number four, three resources you'd recommend to our audience you hit on Freakonomics just recently is there anything else that is your go to or you found really helpful on your journey?

 

Matthew Mulvey  52:48

Yeah, Freakonomics is a great one. It's not specifically experimentation related. But if you're interested in behavioral psychology, behavioral economics, or kind of just science and research in general, it's great. I really like the podcast from A to B, posted by Shiva from formerly CXL, it's a great kind of very funny take on experimentation, it's probably the most lighthearted and enjoyable experimentation, experimentation resource that I've found. The other one, probably not experienced, explicitly experimentation, but I would really recommend atomic habits. the book by James Clear, it's probably the best book I've read in that kind of self-health, self-help or self-improvement space. It's a really takes a pragmatic approach to, you know, how can you build habits and improve your life. So those are probably my three.

 

Gavin Bryant  53:44

So, from James Clear, how have you integrated that into your personal life?

 

Matthew Mulvey  53:51

Less effectively than I would like, I must admit, I tend to be the kind of person that's like, Oh, if I want to build a new habit, I'm just going to set a date. And from that date, I'm going to start doing this new habit. And I'm just, I'm just going to do it. And then that'll last for about a week, and then I'll miss one day and then I'll think to myself, well, I missed yesterday, so I don't need to do it today. And then I missed that day, so I don't need to do it the next day. So, I always fall victim to the kind of the binge bust cycle of habit formation, which he specifically argues against in the book, but I really struggled to actually internalize that advice. So, it's a work in progress, we say.

 

Gavin Bryant  54:33

One of the things I really enjoyed from that book was the notion of not breaking the habit and even if you're maybe feeling unwell or tired, or unmotivated, that rather than do nothing just to do something, and if you previously did this task, 30 minutes but you're only doing it for one minute or five minutes. That's really good to retain the habit. So that's the key thing that I take from that book which is-- It's good for me I'm really sort of process orientated and, and goal driven that I always try and make sure I do something, even if it's, you know, maybe sub optimal or not to the extent that I'd normally do it.

 

Matthew Mulvey  55:13

Exactly and I think that probably does tie into experimentation where it's better to do something than to do nothing, even if something is not great. But also, I know he talks about never missed two days in a row. So, if you missed one day that was a mistake and that's a shame but get back on the horse the next day and keep going. Don't let that first day kind of throw you off the entire plan, which again is probably around trusting the process and building a process rather than striving for one specific outcome.

 

Gavin Bryant  55:43

Matt, I don't think there's a better way to end this discussion. So, let's leave it there. If people want to reach out and catch up with you, how do they find you?

 

Matthew Mulvey  55:52

Best is probably on LinkedIn. Matthew Mulvey. There's not many people with that first and last name. There are a couple which is unfortunate for me. But I'm connected to a lot of the common CRO influencers, so I'm sure you'll see me with a few mutual connections if you're in the field.

 

Gavin Bryant  56:11

Great stuff. Matt, thank you so much for your time today and thank you so much for being so candid, open and honest. Really appreciate it.

 

Matthew Mulvey  56:18

No worries. Thanks for having me.