Erin Weigel: Designing For Impact - How To Design Effective Product Experiments

Episode 017 - Erin Weigel - How To Design Effective Product Experiments

 

Gavin Bryant  00:03

Hello and welcome to the Experimentation Masters Podcast. Today, I would like to welcome Erin Weigel to the show. Erin Weigel is currently Senior Design Manager. Erin held senior product and designer roles at booking.com, over an eight year period. And she’s also the author of a new book, Design For Impact - Your Guide To Designing Effective Product Experiments. Welcome to the show Erin Weigel. 

 

Erin Weigel  00:29

Hey! Thanks for having me on Gavin.

 

Gavin Bryant  00:33

So, just a little bit of preamble for our audience and listeners. Erin was very kind to provide me with an advanced copy of her new book in PDF form. So, I am a reader and I have read the book cover-to-cover and I highly recommend that our audience and listeners, that they get themselves a copy of the book. If you’re working around product design and experimentation, irrespective of whether you’re new to the game, or you’ve been working in the industry for many years, there’s something for everyone in the book. There’s lots of practical strategies, tools and takeaways that you can apply immediately in your role tomorrow. So, it’s a really good reference guide and I highly recommend it. 

 

For our listeners who aren’t familiar with you and don’t know you, can you provide a quick overview of your background and journey to date please?

 

Erin Weigel  01:27

Yes, sure. So, my name is Erin and I am a designer / developer / product person / business person / I don't know, I do all kinds of stuff, which is why my domain name is erindoesthings.com. I actually have a degree in fine art painting. So, I started off my career journey more as an artist, you know, looking critically at things and trying to see and find connections and visually represent them or connect with people through color use form, that kind of stuff. Eventually, though, I realized that I couldn't make money, you know, drawing naked people or selling my fine art, abstract paintings. 

 

So, I ended up learning how to code and doing some email development and design where I did a lot of like, email abandonment campaigns, marketing campaigns, A/B tested a lot of them as well as landing page designs, A/B testing those. And eventually, I found out that I really loved it. Because I do feel like there's a pretty big connection between artists and how they see the world and question things or have a bit of a renegade spirit as well as experimenters, you know, always poking the box and seeing like, what edges and boundaries, you can push just to learn something new or find new connections. 

 

So, it seemed like a pretty natural step that I continued my design and experimentation career where then you know, became a principal designer  at booking.com after working there for a number of years. So, now I'm just continuing to love experimentation. And it's always the first tool in my designer tool bag or decision making tool bag that I grabbed for, if I have the opportunity to actually use it to make better informed decisions.

 

Gavin Bryant  03:07

Fantastic. Let’s jump to the book more broadly. So, what was the catalyst for writing the book? Did you identify a gap in the market? Did you feel the need to put your life’s work out on pages? Or was there some other reason?

 

Erin Weigel  03:22

Yes, so. Okay, so I guess it kind of came from me speaking at conferences. I'm like a reluctant conference speaker and a reluctant author, because it was nothing that I ever actually planned to do. It's something that I just kind of stumbled into and I was like, sure, why not, right. But I had given some internal presentations at booking.com that I was forced to do by my boss Stuart Frisby and he voluntold me to give a presentation at the company conference. And I was like, no thanks. And he's like, No, maybe, I phrased that as a question but I'm your boss. And I'm telling you, you should make a presentation. So, I did it. 

 

And some people at the company including Lukas Vermeer, saw it and Lukas Vermeer, he was a previous guest of yours. I saw Correct. Yeah, exactly. So, Lukas Vermeer, he had been speaking at different conferences. And I guess at a certain point, you know, when they were asking Lukas like, Hey, who would you recommend? He's like, you should probably check out Erin Weigel. Like, she tells funny stories, and it has a different perspective on things. So, then I started getting asked to do that. And then eventually, I was giving enough presentation conferences and hearing questions from people, a lot of the same questions were being asked over and over again.

 

And one day, there was like a Q&A, like, the slide and Q&A, or whatever. And there was a question that popped up to the top of the list, which was like, Can you write a checklist of how you audit a design for things that you should fix or change or whatever, specifically related to you know, when experiment ideas fail? And I was like, oh, checklists. I was like, actually, that's more like a book really and people clapped and they seemed really positive about it. And it wasn't the first time somebody had said, Hey, maybe you should write a book, like I'd read these stories. And I think, it's useful. So, I went back to my hotel room and I tried to write a checklist. And it turned out to be like the first chapter of the book, and I was like, Okay, well, Let's just try it. And then, I wrote a book. So, I had to learn how to write a book, though first. So, it was a long process.

 

Gavin Bryant  04:05

Right. That’s interesting, it was quite a natural process that evolved organically from your time in the trenches and your time on the tools rather than being something that was a very linear decision.

 

Erin Weigel  05:44

Yes, I think, that's how I do most things in my life, I explore and I follow a direction that seems interesting, which is why, I'm kind of against product road mapping and determining upfront, what you should and should not do. Because had I made it a plan to write a book before I turned 40, or whatever it's like. Then, I would had a lot of undue pressure. And that I would have, maybe, it wasn't the right thing for me in my life versus following a calling that I felt like, I had enough signs from people that maybe I should give it a try. So, I like to go with the flow and follow the signs with an experimenters mindset.

 

Gavin Bryant  06:17

Very spontaneous. You mentioned before that you had to learn how to write a book. What was involved there?

 

Erin Weigel  06:28

Oh, yes. So, I'm lucky enough that my father, he's like an award winning journalist, and also an author. So, he has critiqued my grammar and my writing. And, you know, he's read to me since I was a little girl, and I've read his columns and all that stuff. So, I feel like it's kind of in my blood a little bit. But I also love to learn, anytime I try to dig into something new, I do as much research upfront as possible. So, what I did was exactly what you said previously, I first started with market research, I dive into all of the books in terms of like, what was already covered in the experimentation world. 

 

So, I bought all of the books that I could find on Amazon, I went through all of the reviews to mine, what people did and did not like about each of the topics. I saw what people thought was like a unique selling point about the topics, maybe it was like really clear and informational graphics, or funny footnotes, or storytelling to help with concepts and everything or quotes that were inspirational. So, I kind of saw what people liked, but also what they were complaining about to see, what I could then fix in my book or address. 

 

And I think, one of the things that I found was, you either had very kind of superficial sort of, Hey, this test won, this test didn't, you should test because it's smart for your business, or overly academic jargon, where it goes deep into the numbers and stuff and becomes kind of hard to get through and the content doesn't pull you along. So, I felt like maybe, if I could write something that was more deep than the superficial. Hey, these tests work, these tests didn't, because that's also kind of a very simplified version of reality, of how experimentation works, while also scratching the surface of how the math actually functions, so that when people could do it in a bit more of an effective way, that was my goal. 

 

And then, from there, once I had my goal and my position within the product market, I guess you could say, I then started to approach it in the same way that I do any design task, I got some people that agreed to give me feedback. I read a book called “Write Useful Books” by Rob Fitzpatrick. It's a wonderful book, super simple, fast primer, highly recommended to anybody. And he just basically advocates using the design process and sharing your work openly and publicly, which is how I do my design work anyway. So, it came very naturally to me, just be like, Hey, I did this thing. Do you want to see? Is it terrible? How would you make it better? Was this useful? Did you like it or not? And then just kind of iterated my way until I got my deadline. And then I panicked. And then I was like, I got to finish it up. So, that's how it happened.

 

Gavin Bryant  09:12

It’s interesting that the process that you’ve described is effectively like, you know… A product development process that we would normally all work through on a day-to-day basis. So, you will literally applying those same principles and methodologies and approaches just within the context of a book.

 

Erin Weigel  09:32

Yes. The only difference is that, there was no like, get revert. You know what I mean? I couldn't just like control Z my way out of a bad release, which is really terrifying. Like it's published, and it's printed now, so all the little typos, but like they said, if you don't release it with some bugs, and you waited too long, so yeah. 

 

Gavin Bryant  09:53

Progress rather than perfection right.

 

Erin Weigel  09:56

Precisely and so I put it out and now it's out there.

 

Gavin Bryant  09:59

Yes, I think, you had a really good middle ground there. There’s definitely a lot of literature and papers out there that are highly academic. And then, there’s a lot on the other end of the scale that you mentioned, that are quite simplistic in nature. So, with a lot of business books that I read that typically rather than read to complete, I read to learn and once I’ve got those key concepts out of the book, I’m quite happy to put it down but I found with your book that it kept me going chapter-after-chapter right to the end. So, to thumbs up that, maybe, I’m passionate and interested in the subject matter area. So, I was interested to see what kept coming chapter-after-chapter. I think, it’s a credit to the book that you kept people going, kept people hooked.

 

Erin Weigel  10:52

Good, that's super helpful to have that feedback. Thanks so much. Because you know, it's like you never know, if you're actually going to achieve your end goal until you get that feedback loop that comes back to say, Hey, no, I stopped after chapter two because you bored me to death or Hey, it actually did pull me the whole way through and it sounds like you're my ideal target reader. So, Thanks, Gavin for reading the book.

 

Gavin Bryant  11:14

Hey, one of the things I wanted to discuss with you, we just had a little bit of a quick flyby with the process and what was involved in the book. And you posted recently on LinkedIn around, I guess, the worts and all of writing this book and those are quite a toll on your health, your well being, your family and personal life. For people that you know, that like the ideology and the romanticism of writing a book, could you give us the real eyewitness account of what’s truly involved?

 

Erin Weigel  11:52

Yes, it was a lot of hard work. And it was not glamorous at all. I ended up, I think what you're alluding to is that, I actually ended up very vitamin D deficient. And I ended up passing out like two weeks pretty much after I published the book where I sent it off to my publisher to be complete, because instead of going outside and taking my usual long dog walks, getting some sunlight in my eyes, or whatever, I was behind my desk writing or inside on my phone, typing thoughts out. So, I really neglected my physical health. And I cut every corner that I could, which then involved me eventually, like, depleting my vitamin D. 

 

So, that was not glamorous. I also missed a lot of really fun, like family things, they were like, they went to his birthday, for example, and they had nice, apple tart or whatever. And I wasn't able to go because I had to hit a deadline. So, I had to say, Give him a hug for me, I'm not able to make it because I have to hit this deadline. So, you really have to make sacrifices, and I would literally pound whatever thoughts I had into my iPhone keyboard, at all hours of the night, like I would read books to my daughter before going to sleep. And then as she was sleeping next to me at night, I would try to get more thoughts or more structure or write it on my iPhone. Google docs on iPhone is pretty incredible. 

 

And then, any free moment that I could print out a copy, because I always like to see things in paper form, like try to view things in the form factor, in which your readers will be experiencing them. So, even on dog walks that I did manage to take, I always had a printed copy of whatever chapter I was working on to read out loud to myself as I walked. And I eventually towards the end, started reading chapters of my book to my daughter to put her to sleep. And I tested some of the stories on her as well. But it was very taxing on a lot of, I didn't get to see many of my friends. My partner is very frustrated with how dirty the house is now and all that jazz. So, I cut every corner I could.

 

Gavin Bryant  14:06

With the scope of the book, the one thing that really stood out to me was that you covered a lot of complex concepts in the right amount of detail. How did you stay in the swim lane and stay on track because the breadth of that book could have been like a snowball rolling down a mountain that it kept growing and growing. It must have been hard to like put in end points around some of those topics that you discussed.

 

Erin Weigel  14:36

Yes, deadlines really help in that respect, because I probably could have iterated on it to death, you know. And in painting, we have the same kind of thing where you can overwork a painting so, you have to know when you hit your exact finishing point before you start to make the colors muddy. So, anybody who paints knows exactly what I'm talking about. The art of stopping is very important in crafting any sort of, I don't know artistic piece. So, basically what really helped was, I had a data science collaborator, he's a great, awesome person. His name is Lucas Bernardi. He's a principal data scientist. He's worked at Booking, he works at Meta currently. And he was on my team, when I was a data science product owner back at Booking. And we would always have these weird conversations about data science. And he would explain these concepts to me, and we would try different things out. And I would learn about how numbers worked. And I love the way his brain works, because he actually thinks a lot like an artist, I think, where he likes the philosophical concepts behind how math works. And he studied Physics, he, I think, he dropped out of a Physics degree. 

 

So, there's a lot about what you can learn about the universe through Mathematical Analysis as like a language that the universe speaks. So, he would explain these concepts to me. And then luckily, I'm ignorant enough, I would have 1000 questions. And then, I would kind of throw it back at him and say, Well, is it kind of like that? Or is it like this? And he’d say, Yes, it's kind of like that. And then, I would draw a picture. And I would say, is this right? And he'd say, Not quite, like this needs to be over there. So, we had like a back and forth question and answer session. And we would sketch things for one another. 

 

So, we use the language of math, we used pictures to understand one another, and then we used English. His native tongue is actually Spanish. So, it was like a mixture of different people speaking foreign languages to try to figure out, you know, what's the sweet spot? How do we understand it? How do we communicate it in both simplified English as well as in visual terms so that people can actually really appreciate how fun it is? Because I think, math can actually be super fun.

 

Gavin Bryant  16:50

Super fun when you can visualize it too, right. I imagine as a designer and an artist you like to think in pictures.

 

Erin Weigel  17:01

Yes, for sure. I think, what do they say, like a picture's worth 1000 words or whatever. So, if I was able to just draw it, instead of like, telling people I would always be like, can we just draw a picture to show what this means? And Lucas would always help me get there conceptually, so that we could just show people what the numbers mean.

 

Gavin Bryant  17:22

Let’s jump forward and have a more focused discussion around some of the elements of the book. What I’ve done for our audience is to just pull out a few themes and things that stuck out to me as I was reading through. So, we’ll just look to riff on a few of those themes now. 

 

One of the things that I noticed with the book Erin was that start to finish, there was this real underlying theme and concurrent thread that ran through the book around hypothesis development and formation. Could you talk a little bit to that please?

 

Erin Weigel  18:03

Yes, of course, so I think, one of the main common misconceptions about A/B testing is that you're trying to prove your idea right. And I thought that for the longest time, where it's just like, I've come up with an idea, I believe that the idea is better. And therefore, I'm going to run an experiment to prove that I'm correct. However, in actual science, there's something called Null Hypothesis Testing, which actually flips that concept on the head, you have to stubbornly stick to the idea that's already there. So, you stubbornly stick to what is currently baseline. And then you have to have very strong evidence to sway you to believe that whatever it is that you came up with, actually better. So, it's like the exact opposite of what most people think. 

 

And they have a nice, there's… I forget who said it, but it's in my book as well. So, apologies to whoever was the first person to say this. But Test like you're wrong, Design like you're right. So, you have to Design like you're right. So, come up with the best idea that you possibly can and implement it as flawlessly as you possibly can. So, that you really have confidence in putting a good challenger up to the baseline, but then Tests like you're wrong, which means flip that mindset of saying, I actually really believe this is better, but I'm not actually going to believe it until I get strong enough evidence to change my position and accept the challenger as the winner or sorry, rejecting the Null Hypothesis. And then there it actually replaces what the challenger replaces the baseline state. 

 

So, I feel like having that skeptical mindset, and constantly looking critically at the data and saying, Actually, I don't believe that, and then only changing your position really lowers your false positive rates and it makes your job a lot more difficult. However, it also gives you better results because then you know that you've really thought critically about it and you indeed have strong evidence to prove that you know the baseline should be rejected. So, it just flips what most people think on its head.

 

Gavin Bryant  20:08

Yes, I see a lot of teams that even before they’ve performed experiments, they’ve effectively in their mind already rejected the null hypothesis any way they can to accept the alternative and you had a quite detailed chapter on this, around the different biases and there’s all sorts of biases particularly confirmation bias that can come into play.

 

Erin Weigel  20:35

Yes and people also have a tendency to change their hypothesis after the fact. So, then they'll say, Okay, I thought it was going to be better on conversion for literally everybody but I don't see that in the data. But I do see that it did perform well for the Chinese markets on Tuesday afternoons. So, that's not my hypothesis and I'm going to turn that full on what an actually they're probably just looking at like a false positive due to the natural occurrence of data volatility and low sample size skewing result. So, you have to pick, write your hypothesis upfront, be highly skeptical and then don't change it to confirm whatever you know belief you already have. You always have to remain that skeptical. Yea, exactly. Yes, nobody likes a cheater, that's basically what you're doing, you're cheating. You're shooting your gun against the… shooting bullet holes into a wall and drawing your target around where you've already shot, that's just cheating.

 

Gavin Bryant  21:17

Do not be the Texas Sharpshooter, right. 

 

One of the things, that I’ve been thinking about with hypothesis development and formation, and I’m not sure, if you see this in your work as well, is it inherently teams and people they can struggle quite a lot with hypothesis formation. And there was a brief piece in the book that discussed this, the two different types of thinking, Linear thinking and Systems thinking. And I think, a lot of time in business that you know, we’re so caught up in that Linear thinking mode, that it can be hard to move out of that. 

 

And I think to be skilled and really proficient at developing and formulating well reasoned hypothesis, you need to be a really good Systems thinker. And we touched on this a bit earlier about connecting the dots and piecing together all these disparate pieces of information to be able to sense make and pattern make. What’s your experience with that?

 

 Erin Weigel  22:33

Yes, I mean, I do feel like the way that we're taught, at least in most western schools. I only know some Western schooling educational approaches, and they tend to be pretty linear. And they tend to be teaching children to regurgitate facts versus to think critically and look critically at the world and to explore and to discover things. So, it's kind of un-surprising to me that we bring that same way of thinking that we're taught in school, to our places of work, because we like predictability, we like to feel as if we're in control of things. It just gives us that sense of comfort that we need as human beings to be able to function on a daily basis. And I really feel like my education in art school was all about, it was like a masterclass in critical thinking and systems thinking. 

 

So, I feel like it was just a natural thing for me to be able to be imaginative, and constantly ask, well, what if this, what if that, you know. So, I just encourage people to maybe step back and not be afraid to ask a million questions, and then to play with the, you know, what, if this might happen, what are all the things that could potentially be affected by it and embrace that complexity. And of course, you're not really going to know. But it's really kind of fun to learn about what things might be connected, that you don't think are actually connected. And that's when you are actually able to make really meaningful discoveries, if you will. So, I think we just have to learn, how to think that way. And I do believe that it's in every single person can think in that way. It gets taught out of us at school as we need to regurgitate facts.

 

Gavin Bryant  24:12

Yes, I think a good one on there as well is to expect the hypothesis to evolve and refined over time rather than being something that is set in stone to begin / won and done.

 

Erin Weigel  24:28

Yes, exactly. I also think, we have a tendency to not think clear, simple thoughts. I think, the art of thinking clear, simple thoughts is highly under-rated. We tend to glorify people who speak in like really verbose, academic language in ways that can just be completely impenetrable, to just like, tiresome to listen to. When distilling something down to its most simple form where like, if this happens, I believe if that, if we do this, I believe that might happen. Because dot-dot-dot is such a very simple and clear way of thinking that it almost seems counterintuitive, if you want to project yourself as like some sort of, I don't know, authority figure, which a lot of people in business need to have other people believe them and influence them to allow them to test these ideas or, you know, to get your next promotion. So, I think it takes a bit of courage to just speak in very simple terms, have very clear if then thoughts, and just, you know, cut all the nonsense out around it to write a truly distilled hypothesis. Yes, exactly. Nobody wants that.

 

 Gavin Bryant  25:39

You know, cut the BS out. 

 

Something I was thinking about in the book, “typical product development leads to good and bad decisions” … could you talk to that one for a moment please?

 

Erin Weigel  26:01

Yes, so a lot of teams, they do this thing, which perplexes me is, they come up in advance what they want to do, right, maybe based on research, maybe not, and then they make these road maps and they say, First, we're going to do this, then we're going to do that, then we're going to do this, that and the other thing. And they sometimes put out a years worth of work, which is absolutely bonkers to me, because so much stuff can go wrong in between, you know, when you make that list of things, a round of layoffs could be coming, or investors could change their mind and pull funding, or, you know, a new executive might come in and throw their weight around and just pull your roadmap and you spent all that time planning everything. 

 

So, they usually have these big, beautiful roadmaps with all these ideas, and then they tick them off one by one. And the victory is in delivering them right, delivering the tasks that are on their backlog to do within their roadmap. But the thing about working in that way where delivery is the definition of success, is that as you push these things, a lot of stuff that we work on, either has no meaningful result, sometimes has a negative result, and very few times, does it actually have a positive, meaningful and measurable impact on whatever it is that you're trying to measure. 

 

So, when you just ship all those things, you're shipping, that mixed success rate, the good stuff gets canceled out by the bad things you've shipped, the mediocre things just kind of sit there and bloat things, you know, and doesn't really make anything better and potentially adds a lot of tech debt in the end. And eventually, it could combine to be net negative. But when you experiment, it feels like you're making less progress because you're actually shipping fewer things. But instead of shipping the crappy stuff or the non-impactful stuff, you're only shipping the good things, which is actually what moves your business forward. But people like the dopamine hit of ticking the box, and you know, hitting their deadlines on their road map, when that is actually the exact opposite of how you make an impact and drive businesses forward. Which is why, the world's largest most profitable companies in the world, you know, don't really work in that particular way. Or if they do, they always test it and validate that it has indeed had the impact that they were aiming to have.

 

Gavin Bryant  28:27

Yes, the product roadmap is such an interesting one, that the way that I’ve sort of flipped my thinking to think about the product roadmap, to your point, it is often an expression to deliver. But I think, the roadmap should be an expression to experiment. And the way that I think, it’s a really good visual representation of thinking about the roadmap is like one of those really big complex freeway interchanges in LA, for example, where there’s roads going everywhere. And sometimes we’re going to keep going straight down and head to our destination from point A to point B. Sometimes we’re going to sub something out, we’re going to take a freeway exit and take an opportunity off there. But then with one of the road ramps, we're going to sub something in and bring in a new opportunity because we’ve got scope. And there’s now more bandwidth and room given that things have been knocked out and canceled. So, I like that little metaphor to think about the product roadmap.

 

Erin Weigel  29:34

Yes, it's definitely a better one. But I do feel that there's time wasted in planning and managing all the work that comes with the concept of road mapping. So, even if you have a backlog of things and you have concepts in your roadmap, you've still really spent a lot of time maybe, managing the backlog and planning what you moving things around, which is kind of a bit of a waste of time, in my opinion, I'd rather say, what's the most important thing that we need to do right now. And then what's the next most important thing and why. And then really focusing on getting those things done versus worrying about things in the future. But really just focusing on the goal and getting there and not really caring about the path. So, focus on the goal, and who cares about the path. Just get there, you know.

 

Gavin Bryant  30:22

Yes. Another thing that really stuck out to me was… and you touched on this earlier that when we’re designing experiments and testing hypothesis, that we should, you know, test the best version of the theme, that we’re trying to test and then we’ll see what happens. So, one of the points that you made in the book that it is so hard to design experiments that are an accurate representation of our ideas, what are some pieces of advice and thoughts that you’ve got for teams to be able to help in that manner?

 

 Erin Weigel 31:00

I would say, try to decouple the concept of innovating and experimenting. So, because I was guilty of doing this, when I was just a little young baby designer, who had no idea of how the world worked, I would want to use some fancy new technology when implementing an experiment idea. I'd say, okay, maybe we should use this new fancy line of CSS to make this animation happen, or whatever. And my idea of technical innovation actually ended up crashing browsers and wasn't actually me testing the hypothesis that we were trying to test, it ended up being me testing that new line of CSS and how it actually crashed all these different browsers. 

 

So, I would say like, stick to tried and true technology, don't ever conflate technical improvements or design improvement unless that is, you know, what you're actually going for, but decouple the implementation and innovation as much as possible from your ideas. And then I would also say, stick to whatever usability and accessibility guidelines as closely as possible that are based on as much possible qualitative and quantitative research as possible. So, basic things like, ensure that you have good color contrast, unless there's a reason in the hypothesis that you shouldn't have that for whatever reason, if you're doing disabled buttons, or whatever, and just making sure that you know, you're setting yourself up for as much success as possible, make sure that you're using colors in an intuitive way. 

 

Yes, just try not to mess up the execution by being too clever. Just stick to the basics and test the ideas. And if you want to push the boundaries, then, you know, maybe keep the content same. And then, innovate on the design approach or the technical approach or whatever, but don't conflate too many of those things into one or else you start testing the execution more than the idea.

 

Gavin Bryant  33:04

Good point, thinking about a cultural element of product design and experimentation within the book, there’s a really good comment. It takes a strong and confident leader to give up their control to the employee base. Can you explain to our listeners, how you’ve seen that play out in booking.com for example?

 

Erin Weigel  33:29

Yes, so the reason I went to work at Booking is because power systems and hierarchy is just in my brain, absolutely bonkers. It is such a human construct that we put people on pedestals. And one of my favorite movies, when I was growing up, even now is The Wizard of Oz. Like, I'm always looking for people that are skeptical or people that present themselves as bigger and better than what they actually are like The Wizard of Oz was just like this great and powerful Oz, who had a whole kingdom built around him. And nobody ever really saw him. And he was like this huge head with smoke and mirrors and all these wonderful things. 

 

But then, this little dog pulled the curtain back, and it just to do with a technical setup, who didn't actually know, what he was talking about who scared everyone. So, it's like, I'm really skeptical of people that present themselves as something more than just a human, who's trying to do their best in the world and just wants to help and do better. And Booking.com had a great way of hiring leaders, at least when I was working there or finding really humble people who trusted people and was more curious about how they could set up systems and processes that allow people to fail in a safe environment while still pushing the business forward. 

 

So, like the CEO at the time, his name was Darren Houston, he didn't have any office. Gillean Tons also, she was a CEO later, she also didn't have an office, they just walking around, drinking out of the same shitty coffee machine that everybody else drank out of and eating the same lunch in the same canteen as all of us. And they knew our names, just having casual conversations. And even when I started, there was only like a few job titles like nobody was really senior. It's just if you were hired, you're a designer, or you're a developer, or you're a data scientist or a product manager, that hierarchy just didn't really exist. And then, that meant, we could really talk about ideas and data in a very objective way, which allowed us to… Yes, it just allowed us to focus on what mattered and not necessarily posturing to try to achieve our own personal goals, because we were working together towards achieving a collective goal, which is much more motivating, in my opinion. 

 

But eas soon as you introduce those hierarchical things, then you have people throwing their weight around, and then using their titles to push their opinions forward. And then, that's when the system starts to break down. So, leaders have to be totally okay and confident enough in who they are as a person, to allow other people around them to shine and to get those wins and to see that the employees succeed. It's not about them, it's about the employees. But that's a rare type of person to find, in my opinion.

 

Gavin Bryant  36:25

Yes, really good point. And we’ve got a few minutes left to close out our conversation. Let’s go with our fast-four closing questions. Number 1. What’s your biggest lesson learned from experimenting in product design?

 

Erin Weigel  36:41

But everything that I thought was right should ultimately be challenged, even what people consider to be best practices, executed inappropriately or within the wrong context can actually be horrifically harmful. So, don't ever take anything for granted as being better just because you read somewhere that it worked for somebody else, or it's a best practice and has a lot of research behind it. The world is much more complicated than we think. And we should be testing as much as possible.

 

Gavin Bryant  37:11

Number two, what’s a common misconception people have about experimentation?

 

Erin Weigel  37:18

I think, people over-inflate the amount of impact that people are actually having, because people misunderstand what the math means, like the reported mean or the estimated mean that you see back in your tool, and the words of my friend Lucas Bernardi is just a lucky number, you know,              And the one that it's reported is just the luckiest one in that time. So, I think, people should stop saying, Hey, we're going to deliver X, Y, Z, like this change has generated this much revenue. Because it's a lucky number. And you should really consider the entire range to say, you know, at its best, it could be this much, but that's likely overinflated. And, you know, speaking very conservatively, this is the lower bound and this is like, I always like to report on the lower bound because I feel like it's a more honest indication of what the impact actually is.

 

Gavin Bryant  38:15

I had read recently that some organizations are now offering like a discount factor to that value of 20% to 30% which is probably more in line to your point of, you know, a lower bound or worst case scenario.

 

Erin Weigel  38:28

Yes and people assume that just because they measured in that one period of time, that amount of impact doesn't mean that the world is not continuing to evolve and that the impact changes over time as the world evolves. So, just because something works now or worked when you tested it, it could be having a negative impact later as well. So, these things are not absolute, things are constantly changing. So, don't take that for granted either.

 

Gavin Bryant  38:52

Yes, customer needs and motivations. Marketplace is in a constant state of flux, right? Nothing’s permanent. 

 

Erin Weigel  38:58

Everything is in flux.

 

Gavin Bryant  39:00

Yes, we’re chatting about this on the last episode of the Podcast that product market fit that, you know, companies espouse and talk about obtaining this magical point in time where they achieve product market fit but, it’s not finite that the product is in a constant state of flux and evolution and as fast as you achieve market fit, you can lose it so it’s a constant battle that every product is fighting, month-in-month out and year-in-year out.

 

Erin Weigel  39:31

100%. Couldn't have said it better.

 

Gavin Bryant  39:35

Number three, once a strongly held belief on experimentation that you had, since you changed your mind?

 

Erin Weigel  39:45

Oh, my God strongly held belief, experimentation, that I've since changed my mind. I think, the most obvious one is the one that I've already said, which is I thought, that I was trying to prove my ideas is better and that's an absolutely not at all, what I'm doing. And I think, also, I've embraced more as I've grown up. And I've learned more about how the world works. I've also learned that it's not about absolutes and finites, it's still just about capturing a moment in time to inform a decision. And then just being a lot more. I don't know if it's like, loose or philosophical with how experimentation was, I used to use it as a form of certainty and comfort to say, I have true confidence in this, but now I'm okay with saying, there's a really good chance that this is better, you know, but I won't say, look, I'm great I did I increased blah-blah-blah, such and such as, like, I feel pretty good that this is not terrible. So, I've really scaled back the language that I use, that's what I've. That's a strongly held belief going from definitive to more nuanced language.

 

Gavin Bryant  41:02

Good point. Final question in our fast four. What’s the one key takeaway our audience should take from your book?

 

Erin Weigel  41:12

Make things better, not just different. Like that was probably the biggest thing that I learned is, that every time, I thought I was making something better using my fancy design research that I had done or using, like what I learned from a usability test. And it would just tank, it would be terrible or be negative, I'd say, Wait a second, I was certain that I had made this better, like what happened. So, again, if I had not been testing, I would have just released all the changes that would have conclusively made the product absolutely worse. So again, the main key takeaway is to make things better and not just different and collect highly convincing evidence or find, like, only believe, it's better if you have significant evidence to prove that it actually is.

 

Gavin Bryant  42:01

A great way to close our conversation today Erin, if people want to get in touch with you, where can they find you?

 

Erin Weigel  42:07

Well, I am an internet Gremlin. I love trolling on LinkedIn. So, if you like sassy comments and lots of emoji’s on LinkedIn posts, you can follow me there. And I also have my website, Erin Weigel does things.com, where you can buy my book, if you're located in the European Union, but those are pretty much the only places that I live on the internet these days. And then the rest of the time, I'm just in Amsterdam, in the Netherlands. So, you can find me there.

 

Gavin Bryant  42:34

Fantastic Erin Weigel, thanks so much for your time today. We really appreciate it.

 

Erin Weigel  42:38

Yes, thanks so much for having me, Gavin. It was a great chat. I loved your questions. Thanks so much.

 

Gavin Bryant  42:43

Speak soon.