Ruben de Boer - A Framework For Experimentation Change Management

Episode 007 - Ruben de Boer - A Framework For Experimentation Change Management

Gavin Bryant  00:03

Hello and welcome to the Experimentation Masters Podcast. Today I would like to welcome Ruben De Boer to the show. Ruben is the Lead Conversion Consultant at online dialogue. He's been working in experimentation and optimization for more than 10 years with a portfolio of world class clients, including Vodafone, and eBay. Ruben also conducts online coaching in education through his company conversion ideas, which has touched more than 7000 students around the globe. 

 

In this episode, we're going to discuss a Change Management Model that Rubin has developed, which helps organizations to manage the business change process for experimentation more effectively. 

 

Welcome to the show, Ruben.

 

Ruben De Boer  00:51

Thank you, Gavin, and thank you for having me.

 

Gavin Bryant  00:54

Great to have you here tonight. And I thought we'd make a start by just getting you to share your background and journey with experimentation today.

 

Ruben De Boer  01:04

Sure. So yeah, like I said, my name is Ruben. And I've been in the business of data and optimization since 2009. I think I did my first A/B test in 2000, early 2011. In 2009, I was helping a company that teach people to become happier, more confident with improved social skills. And at that time, I really learned the importance of helping and inspiring others with doing what you love. But also was fascinated with the question that if we can change the behavior of people offline for coaching, how can we do this online, so I really started getting an interest in psychology and human behavior, and changing it for the better. 

 

Furthermore, I've always been a data guy, I love data courses, or lessons much more at school than the languages. And I really have enjoyed optimization of optimization mindset. 

 

So the combination of this all of my interest of my passions kind of naturally led me into the field of conversion rate optimization. And many years later these days, I've worked with many clients, great people around the around the globe. And I'm really happy to help people learn and excel in conversion rate optimization for my company conversion ideas.

 

Gavin Bryant  02:23

Awesome. So thinking more broadly now, what are your guiding principles that you've developed over the years, or your philosophy around experimentation.

 

Ruben De Boer  02:35

Of course, there are several guiding principles which you can find in many blogs and articles and books like data first, challenge assumptions, be curious, be customer centric. 

 

But let me mention the three important ones here, which are maybe often by some forgotten. 

 

First of all, I always like to keep in mind a simple formula for the return of investment of your experimentation program. That return of investment is the quantity times the quality divided by the costs. 

 

The quantity is simply the number of experiments you run, if you run more experiments, you'll find more winners or losers. And you will learn more, and of course, very beneficial for you for your team and organization. 

 

Quality is simply the number of winners you find, or the number of losers you prevent. And the win rates, of course, with all the correct statistics and stuff, so you don't run into false positives, etc. 

 

And the cost is simply the efficiency of the team. So when you're aware of this, and keep this in mind, especially if you are the one who runs the program, the process, when you're aware of this, you can always structure your work and your progress and keep optimizing the program itself optimization program. So, it is one. 

 

Two others. Always aligned with business goals. I see a lot of CRO teams, they have a lot of specialists who are sitting in the back of the office and doing small copy changes, for instance, but do line with the business and show that experimentation can really help those business goals really grow the whole business and involve others. Your job is not just to run A/B tests you want to really get in culture of experimentation, you have to involve others especially also higher management involve them as soon as possible. So, those will be three guiding principles.

 

Gavin Bryant  04:33

So quick question on quantity and quality. Do you see teams sometimes they make the mistake of prioritizing quantity over quality?

 

Ruben De Boer  04:46

What I generally see with my clients that it's actually the quantity that's lacking usually when I start with my clients, that quality is okay, they have a reasonable win rate. In some cases, it's not the case, and I might change their better research at a psychologist, thing more about what we learned in an experiment, but in general is the quantity, that's the bottleneck. Too many meetings about a design before it's been tested, too many people involved. Too many design iterations, etcetera, etcetera, generates the quantity that needs to go up. And the quality is reasonable for most clients. So, but it's... Again, if you keep this formula in mind, you can see for your own organization, what should I do if the win rate is, let's say, sufficient 25% or higher 1/3, then really focus on the quantity get those bottlenecks out of the way, and make your design less perfect. It should match the hypothesis but not 10 colleagues should have an opinion on it. Have discussion about data... Yeah, increase the quantity. That's what I generally see.

 

Gavin Bryant  05:58

What are some of those common bottlenecks that you see with your clients?

 

Ruben De Boer  06:02

It's generally too many people involved. I mean, especially when you look at the design. It's a matter of taste, and you never get a design that everyone absolutely loves. So as a CRO specialist, or someone responsible for experimentation, you have to ensure that the design matches the hypothesis of your experiments, and then go and don't ask 10 other people. It's better to have discussion after the experiment, because you have the data. Maybe it's inconclusive, then yet all these meetings, all these discussions for nothing

 

Designed by a committee does not work upfront.

 

Ruben De Boer  06:41

Exactly.

 

Gavin Bryant  06:43

And just before we move on, so thinking more broadly, you mentioned that experimentation doesn't operate in a bubble, that there's broad stakeholder engagement and management required. So what are some of those key things to do to ensure that stakeholders are engaged, particularly senior stakeholders?

 

Ruben De Boer  07:06

It really depends on... but we'll get.... I think, I guess we'll get to that during this interview. It really depends for each company. You know, I sometimes hear like, bold statements that if you want to improve hire management, always talk money, that could work for some management's. But companies that really thrive with experimentation, they do not want to see the business case of every experiment you run, they simply see experimentation as a way of better decision making. 

 

So, in some cases, it works to talk money. In some cases, the higher management should be involved differently, for instance, test their ideas and show the importance of experimentation or improve them. Get them to brainstorm session. It's different for a company. So the experimentation mindset we have, we also used within our own companies. See how you can involve higher management, see how you can involve other colleagues, which you send update emails, you can invite them for brainstorm sessions, to just simply sit with them, and try to achieve their goals with experimentation instead of your own goals is different for each organisation, but involvement is always important.

 

Gavin Bryant  08:15

Good point. So thinking about some of the key challenges that businesses experienced with managing the change process with experimentation what are some of those challenges that you observe in your work with clients?

 

Ruben De Boer  08:29

Good question. I would like to state two main challenges. The first one starts with the mindset. And the beliefs of the CRO specialists themselves or the ones responsible for experimentation, it is very important essential for them to understand that their job is not just to run A/B tests, a large proportion of their work should be dedicated to change management, if you're really want to thrive with your experimentation program. And you really want to grow as a whole company into an experimentation culture, and thus really grow as whole company in its results, and in total, you will not get there by simply running A/B tests. Your job is to really motivate involve and get others on board. That's when your CRO program really start thriving. So your job is not just to run A B tests. It's also change management and a large portion should be dedicated to change for instance. 

 

Another challenge is obviously.... an obvious one is legacy. Many companies and people, some companies people have been working in those companies for many years with the same culture, same mindset and same way of working. It's really hard to change that.

 

Gavin Bryant  09:46

Two really good points there that you don't overestimate the status quo, and also managing that change process more broadly. So previously, I read that when thinking about organizational change, you'd highlighted System-1 and System-2 elements that are really important to that change process. Could you provide listeners with a little bit of an overview of those System-1 and System-2 elements, please?

 

Ruben De Boer  10:21

Of course, I can talk hours about this topic. Let's briefly start with it with beginning because I was generally for organizational change, but also personal growth and personal I was fascinated with one shocking study that really got me thinking and also working with change management and with this model. 

 

So, study from 2005, by Dr. Edward Miller, and it is study on a patient with clogged arteries. And this is a serious condition, it hurts thoroughly. And it leads to a premature, painful death. And actually, all these patients with clogged arteries, they can survive, it's actually the solution is pretty simple. They need to change their lifestyle, they should stop smoking, stop drinking, eat healthy, start exercising, and reduce stress in your lives. And when they do that. And when they succeed with that, almost all these patients will survive, and they stop the pain and they get healthy. And they'll see your children growing up as the children get married. They just have to do this. And almost everyone can survive. 

 

So it's a matter of life and death, in that case they obviously want to live. But the numbers show that only 10% of those patients, so 5% to 90% feels to change their lifestyle.

 

So that got me thinking because if change is so hard, even in the life and death situation, then how are we as conversion specialists and the ones responsible for experimentation? How are we going to change the culture, the mindset, a way of working of entire organizations? 

 

Unfortunately, there have been quite a lot of studies and good books on change management. And that's when I started combining our knowledge as CRO specialists and experimentation specialists with change management because we already know quite a lot about change. 

 

Ruben De Boer  12:28

So I started combining theories, and indeed came up with this model, which in the basis is indeed System-1 and System-2, which probably a lot of listeners will know from the book from Daniel Kahneman Thinking, Fast and Slow. 

 

So System-2 is a metaphor for our rational brain. It's a part of our brain that can reason and think in the past, present and future. And for change to succeed, it has to be smart enough, it has to know how to change this know what to do. So if you want organizational change, our System-2 educates your colleagues. But we know that's not enough, right? We can tell our colleagues how to do experiments, we can tell them that big tech companies do it, that our competitors do it, but it does not result in lasting change in shifting culture and way of working. 

 

So we also have System-1, our primal brain, that's our old primal, strong brain. And that part of the brain, it's kind of reason and cannot think in the past, present and future. It's our subconscious brain. And rule of thumb is 95% of our decisions are made by System-1. System-2 is rational brain; System-1 is our primal brain. And if you want System-1 on board, it should be motivated. We need to motivate the System-1s of our colleagues. So there are couple of things you can do. It needs autonomy, you need to make experimentation accessible because it does not like to put in the extra effort. Set small changes, it doesn't like big changes. But other than that, it's really hard to motivate System-1, it's really difficult. And when change feels it is often caused by System-1, not being motivated and not working together with System-2. 

 

So, you can think about it when on a Sunday morning you set your alarm clock early in the morning to go for a run before having breakfast with your family. So the night before just in two sets the alarm clock because it can think like 'Okay, tomorrow wake up early. I can go for a run and if I keep it up in six months, we're going to be healthy and fit and well-trained. Now when the alarm clock rings, no Sunday morning. That's when you feel System-1 and System-2 fighting together. System-2 knows I have to get up because I have to go for a run. Right now I get fit. I get in shape. And when I'm back I can have breakfast with my family. But system one primal brain it cannot think in the past, present and future. It doesn't care about getting fit, it doesn't care about going for a run, it only knows that you're in a very comfortable best, your head resting on a pillow it all wants to do is turn the alarm clock and get some more sleep. So, System-2 once you get out of bed, System-1 is the strong part of the brain that wants to stay in bed and sleep. 

 

So that's a struggle we see in that morning, which probably a lot of people recognize, but also with organizational change, personal change, it's that struggle that makes change very hard. And to create lasting change we want System-1 and System-2 to work together. Long story....

 

Gavin Bryant  15:45

That's a great overview and yet highlights why it's necessary for teams and programs to be able to draw on both of those elements of human psychology, if they want their organizational change program to be effective.

 

Ruben De Boer  16:06

Together and motivate System-1, that's why I introduced this model.

 

Gavin Bryant  16:12

It's definitely a good way of framing it up. So thinking about introducing the four quadrant model that you found effective in eliciting change for experimentation programs. So what are the key elements of that four quadrant model.

Ruben De Boer  16:31

So to have System-1 and System-2 work together, we need to tweak our environments. Because that's how we motivate System-1 and have them work together. So we can tweak the physical environments, we can check tweak the social environments, we can set rules and norms. And we can provide rewards and incentives. And when we tweak those four, and we think about it for our organization, that's when System-1 and System-2 can work together. And that's how we can get to organizational change and culture of experimentation.

 

Gavin Bryant  17:07

So it's about attacking it from every angle to try and get the maximum output.

 

Ruben De Boer  17:12

Precisely.

 

Gavin Bryant  17:15

So let's talk a little bit more specifically about the physical environment. What are some of those core elements that people should consider for unlocking the physical environment?

 

Ruben De Boer  17:27

Yes, so as mentioned, I tried to get theories from change management, and combined with our knowledge, and especially when working on A/B testing experimentation, we tweak the physical environment of our website visitors every single day with the goal to change their behavior. 

 

So we actually already noticed we're experts at this because we can also simply use that idea, that mindset to tweak the physical environment in our office. So we can test with sending update emails, with insights from experiments to our colleagues, we can apply gamification, we can create accessible data dashboards and place them on monitors throughout the office. And of course, when your office opens again, you can share a forum for which colleagues can easily hand in their test ideas. Another nice example is a failure trophy, which I recently read an article in the Amazon, they have a failure trophy for someone who breaks the website badly. And it's seen as a badge of honor because this person did and tried something so bold, it's broke the website and it gets to places the failure trophy on his desk and you will get praised for it throughout the week. It's a perfect example of a fiscal tweak.

 

Gavin Bryant  18:47

Yeah, very good example though, I'd also read the example about the Amazon failure trophy as well. 

 

Ruben De Boer 18:53

Exactly. 

 

Gavin Bryant  18:55

For some of those elements there we talk about gamification. So guess the test type concept, which I've seen to be very effective with teams and it's highly engaging, because it illustrates that majority of the time that we're very bad at guessing the quality of our own ideas. And are there any other of those elements that you mentioned that you've found her really good cut through with organizations or a case of depending on the organization?

 

Ruben De Boer  19:30

Yeah, again, and as a good consultant, the answer is always it depends. Like I said, in the beginning, we have to use our experimentation mindset here as well. 

 

For instance, I also see some bold statements that dashboards purely in experimentation maturity, should not be shared with organization because then they see data, but they're not used to seeing data. So they'll draw the wrong conclusions based on what they see and they make the wrong decisions. It could be the case, but for one of my former employers, as well, I was the first year specialist as there was no experience yet with data driven work at all. But I placed a dashboard with them with some general numbers at the entrance. So everyone would see data several times a day. And it was some basic numbers like pageviews per section on the website and a number of people on the website. And it really worked because it was a live analytics dashboard. So the numbers flashed on every update. So when you walk past it, it automatically draws your attention. So people got to see data. But also, an internal competition arose, because pageviews are quite understandable. And I showed the pageviews per section for each product team. So when your product team has the most page views, that section of the website, you've got go to the other product team, and they started teasing each other. 

 

But I think the most impactful one was that visitors also got to see the dashboard. And they automatically assumed that the whole organisation is data driven. So every meeting with a visitor, including at meetings from board directors, who see visitors on a daily basis. This started with visitors stating, like that's so awesome, that organisation is data driven. So all of a sudden, the Board of Directors started hearing your organisation's data driven, and it really got them involved and triggered. 

 

So moral of the story is see what works for your organization. And what doesn't. For me the dashboard worked very well. But I can also imagine that people draw wrong conclusions based on the data and it doesn't work in other organisations. So experiment!

 

Gavin Bryant  21:48

That's a fantastic example. So thinking about the social environment, what are the some of the elements of the social environment that people should consider?

 

Ruben De Boer  22:01

Good one. And also here, social environments we notice right as optimizers. We know from our A/B test from blogs, from presentations, we know the power of social proof, ratings, reviews, testimonials, success stories, you see them everywhere. 

 

So again, we have this knowledge. And now we can apply it in our organisation to get to a culture of experimentation. So create a community of enthusiasts and promoters, experimentation, host brainstorm sessions, celebrate successes, and especially failures together, and have colleagues watch usability tests together. Everything that you can do to involve people, get them together, and create a community to create social proof in your organization. 

 

I think one of the most important things to understand and to see is that the most important person in social experiment is you. I mean, you're also part of a social environment. If you're going to be like very demanding, very System-1 driven, you have to do this because booking does it and competitors do it. You're not getting anywhere. And most of you will know Cialdini two of those apply to you as a person in your organization, be a likeable authority. 

 

So be passionate, but humble, be approachable, show interest in others. And as mentioned, respect people, respect their opinion and help them achieve their goals with experimentation instead of trying to achieve their own goals.

 

Gavin Bryant  23:40

Fantastic summary there, be humble, be curious, be approachable, be collaborative, and bring the organization together and rally around experimentation rather than being as you stated earlier, to be locked away in the closet.

 

Ruben De Boer  23:56

Exactly.

 

Gavin Bryant  23:58

Okay, rewards and incentives. This is a really interesting dimension of the quadrant. So what are some of the effective strategies that you've seen to align organizational rewards and strategies for those incentives?

 

Ruben De Boer  24:16

Yes, rewards can really make new behavior stick, its linking something we want with something we have to do. And of course, a couple of examples we can have there, we can tie nice rewards to our which test won competition. And we can celebrate again failures and success together with free foods and free drinks. But also, nominations for external awards can really work because if you get nominated for an external awards, it is a recognition of that you're doing a good job and you can take it with you within the company. But you can also tie experimentation to salary increase for instance, with your personal assessments. And it's very important to have consistent reward. 

 

If your organization state you have to experiment, that failure is not something we want. Of course, with experiments you're gonna feel, or you're gonna find make changes that don't work. So if you want, I'd say you have to experiment but failure is not accepted, then rewards contradict, and its results in less experimentation. 

 

One case I've seen really work was last year with a client. And with a team, we had to go to involve more people. We wanted more test ideas, ideas to test to come from the organization itself, one to have them think about how can we improve the product, how can we be more aligned with our customers and the problems they face on our platform. And we tried a couple of things. We shared a form to handling a test ideas during meetings during update emails and presentations, that maybe we got one or two ideas. That's when we started to work with the rewards and incentives, we set up an idea competition. 

 

And we had three rewards; one for individual reward, the best idea, and it was a health workshop, we had team awards, and one team have worked for, for the team that handed in the most and best test ideas. And another individual rewards for the one with the highest average points per test handed in and are quite nice rewards. 

 

I think we had a budget of, let's say, 700 euros, which is, I guess, 800-900 US dollars. And that, really, because the internal competition for teams and individuals, it's really started to become big and huge within the organization. We received test ideas from 50% of the organization. And I think on average, three test ideas per person. That's almost awesome. Everyone's involved because we have nice rewards tied to it. And the nice thing is, we know who hid it in the test ideas. So when we start testing their ideas, we can notify them, we automated, it's everything, of course, to increase the quantities, you have more time for more experiments, we notify people automatically when their idea becomes tested, and what the result is. And when results in the winner we can, for instance, bring a cake to their departments which they can share, which keeps experimentation top of mind. That was very successful case with rewards and incentives.

 

Gavin Bryant  27:30

Yeah, I think that's a really good point that when the organization contribute ideas, to innovation, product development and experimentation, then it's really important to keep them on the journey, and to keep them updated on the progress and the outcomes with their ideas. Not only does it provide them with development opportunities to grow and expand their skill, and capability, but then also continues to build that flywheel so that next time that there's a request for ideas that people are really willing to continue to provide the ideas because they know they're valued, and they're providing outcomes for the business. 

 

Ruben De Boer  28:11

Yeah, exactly. 

 

Gavin Bryant  28:12

One of the other ones that really interests me as well is is more formally connecting those rewards and incentives back to people's development plans and their annual review plan. And I've experienced this personally myself that, with an experimentation program that initially that those rewards and incentives were not aligned to the success and growth of the experimentation program, and drove behaviors that were counter intuitive to the growth of the experimentation program that those rewarding incentives were largely aligned around delivery of product and features. So it effectively drove a launch culture and a feature factory. 

 

However, it wasn't until probably two years later that those rewards and incentives more tightly aligned to the experimentation program and valued exploration discovery, and learning over delivery, that there was a real shift in the alignment of those behaviors to the growth of the experimentation program, and I think that this is a challenging one for organizations because on one hand, we need product teams delivering value for customers, but also on the other hand, that we also need the exploring and discovering and learning through experimentation in an iterative manner. So it's quite challenging to strike an even balance between both of those dimensions.

 

Ruben De Boer  29:48

Very much, I think it's an excellent example, and we have to keep in mind that of course rewards and incentives don't have to necessarily cost a lot of money. It can also be symbolic rewards but look at one of my clients. You know, as you mentioned, developers always deliver stuff. And then when there's some impact and increase in growth, and product owner starts presenting on what they achieve with their team. At that client, we decided to have developers not present the things they delivered. But the impact they made with their deliveries. So all of a sudden the developers got a stage, everyone got to see the impact they actually made, but also develop for themselves. So the impact they made instead of simply what they delivered. And that's when they also get more involved and are more proud and think about the impact they make, and come up with suggestions themselves.

 

Gavin Bryant  30:48

It's much more motivating for the team that their work is rewarded, valued and having commercial impact for the organization. 

 

Ruben De Boer  30:58

Exactly. 

 

Gavin Bryant  31:00

Okay, so let's think about the last element of the quadrant; rules and norms. What advice would you have for listeners on rules and norms?

 

Ruben De Boer  31:12

Yes, the rules and norms can be a bit more challenging for CRO specialist, for instance. But rules and norms can make certain behaviors mandatory, which is of course, think about the current COVID situation over last few years, we've shown very different behavior, due to all of the governmental regulations regarding Covid, wear face mask, keep your distance one meters, two meters, lines on the floor where to stand. And the same applies to the workplace actually arrive at work on time, clean up your desk at the end of the day, come prepared to meeting and hopefully at some point, do an A/B experiment for every change to the websites. But especially rules generally come from higher management. 

 

So again, it's very important to involve management as soon as possible. And friends, higher management can tie rewards for experimentation to personal assessments, but also from what is so within an online dialogue where I work. What someone did was, she set up a Fuckup Friday's what we call it. 

 

So, on the last Friday of each month, between 4pm and 5pm everyone closes their laptops, everyone stops working. And we share our biggest fuck ups of that month together. And other colleagues respond, of course, with a bit of laughter, because it's always fun, but also applause, empathy, and hey is there still something we can help you with, and this creates a very safe environment. So, it really sets the norm that failure is okay. And of course, this helps people to take risks to experiment and not stay on the safe side always.

 

Gavin Bryant  32:59

Yes, I think that's a really good point, and I'm seeing this to be a common theme across many successful experimentation programs, that there's open forums of sharing where there's a normalization of failure. And over time, that normalization of failure can then shift and change that culture to, as you pointed out an acceptable risk taking culture.

 

Ruben De Boer  33:26

Yeah, exactly. 

 

Gavin Bryant  33:28

Okay, so thinking about your pieces of advice for organizations that are just getting started with experimentation. They're on that runway, what are some of those key pieces of advice you'd give them to try to shift the organization and to manage that organizational change process?

 

Ruben De Boer  33:52

Yeah, good ones. So it's basically a summary of what we mentioned, we have to understand that we already have a lot of knowledge, most of us know about, or some of you know about System-1 and System-2, we tweak the physical environment of our website visitors every single day with our experiments with the goal to change our behaviour. We know about the influence of social proof sheets in our A/B test in our websites, on all big websites we get to and we also know for those who know Cialdini's The 6 Principles we know about likability and authority. And that applies to us as a person within our working environments. But most importantly, as mentioned a couple of times, we also know how to experiment we have that mindset, we have that experimentation, optimization mindset. And we should apply that in this case as well in our organization. 

 

Find out what works for your board of directors, for your other colleagues, find out experiments how to involve them. And of course, you're going to feel sometimes maybe people will hate your update email and straightaway send it to their spam folder, for instance, but then try something else, it's very important to understand that there's no 5, 6 or 7 steps to get to an experimentation culture, otherwise, it would be easy and every company would have done it. And then this podcast would not be so interesting. So the thing is, sit with your team, or experimentation enthusiasts, and brainstorm ideas based on this model, and plan and start experimenting, we already plan our A/B tests. Also plan your experiments within your organization and make it part of your stand ups. So people aren't take responsibility. And you can learn and see what works and doesn't work.

 

Gavin Bryant  35:47

Excellent point. Just to elaborate on that final point there about understanding what's learned that I think a good way to think about the experimentation program change process is also more formally as well. 

 

So thinking about that communication and change process. What is the communication and change strategy? Who do you need to speak to? When do you need to speak to them? What do you need to speak to them about? And what are the channels that you're going to utilize to be able to access all the stakeholders at every level of the organization, and every stakeholder group will have different needs and expectations and types of information are required. So I even think that formally sitting down and mapping out that strategy is also a very useful exercise for experimentation teams, and not to underestimate the communication and change process.

 

Ruben De Boer  36:48

Exactly.

 

Gavin Bryant  36:50

So fast 3 closing questions. Do you have an example of an experiment that you've performed that reframed organizational thinking, or organizational perspective?

 

Ruben De Boer  37:05

Love this one, often, it's not one A/B test. Of course, it's never about one A/B test. It's about the whole program, and in general, it's always a series of A/B tests confirming a hypothesis, which gives great insights which you can share, which are useful for the growth of the organization. But really to answer this question, when you have a redesign, you can have this one A/B test that changes people's perspective and beliefs about experimentation. So last year, for a client, we had an experiment, they were going through redesign on migration, new backends. And one of the plans was to have to check out in one full page. So the whole form in one page instead of several steps. And we tested that checkout. So several steps, several pages, and one long form, which was the plan for the redesign. And the long form resulted in a conversion drop of 12%, which is a lot of money. So when we presented this to the board, they really understood that, okay, experimentation is important, and we cannot just do a redesign without experimenting first, because then we would have cost a lot of money.

 

Gavin Bryant  38:16

Excellent example. 

 

No#2. Your top three resources. A book, websites, blog, or documents that you'd recommend to our listeners.

Ruben De Boer  38:30

Yeah, so number one, the one that I like the best myself is events, go to events and not just learn from the speakers on stage. But also share knowledge and learn together with the audience. Talk to each other, share experiences, and learn from each other. That's a very, very valuable source of information. 

 

Secondary resource newsletters. I have newsletter on my website with a free book, but also other curated newsletters, like Experimentation Nation newsletter, or Experimental Mind are excellent resources, they scan the web for good articles and they send them to you, saves you a lot of time. And for change management in general, my favorite book is "Switch: How to Change Things When Change is Hard from the brothers Heath, I can definitely recommend reading that.

 

Gavin Bryant  39:24

Excellent. And last one, where can listeners reach you if they'd like to get in contact?

 

Ruben De Boer  39:31

Well, everyone's always free to contact me because I love hearing stories and sharing experiences and learning together. So everyone is more than welcome to add me on, connect with me on LinkedIn. I post something on LinkedIn every single week about experimentation and experimentation culture, and everyone can always free to email me at [email protected]. More than welcome today to hear from you.

 

Gavin Bryant 39:57

Fantastic. Well, thanks so much for your time today, Ruben. We really appreciate it.

 

Ruben De Boer  40:03

Thank you.