Take a second and think of how many experiments your team ran last year. This includes actual A/B tests, ideas that you implemented and new product features. What if you did half of that number this year but focused on increasing the probability that each of those ideas will have a significant impact on your KPIs?
Most teams are running or spending too much time on their experiments. The culture of experimentation has been embraced and companies now think in terms of volume. If we could run 100 tests instead of 50, we could double our growth.
Reality doesn’t work like that because as you increase volume, slopiness seeps in. I constantly see companies that are running A/B tests that are unlikely to make significant changes on their KPIs but the team feels good because they got something out. They are “hustling” and getting stuff done.
Let’s look at how to set up a culture of meaningful growth, not experimentation.
Experimentation is Good But Not For Its Own Sake
To get there, you will need to experiment but whether that is done through 5, 10, or 50 experiments is beside the point.
In the NBA, the Philadelphia Sixers are knowing for their “trust the process” mantra. If you focus on doing the right things, you will eventually succeed. This is fine but let’s not lose track of the actual goal of winning championships.
If you’re consistently failing to get here, you need to evaluate and adjust. The Sixers haven’t won a championship yet but came close in 2019.
Don’t Fetishize Failure, Aim to Win
I get the intent behind phrases like “fail fast, fail often”. You’re trying to get over the fear of making a mistake which can paralyze your team from making any decision.
However, I have seen this taken too far when companies become focused on how often they are failing. They assume that as the failure rate increases, they will get something right. Mark Suster has a great argument against this mentality:
“We have taught a generation of young entrepreneurs that “failing fast” is ok. It’s quick and easy. It’s a way out so that you can focus on your next big business idea. Why waste your time on this one?” – Source.
Figure out what success looks like and aim for that. Design experiments that could work instead of playing roulette with your ideas.
Run Fewer But Higher Quality Experiments Even If it Takes Longer
So how you should approach experiments? Fewer and higher quality is the answer here. Let’s get some basics out of the way:
- Run tests for 2-3 weeks unless you have extremely high traffic
- Aim for a 95% statistically significance on your results
- Get a few hundred conversions per variation
- Don’t go extremely micro e.g. copy on a CTA button unless you’re Google
If you take these guidelines, you will be instantly limited to how many experiments you can run at any given time. That’s the point. Don’t assume that you have unlimited bullets, instead be selective with your experiments.
Instead of hoping for the best, do the work upfront. Think about the biggest areas for impact within your customer journey (this is rarely the copy of a CTA) and what has a high probability of success.
Time spent thinking of tests upfront will pay off in the long term as you get more valid results that are actually worth implementing.
Plan for Exploiting Success
Lastly, let’s plan for exploiting success. Did you run an experiment with email campaigns that drove a high number of sales? Great, how can you double down on this and run a similar experiment?
Instead of jumping in between random areas e.g. email and paid ads, focus on exploiting success.
A client of mine recently had a product launch and decided to run surveys for existing users. They provided multiple choices that were all negative on the new product. They assumed that people will only give negative feedback so they never planned for positive responses. You need to plan for success and assume most people will likely something.
Let’s move away from a culture where failure equals success. Winning is success and we should do our best to get there. We will miss the mark on occasion but let’s plan for success, not hope for failure.