A/B testing is fooling you

Since Lean Startup became a fashion thinking for hipster entrepreneurs a Validation Run installed itself in our habits. Through (more or less…) Big Data Gathering we scan users behaviour on our site… Ok , I’m too judgmental here am I not? But hey, wait a little, awesome dreamers of successful startups, do you remember that “step-out-of-your-comfort-zone”- oh, sorry “out-of-the-building” I ment – attitude? How many times did you get out of the building to talk to real people ? How many times did you say “Nay, my users are not outside of my office door, they are … elsewhere. How much do you prefer to stay a little bit hidden behind data collected on your website to make an educated guess about what that data might tell you about your customers ? How may times you longed for a good “how-to” on implementing A/B testing. But you know what ? A/B testing is narrowing your options. Here is why I think it does so.

I wonder whether or not …

When was the last time you had this type of dilemma : “I wonder wether or not” .. do a specific action. Then eventually agonise on the decision. Then build Pros and Cons lists. Then throw them away ( or loose them…) , because we’re not quite happy with what that list is telling us. To be more specific, let me give here a set of examples , hopefully you will recognise yourself in one of them at least :
I wonder whether or not I should take that hob
I wonder whether or not we should let go this member of our team
I wonder whether or not I should buy a new smartphone
I wonder wether or not I should go into that mountain hiking
….

The “whether or not” situation is not a real decision making situation, because we position ourselves in a one-dimensional Two-ways option type. Having only one option is not an option. Having one option and its opposite it is not either.

The “whether or not” situation was coined beautifully by Dan and Cheap Heath , who say that “whether you are asking yourself “whether or not” step back”. You don’t have enough of the big picture. In this case focusing to much creates blind spots of other options we might have at hand.

“I wonder if whether A or B”

An “advanced” form of “I wonder if whether or not” is the “I wonder if A or B”. Now, this might seem different to you, but it is not really. You are still in a kind of one dimension decision making process, because not doing A means implicitly doing B. Choices are narrow and you’re stuck in your options.
So let’s see some examples here :
I wonder whether I should buy a more expensive smartphone or stick to a basic one
I wonder whether I should accept the offer from Harvard or Stanford
I wonder whether I should pick the blue shirt or the white one
I wonder if web users will like a green call to action button or a red one?
I wonder if I should write a new blog post or prepare dinner ,
I wonder if my customers want a call-back button or a chat space….
I hope that I recognised at least one of the situation you eventually were in. And I hope for you that the majority of you were scanning for the answer to question number 2 🙂
The business experimentation movement accelerated by LeanStartup has came up with receipt to answer questions like #4 and #6 in my example : The A/B testing! Yey, shiny! A/B testing says that we will implement non A or B , but A and B and then we wait and see.

The Answer To A Question That Was Not Asked

So here is the moment of gathering the data after the A/B testing. To all that implemented A/B testing I ask a question:
What did you ( really) learn?
The feed-back I have after each A/B testing starts with “hmmm…”.
Then it can go like this :
“It seems A has more hits than B. But B is used heavily from 8:00 to 9:00 am. We should understand why” or/and
“It seems that A has more hits , but hey , isn’t it because it’s right in the middle of the page. B has very few hits but it gets traction each time.”
So, the global conclusion is we have collected very interesting data, just we don’t have a clue what to do with it.
In the “Whether A or B” situation we are still in a narrow focus situation , where we only think of A or B as options. In the specific case of A/B testing tool, the results are confusing because they just feed a behaviour data that blows at our face because they are not in our narrow scope of focus. That data simply answers to questions that we didn’t fully ask. It’s like having a lot of indices , but no clue how to solve an enigma. The gathered data is just like the messages intercepted by the British secret services: encrypted by the Enigma machine during WWII they sound like gibberish.
Once again , stepping back to have a bigger picture is necessary.

Enigma Encryption Machine

The Vanishing Options Test

What if instead of picking from that 1-dimensional-2-ways options ( Yes/No, A/B) we just force our brain to unfocused a little bit to get a bigger picture? Because, one field where focusing doesn’t help is exploring (or identifying) real options.
My favorite tool to” unfocus” to unfold creativity ( ie new options) is the Vanishing Option test , also defined as such by Dan and Cheap Heath.
The test goes like this :

Imagine that all the options you have thought about are gone. E.g. you’re stuck with an Yes choice, there can be no A and nor B, or there is only A….
Now think at the following question :
What would you do in this situation to reach your goal?

Let’s take an example : imagine you’re sucked with a “light green colour/white text” call-to-action button on your page. Can’t change that! How would you improve your hits?

Leave The Data Basement

Collecting data is good, but remember, data is encrypted. Just like having the Enigma Machine, didn’t help allies to understand the messages, having a (BIG) data is simply not enough. We need a decryption key, don’t we? The bad news is this : the only decryption keys available for us are our own cognitive biases. So we turn gibberish to very probably distorted messages.
Nevertheless, there’s good news , and it’s called hope. As in many situations ( just like in Enigme decryption story, by the way), better answers come from changing perspective. There is one simple way to change perspective for data interpretation:
Leave the deep basement behind your complex data graphs screen and go observe real users in the light. Talk to them. Ask them why A ? What does B mean to them?

And enjoy the sun!

I’m organizing “Data Gathering and Inspection” workshops and can coach your teams on “Data UX driven Interpretation”, just contact me!