7 A/B and MVT mistakes... i've made. Yep.


San this post:

I'll be honest. My initial intend was to write a "5 mistakes you'll make in A/B Testing". But first: there are tones of these posts, and second... what do i know about other's mistakes?

So let's be clear and look back to make these #fails - i once or another, have felt in to- become the best of the advices for you:

  1. Not having an Hypothesis. This is what i call: "mAyBe testing", it is: testing "by instinct" without trying to understand the causation of the improvement. You just test a Call to Action, and "let's see". The test is the way to prove your hypothesis. Learn about it in this post on the Scientific Method. CRO is all about it.
  2. Getting obsessed with quickwins. I've really been there. Reading the conversion experts and looking for the conversion-doubler, wasted much of my time. Finally i understood that we are not all starting in the same point. The most stablished, trusted, and satisfying your website is, the toughest it'll be to increase conversion. Remember then that improving 10 parts of your Landing by 3% equals a 30% improvement.
  3. Copying other's tests. This is lead by mistakes #1 and #2. At beginning as you have no hypothesis, any test is OK. Totally wrong. Forget the "10 tests you must do in your Landing". May work for their audience, sector, season.... yours is different. Listen to your visitors. Get to know your target and their motivations.
  4. Just testing (not changing big things). MVT with tools like Monetate and Optimizely is a dream come true. These tools can make you increase your conversion rate to a particular point in a simple and clean way. But sometimes the best test is to be done against a new solution, an A/B testing. Don't get mad struggling to improve your landing page 1% when the action is to begin a  redesign from the start.
  5. Data dredging. You know: "If You Torture The Data Enough, It Will Confess Anything". Your hypothesis can be wrong. Really. (Don't miss this brilliant post on A/B testing by Lance Kidwell about the subject). Sometimes i found my self testing again and again an hypothesis that i wanted to prove true. Then i remembered that UX saying "People wont' necessarily use things the way  you deisgned".
  6. Forgetting the user: It is, focusing only in Layout and UX, and not testing copy's, UPV's, special offers, FUD's, etc...
  7. Confusing Correlation and causation. You know, pirates and global warming. We just forget it! Or at least i do (sometimes). We say "the test proves that doing this we get this" in stead of saying "the test proves that there's a correlation between this and this. Let's find out if there's causation". This is so important in CRO and LPO that we'll get on this subject deeper in another post.

So... yes, these are some of the mistakes i've done, and i must be focused not to done them again. Learning from other's mistakes is cheap and fast, but nothing like experience will give you "the edge". Don't be afraid of mistakes. Just fail fast.

That said... ¿Did you do any of these mistakes? ¿Any other mistakes you comitted?