Thanks and I look forward to your next insightful comment on our blog. Always helps to get people like you to ask the 'real' questions and point out some weaknesses in one's reasoning. Helps my learning a great deal. Thank you.

Usability is a very broad concept, and we stay very focused on conversion optimization… so we do Usability concepts to generate hypotheses worth testing. You clarification has helped me see this more clearly.

Best regards,

Raquel Hirsch

Thanks for your comment, I really appreciate the feedback. I am not sure if I miss the point with A/B testing totally althought if I do I have done it wrong for the last 20 years and that would be terrible for our clients if it were this way. I think some of your comments may have been triggered by my lack of communication skills.

Hence, let me make an attempt to clarify some issues. You state:
“If you define a conversion objective, testing will allow you to find a variation that improves your conversion rate, no doubt about that. It may not “guarantee the best solution for design and copy” – but you are very likely to get an improvement on what you currently have – before you run your next test and improve on it! “

My response would be that it is not always possible to assess things as a simple challenge of improvi8ng the conversion rate. For instance, designing a website like ours does not mean that all we want is that people sign up for http://My.ComMetrics.com (upper right button) but we also want them to read the material and do a few more things.

“If you “ask your clients for feedback” only outliers will respond. Do you want to hear nice things or do you want to convert more web visitors and increase revenues? “

I disagree that only outliers will respond. If you have created a level of trust with your clients and associates, call them up, visit them and do a structured interview to find out why they made the choices they did.

The above structured interviewing will reveal a ton of information simple A/B testing (multivatiate or not) will never give you. This does, of course as you point out as well, not absolve you from designing a methodologically sound A/B test in the first place. We surely agree on this one.

“”A/B tests can tell you how the bounce rate of those shown design A compares to design B” – the bounce rate is a poor metric of success. Infinitely better to use a tool like GWO and measure actual conversion actions.”

Totally agree with your statement above…. I apologise if my comment might have suggested that I rely on the bounce rate only.

I especially recommend your case studies

===> http://www.widerfunnel.com/proof/case-studies

Thanks, Raquel for sharing your important insights and giving me a chance for clarifying may point due to the lack of my communication skills.

However, with respect, I think you are totally missing the point of A/B/n (and multivariate, for that matter) testing.

If you define a conversion objective, testing will allow you to find a variation that improves your conversion rate, no doubt about that. It may not “guarantee the best solution for design and copy” – but you are very likely to get an improvement on what you currently have – before you run your next test and improve on it! More info here http://www.widerfunnel.com/our-process/ab-split...

If you “ask your clients for feedback” only outliers will respond. Do you want to hear nice things or do you want to convert more web visitors and increase revenues?

“A/B tests can tell you how the bounce rate of those shown design A compares to design B” – the bounce rate is a poor metric of success. Infinitely better to use a tool like GWO and measure actual conversion actions.

To succeed in a Conversion Optimization strategy you need an air-tight process. Here is an example of one: http://www.widerfunnel.com/our-process/website-...

Interested in case studies? Go here: http://www.widerfunnel.com/proof/case-studies

Thanks,

Raquel Hirsch
WiderFunnel Marketing Optimization

Thanks so much for your comment.  I agree with what you state and would add that successful A/B testing requires that we control for:

- mediating AND
- moderating

variables.  Failure to do so will cause us a lot of problems because we may no longer be sure what caused what.

Another concern I have with much of the A/B testing is that whilst there may be a correlation between factor X and Y, it does not mean that X causes Y. In other words, just because people dislike the color yellow does not mean this causes to dislike them design A and thereby lets them choose B instead. It is possible that something much more important triggered this decision.

Accordingly, unless we digg deeper we will not find out what causal relationships might exist.  In turn, A/B testing is not a cure all as many people might want us to make believe.  It takes a bit more to get the design or web shop that serves your clients' needs best (PS: consider cross-cultural differences as well.

Thanks so much for sharing. Please have a look at this blog post where I try to further explore these important issues you raised.

Using great visuals = Failing your customers

But maybe I was not clear in my explanation, we used both A/B testing and then interviews to find out more.

A/B testing is not an exact science so I am not sure if using the word “proof” is correct in this context. Maybe what we can say is that one failed to disconfirm the null hypothsis (Design 1 and 2 are the same) or one was able to reject it.

In that regard, I describe how we use additional information to get a better handle on the reason why Design 1 was favored over Design 2.

What we found was that sometimes it is a little thing such as the font that leads to the rejection or acceptance of a design. Re-testing with a changed letter font may then result in Design 2 being favored over Design 1. Put differently, just because the font is not ideal with Design 2 does not mean it is weaker than Design 1. A classical A/B test will not tell us why people preferred Design 1. In fact, after changing the font things may be different.

In our case, why throw the baby out with the water when a small tweak or adjustment can allow us to use a possibly better design that was rejected using a classical A/B test.

I hope that helps clarify my post above and I appreciate you taking the time to share your experiences and thoughts. Look forward to your next comment on our blog. Thanks again and have a nice weekend.

You have this interesting piece of information which you can directly apply or you can A/B test against your existing version.

I prefer the latter case because no proof is higher proof than an idea succeeding in the wild (and not just in customer's mind).

Again, A/B testing is a methodology – you have to wiggle your grey cells in order to come up with hypotheses and usability tests, feedback, expert opinion can be a great source for those hypotheses.