Beyond A/B testing: hypothesis testing for startups

We do a lot of A/B testing at SlideShare. Such tests (or at least the ones we run with Google website optimizer) are mostly tactical. They are about getting your call to action, the size of your button, or the copy of your landing page right.

While it’s important to do these experiments, they are mostly useful for refining ideas. As Andrew Chen would put it, you can get caught in local maxima if you focus only on them. The substantive decisions : to try a different product strategy, to build new functionality are not tested by A/B tests.

Coming from a scientific background, I have often wondered what other type of testing makes sense for startups. Scientists are used to testing major ideas, and significant advances through a rigorous, metrics based approach. Why should testing with startups be about simple refinements?

After talking Steve Blank and Eric Ries about testing at a Startup2Startup dinner, I had an Aha moment. I realized that the type of testing they advocate is about testing your vision against reality. This is very similar to what goes on in science, where you have to articulate your hypothesis in clear terms, identify your independent and dependent variables, and then test it.

For startups, that first articulation (or the core hypothesis) is often the founding vision. For example, we (founders of SlideShare) envisioned SlideShare to be a particular kind of sharing place. While that vision has evolved, but in many ways, SlideShare is what we first imagined it to be. I remember when Jon described the idea to us. The three of us had been bouncing startup ideas for months, but this was the first idea we completely agreed on. I remember the first mockups (done in paper then powerpoint strangely), and that guides us even today.

For the scientific community, that first articulation is the hypothesis. It could be very simple. X will lead to Y. For my PhD work, my hypothesis was that people with damage to hippocampus (a part of the brain related to memory) would be impaired at categorization tasks. (Yes, I came to startups from cognitive neuroscience!).

A founding vision for a startup is similar to a scientific hypothesis. It’s an articulation of a relationship between a product and a market. For SlideShare, the hypothesis was “People will want to share presentations on the web and with each other, and prefer this to email.” The first version of the product (what we launched with) was a test of this hypothesis. We built a basic product (very simple, just basic uploading, viewing, few social hooks).

The answer we received from the market (in a very short time) was a yes, “People do want to share presentations on the web”. But we could have received a different answer. For example, the answer could have been, “People are too sensitive about their powerpoint and will not share it in a public forum on the web”. In that case, we should have gone back to the table, and figured out how to share while taking care of concerns about public sharing.

The goal of the Minimum Viable Product should be to test the founding vision or initial hypothesis. You need to be open to different answers – the answer might be a yes, a qualified yes, or a no. By framing the founding vision like a hypothesis, you remain open to multiple answers.

However, hypothesis testing is not just relevant at the time of founding. It comes into play everytime you make a significant move, a change in direction, a new feature or product.

If you’re a startup, go beyond A/B testing. Think about testing your hypothesis.

16 responses to “Beyond A/B testing: hypothesis testing for startups

  1. Pingback: Twitter Trackbacks for Beyond A/B testing: hypothesis testing for startups « Rashmi’s blog [rashmisinha.com] on Topsy.com

  2. Nice sum up Rashmi! In many cases the A/B bucket tests often become over utilized and abused. However, their ease of deployment and clear yes/no results make them rather effective for hill climbing. Couple that with ‘big web company ______’ can test 1 million users within the hour and people go ‘wow’ all to easily.

    It would be good to see more hypothesis testing in effect. Many times, the problem isn’t even testing the hypothesis, but rather, one of instrumentation and metrics. So, one could expand your hypothesis by testing for how are slides shared? Is the referrer commonly gmail? or facebook? In some cases, many sites wont even instrument the ‘share this’ button.

    To date, we really aren’t sure how to measure engagement and we’re even worse at finding motivations. But its good to be thinking about what tool will accomplish the right job.

  3. @ayman
    What you are describing is the open ended exploration rather than hypothesis testing. In these cases, you are looking at the data post-hoc for trends (as with the where traffic came from case), which is often termed “fishing”.

    So that makes it three types of testing: A/B, hypothesis testing and trend-analysis (or fishing).

  4. At my company (granted a large one, not a start-up), the language of hypotheses is very strong. What’s often hard is to push a team to a statement that’s focused enough to be falsifiable. It’s great when I can do it.

    But I also like to leave space for observation and fishing.

  5. Bear in mind thought that hypothesis testing has its limitations and is influenced by pre-conceived ideas. It’s not even possible (or particularly useful) to “observe [a] system without any preconceived notions but these preconcieved notions can influence what is observed (as example see http://barefootbum.blogspot.com/2008/01/hypothesis-testing.html
    I just wonder if there is a way that Bayes Theorem can be used in this context?

  6. I keep want to start this comment with ‘good’ or ‘nice’ or ‘great’ but none of these seems strong enough, or appropriate enough for what you just posted.Just fantastic and mindblowing blog keep it up..!!!

  7. I think when you’re trying to start a new business, anything you do to establish that idea is a good idea!

    Too many entrepreneurs are just too dazzled by their own idea to really look beyond whether their customers are even excited about it.

    How many entrepreneurs do you know who can sum up their company’s value statement in less than 10 seconds? If you can’t do that – there’s a fundamental problem.

    Great post, Rashmi.

  8. Really insightful post Rashmi!

    As a PhD scholar in Marketing Management I appreciate your point of view of utilizing the concept of Hypothesis Testing while preparing for a start-up.

    But at the same time I’m having a doubt that I can’t resist asking. Though when the entrepreneur is sure about the vision having the method of Hypothesis Testing is good. But what happens in Hypothesis Testing (similar to many case in research as well :P) – once the hypothesis is decided – people tend to manipulate the data/findings to prove that HIS/HER hypothesis was right. Obviously, if the entrepreneur is open minded and ready to accept flaws in initial hypothesis – it is still good. But in case many of us are not – don’t you think an approach similar to iterative and incremental approach like they follow in software development – may help. Specially when the start up is in an uncertain environment where takes time to crystallize your vision.

    By the way, thanks a lot for the insightful and thought-provoking article. :D

  9. Hi Rashmi:

    Testing a product/business idea through a hypothesis-like fashion will help the team prepare for scenarios of acceptance/rejection. Valid.

    But often times, when you are a startup and a unique one with a hitherto untold story or functionality, its difficult to extrapolate the feedback from the audience to what they actually need.

    People are good at making a judgement call when they have a reference point against which they measure your product. If I dont put out my grand idea with all its glory and just open up incremental features for test, it might be a pragmatic, economical way of product development but not it may not result in valid feedback.

    In short, users may fail to understand the grand plan, without which they can reach completely opposite conclusions based on limited preview.

    Would love to know what you think!

  10. What do you think of services such as www. performable. com or other services that assist in doing a/b testing? You think they are worthwhile endeavors? Or do you think that you can accomplish more without them?

  11. Hey both services seem to be reputable. Go check them out – http://www.performable.com/ as well as the place where real people share test results – – http://www.abtests.com/

  12. bana kin inizce öğretecek tam yorum yazamıyorum.

  13. I know it’s hard to find good free music for videos and websites so thanks for sharing these resources with us

  14. Pingback: Facebook status trends 2010 « V E X E D

  15. Thank you, Rashmi. I appreciate you bringing the curiosity of science to entrepreneuring. Curiosity is what separates a theory from a hypothesis.

  16. Hi, we did this with Reedge.com we used our own a/b testing tool to figure out what people were looking for. First we made a very big website with lots of information and features (we did not even have at the time). Then monitoring all feature-groups and engagement per group we figured out what where the most popular groups of features.

    When we wanted to make those features we talked to beta users and made a top 5. Then used a/b testing to variate homepage messaging to see how to position the product (what features converted to benefits the best and which ones drove Sign-Ups).

    So using testing to get product market fit is a smart idea. Polls do not always work, we prefer the combination of polls and sign-up of payment behaviors to figure out preferences.