We do a lot of A/B testing at SlideShare. Such tests (or at least the ones we run with Google website optimizer) are mostly tactical. They are about getting your call to action, the size of your button, or the copy of your landing page right.
While it’s important to do these experiments, they are mostly useful for refining ideas. As Andrew Chen would put it, you can get caught in local maxima if you focus only on them. The substantive decisions : to try a different product strategy, to build new functionality are not tested by A/B tests.
Coming from a scientific background, I have often wondered what other type of testing makes sense for startups. Scientists are used to testing major ideas, and significant advances through a rigorous, metrics based approach. Why should testing with startups be about simple refinements?
After talking Steve Blank and Eric Ries about testing at a Startup2Startup dinner, I had an Aha moment. I realized that the type of testing they advocate is about testing your vision against reality. This is very similar to what goes on in science, where you have to articulate your hypothesis in clear terms, identify your independent and dependent variables, and then test it.
For startups, that first articulation (or the core hypothesis) is often the founding vision. For example, we (founders of SlideShare) envisioned SlideShare to be a particular kind of sharing place. While that vision has evolved, but in many ways, SlideShare is what we first imagined it to be. I remember when Jon described the idea to us. The three of us had been bouncing startup ideas for months, but this was the first idea we completely agreed on. I remember the first mockups (done in paper then powerpoint strangely), and that guides us even today.
For the scientific community, that first articulation is the hypothesis. It could be very simple. X will lead to Y. For my PhD work, my hypothesis was that people with damage to hippocampus (a part of the brain related to memory) would be impaired at categorization tasks. (Yes, I came to startups from cognitive neuroscience!).
A founding vision for a startup is similar to a scientific hypothesis. It’s an articulation of a relationship between a product and a market. For SlideShare, the hypothesis was “People will want to share presentations on the web and with each other, and prefer this to email.” The first version of the product (what we launched with) was a test of this hypothesis. We built a basic product (very simple, just basic uploading, viewing, few social hooks).
The answer we received from the market (in a very short time) was a yes, “People do want to share presentations on the web”. But we could have received a different answer. For example, the answer could have been, “People are too sensitive about their powerpoint and will not share it in a public forum on the web”. In that case, we should have gone back to the table, and figured out how to share while taking care of concerns about public sharing.
The goal of the Minimum Viable Product should be to test the founding vision or initial hypothesis. You need to be open to different answers – the answer might be a yes, a qualified yes, or a no. By framing the founding vision like a hypothesis, you remain open to multiple answers.
However, hypothesis testing is not just relevant at the time of founding. It comes into play everytime you make a significant move, a change in direction, a new feature or product.
If you’re a startup, go beyond A/B testing. Think about testing your hypothesis.