Beyond A/B testing: hypothesis testing for startups

We do a lot of A/B testing at SlideShare. Such tests (or at least the ones we run with Google website optimizer) are mostly tactical. They are about getting your call to action, the size of your button, or the copy of your landing page right.

While it’s important to do these experiments, they are mostly useful for refining ideas. As Andrew Chen would put it, you can get caught in local maxima if you focus only on them. The substantive decisions : to try a different product strategy, to build new functionality are not tested by A/B tests.

Coming from a scientific background, I have often wondered what other type of testing makes sense for startups. Scientists are used to testing major ideas, and significant advances through a rigorous, metrics based approach. Why should testing with startups be about simple refinements?

After talking Steve Blank and Eric Ries about testing at a Startup2Startup dinner, I had an Aha moment. I realized that the type of testing they advocate is about testing your vision against reality. This is very similar to what goes on in science, where you have to articulate your hypothesis in clear terms, identify your independent and dependent variables, and then test it.

For startups, that first articulation (or the core hypothesis) is often the founding vision. For example, we (founders of SlideShare) envisioned SlideShare to be a particular kind of sharing place. While that vision has evolved, but in many ways, SlideShare is what we first imagined it to be. I remember when Jon described the idea to us. The three of us had been bouncing startup ideas for months, but this was the first idea we completely agreed on. I remember the first mockups (done in paper then powerpoint strangely), and that guides us even today.

For the scientific community, that first articulation is the hypothesis. It could be very simple. X will lead to Y. For my PhD work, my hypothesis was that people with damage to hippocampus (a part of the brain related to memory) would be impaired at categorization tasks. (Yes, I came to startups from cognitive neuroscience!).

A founding vision for a startup is similar to a scientific hypothesis. It’s an articulation of a relationship between a product and a market. For SlideShare, the hypothesis was “People will want to share presentations on the web and with each other, and prefer this to email.” The first version of the product (what we launched with) was a test of this hypothesis. We built a basic product (very simple, just basic uploading, viewing, few social hooks).

The answer we received from the market (in a very short time) was a yes, “People do want to share presentations on the web”. But we could have received a different answer. For example, the answer could have been, “People are too sensitive about their powerpoint and will not share it in a public forum on the web”. In that case, we should have gone back to the table, and figured out how to share while taking care of concerns about public sharing.

The goal of the Minimum Viable Product should be to test the founding vision or initial hypothesis. You need to be open to different answers – the answer might be a yes, a qualified yes, or a no. By framing the founding vision like a hypothesis, you remain open to multiple answers.

However, hypothesis testing is not just relevant at the time of founding. It comes into play everytime you make a significant move, a change in direction, a new feature or product.

If you’re a startup, go beyond A/B testing. Think about testing your hypothesis.

Why VCs don’t invest in women led startups – my perspective

I just noticed lots of tweets citing me as an example of VCs investing in women led startups, and felt that I had to speak up about the issue. First of all, some VCs do invest in women led startups and don’t distinguish between women and men leaders. The VCs who funded SlideShare for example. Venrock has a history of funding women. Specifically, for David Siminoff and Dev Khare who funded SlideShare – a woman leading the company was never an issue. David Siminoff has funded other women as well, e.g., Lisa Stone from Blogher (I knew about that when I met him, and it did influence my going with Venrock). All our angel investors, Dave McClure, Hal Varian, Ariel Poler, Saul Klein, Mark Cuban – none of them cared one way or the other that SlideShare CEO is a woman.

So note to women entrepreneurs – there are VCs out there who will fund you. My biggest takeaway – look for a history of funding women. If someone has been a VC for 10 years and has never funded a woman, chances are they will not fund you. If a VC firm cannot speak of women entrepreneurs they have supported, take that as a sign and move on. If you find out that they have replaced their women founders as CEO’s, take that as a sign and move on.

In looking for funding for SlideShare, there were a few VCs who I knew after one meeting – that they would never fund a woman. In looking back, what was common to these VCs? I don’t think they understood the leadership style of women. Women can have a different leadership style and if you don’t understand that style, it can be a blocker in funding women. I felt this time and again – they don’t how I operate, how I lead. I think they felt I was weak, when it was just a different style. I don’t think they were against women CEOs, rather it was the particular leadership style they did not want to fund.

This is something I know through my psychology background as well – men and women can have different leadership styles. Culturally, we are sensitized to identifying a male style as a leadership style. And VCs in particular, seem to most understand leadership styles of 22 year old, male Stanford students (yes, I am stereotyping here :-)).

Having said that, I want to emphasize that overall I don’t feel discriminated against. I know that statistically speaking women have a lesser chance of getting funded. But I also have lots of male entrepreneurs friends – it can be hard for them as well. And some of the stylistic biases can operate against them as well.

Women Entrepreneurs: there are VCs out there who do fund women. Look for them. Look for a history of funding women. The best way to predict the future, is to look at the past. Ask them for a list of women entrepreneurs they have funded.