Lies, Damned Lies, and Statistics

Don’t believe everything that companies tell you.

Very impressive email, HubSpot. You obviously worked hard on it. One small problem – it’s not true. Those numbers are fabricated. Someone outed you.

Laura Ramos, formerly with Forrester Research and now with Xerox actually downloaded and read the survey. She was impressed with the quality of the study – better than most she had seen. “But what bothered me was how the vendor-sponsor exaggerated the results to a degree that I felt were unnecessary.”

Laura recommends people look at three things when reviewing vendor claims:

1. What the base? – How many participated in the study. The HubSpot headline claims 5,000, but the survey had 214 respondents and most were small companies.
2. Does the headline stand up when you scrutinize the data? – You need to compare the headline to the details. For instance, small numbers and lack of time frame can skew the data. The study results (page 12) show that 83% of the respondents and 32% of those show their leads increase 50%. But who are those respondents and how long did it take to get those results?
3. How credible is the source/authors? – Who wrote the report? Are they experts or not. In Hubspot’s case, it appears that the report was done by two MBA graduate degree candidates. While they did a fine job, does this give Hubspot the right to link the MIT brand to the report? That is quite a stretch.

The bottom line is that marketing automation is highly competitive and vendors find very tempting to highlight customer results to catch the eye of busy marketing executives. But don’t believe everything you read. Successful B2B marketing today is hard work, and requires great processes and awesome content. As Laura says, software should be a distant fourth on the list.

What do you think? Do you know of vendors who skew the data? We love comments.

Sources:

Hubspot