Why You Need Anecdotal Evidence

Have you ever heard the phrase "That's just anecdotal"? Usually, it's used to discount somebody's opinion about the way something should be. In UX terms, if you've observed a user, saw them have a problem, and want to respond to what you saw, you may have someone tell you it's just anecdotal -- in other words, you don't have hard, objective, statistical evidence, so forget it.

But there's a contradiction here. There's a belief that, if you gather enough anecdotal evidence, it magically becomes hard evidence. You can even see this in the latest ad campaign for Windows. Millions of people sent in their ideas, or were surveyed or studied, and their opinions -- anecdotal evidence -- is now real evidence. How does this magic work? Well, it can't. Two people who said different but similar things, or even the same things under different conditions, can't be lumped together in some statistical box.

But that doesn't mean that anecdotal evidence is worthless. Quite the contrary. It's invaluable (and Microsoft should be commended for actually listening to users). Anecdotal evidence provides you with something that hard data can't -- feelings. In a typical usability test or, these days, a web site A/B test, success is measured by whether or not somebody succeeds in a task. How about whether they frowned or smiled, tapped their fingers on their desk impatiently, hummed to themselves, or cursed at their computer?

I recommend that you gather as much anecdotal evidence as you can and let it infect your world view. Try to feel what your users feel, think like they think, and use that to design your products. And, after that, gather hard evidence on whether you were right or wrong and move forward from there. But, if you don't start with a feeling about what's the right thing to do, no amount of hard evidence will help you.