Web-Based Usability Testing Rocks for Agile Software Development!

Are you doing web-based usability testing for your web-based software? If not, start now. Your user experience will improve rapidly!

How Can We Tell Whether Our User Interface Rocks or Sucks?

When you’re building a new application, one of the fundamental questions is “Will prospective customers understand and be able to effectively use the application’s user interface?” Members of the product development team (including product managers, engineers, and quality assurance) all quickly become poor judges of application usability for beginning users because development team members quickly gain experience with the application and become intermediate to advanced users of it. As a result, their mind becomes able to easily “fill in the blanks” on issues that will be stumbling blocks for new users.

Different members of the development team will each have their own theories about what users will find most understandable. But they’re just theories. If they disagree, experience has shown that passionate arguments not based on actual data are an unreliable means of determining the correct answer. Even when everyone on the development team agrees about the best user interface design, they may all be wrong. There’s simply no substitute for taking the product out of the development lab, putting it in the hands of real users, and seeing what ACTUALLY happens when external new users try to use the application for the first time.

Web-Based Usability Testing Versus Traditional Usability Testing

Traditionally, usability testing involved paying a usability testing firm $10-15,000. For this fee, they would identify a group of testers matching your target demographics and pay the users to come in to a physical facility one at a time to do a formal usability test in a room with one-way mirrors, a video camera recording their face, and a screen recorder capturing the screen. The firm would then analyze the findings and present you with a list of recommendations and an edited “highlights” video. I have used such firms, and they’re excellent. They’re also simply too expensive for a lean, self-funded startup.

Web-based usability testing services such as UserTesting.com provide a more cost-effective solution. UserTesting.com will find users matching your target demographic and conduct an online usability test recording the screen activity and the user’s voice for $39 per session.

Web-Based Usability Testing: A Natural Fit for Agile Software Development

I cannot overstate how helpful web-based usability testing is for Agile software development. Doing frequent usability testing has enormous benefits for any software development team, but it works especially well for teams doing Agile software development.

• You can get rapid feedback. You can post a usability test at the end of a long day of development and usually have your results back the following day. When testing services have testers in both the United States and other countries, you can benefit from time zone difference have people testing your software while your team sleeps. Agile software development methodology produces a potentially shippable version of the product at the end of each brief sprint. You can take that release, run an online usability test, and immediately find out how well new users actually understand and are able to use the user interface.
• Testing shortens arguments about user interface design by grounding them in data instead of theories and opinions. When engineering, product management, and/or design fail to agree about what is the best user interface design, there has historically been no cheap, quick, reliable way to break the deadlock of data-free opinions. Data from web-based usability testing breaks the Gordian knot of conflicting opinions. No matter how sure some team member is that a particular design is the best, if users can’t use it, you can see that right on the screen recording and the team member will have to let the facts trump their opinion. Frequent usability testing reduces the risk of a design being horribly, unusably wrong.

Sample Size Too Small for Statistical Significance? Who Cares?

All too often in business, one hears the following argument: “A test must have enough trials to generate statistically significant results for it to be of any value. We lack the time or money to do enough trials to produce statistically significant results. Therefore we should not do any tests.”

I’m glad that people are aware of the concept of statistical significance, but the above argument falls into the domain of “having just enough understanding to be dangerous.” People making such arguments are overlooking a couple of critical realities:

• In business, one rarely has the luxury of achieving 95% confidence that you’re correct. You can build an extremely successful business by being correct 80% of the time and quickly detecting and correcting your mistakes the other 20% of the time. People arguing for an “all or nothing” approach in testing often implicitly assume without thinking about the question that it’s necessary to be 95% confident that your conclusion is correct. It’s not.
• A sample size of zero is even less likely to be accurate than a small sample size. When you do no testing at all, you’re relying solely on your own opinions uninformed by any testing. As we already discussed, members of the development team are biased by their own knowledge and experience and unlikely to be good judges of what will work for users lacking that knowledge. Using small samples sizes doesn’t guarantee a good result, but using a sample size of zero does almost guarantee certain kinds of issues will be overlooked and certain kinds of bad decisions will regularly be made because of development team internal bias. Jakob Nielsen explains in greater detail why you only need to test with 5 users. Don’t Make Me Think similarly argues that frequent, unscientific tests of usability are a good way to improve an application over time. They’re both right. If you have the luxury of enough money and staff time to quickly do enough tests to generate statistically significant test results, great, but don’t let the lack of that stop you from doing ANY usability testing!

Examples of Things We’ve Learned Through Web-Based Usability Testing

We discovered through initial usability testing that we needed to make the following changes:

• Make some objects stand out more so that users could find them more easily.
• Move some objects so that users notice them more quickly.
• Reorganize some features in the user interface so their relationship to each other was clearer.
• Increase contrast on menu text so it was easier to read.
• Provide more affordances to ease navigation from one part of the application to another.
• Clarify the names of some objects so users could recognize them more easily.
• Turn various words in the user interface into links so users could use them to navigate as they expected they could.
• Remove placeholder elements in the user interface to avoid distracting and confusing usability testers.
• Add additional content to the online help.

UserTesting.com Rocks for Agile Software Development!

To date, we have been using UserTesting.com for our own testing. I am sure there are other, similar services out there, and they are welcome to post brief plugs for themselves in the comments section of this article. Here’s what I like about UserTesting.com:

• The price is right. Each usability testing session is only $39. This is well within the budget of even the leanest startup, and the results are more than worth it.
• You can get immediate, overnight feedback. They have testers in both the United States and some European countries, so you can post a test before going to bed and start watching the results the following morning. (If you’re really concerned about possible differences between American, British, and other users, you can specify which countries you want the testers to come from.)
• Satisfaction guaranteed or your money back. If you’re not satisfied with the performance of a usability tester, you can request a full refund of the testing fee. A single one of our first ten testers rushed through the test too quickly, didn’t read the instructions carefully, and got stuck as a result. We requested and got a refund on that test. The other nine were all excellent and well worth the cost. You can rate the performance of the testers and provide feedback comments as well. Presumably UserTesting.com will weed out any testers who get consistently poor reviews.

If you aren’t doing frequent web-based usability testing already, get started now. Your users will thank you!

Neither the author nor Voximate had any financial relationship with UserTesting.com at the time of publication other than Voximate being an ongoing customer for their services, and neither the author nor Voximate received any compensation for this blog post. UserTesting.com was not informed of this article in advance of publication and did not review or approve its contents.