Want to learn usability testing? I promise that any complete beginner who applies the lessons in this guide thoroughly will see a higher conversion rate in 30 days, guaranteed!
Q: What is usability testing? What is remote usability testing?
A: Usability testing is the process of having people interact with a website and provide feedback on it. The purpose of this testing is to find out why users are using the site as they are.
The value in understanding why users behave a certain way is that you get actionable insight, which raw data on what users are doing doesn’t tell you. You can see what’s tripping up users and fix it!
Note: Usability testing is also used offline for testing products, but for our purposes we’ll stick to website usability testing.
Remote usability testing is a method of usability testing where the users testing the website are in a different location than the person giving the test. The test giver, known as the moderator, sets up the test and provides instructions on what to do.
For now, let’s see how usability testing works. Later we’ll address specific issues like remote vs in-person usability testing, moderated vs unmoderated testing, and what tools to use for specific tasks.
How Usability Testing Works
Photo credit sxc
There are 5 steps to running a usability test.
1. First, you decides on tasks you want people to do on your website, such as search for a product and add to cart.
2. You recruit users, ideally ones representative of the site’s audience for the test. Typically, 3-5 testers are used in each round of testing, after which changes are made and another round of testing begins.
(If you can’t recruit users, there are companies who do that and usability testing tools with panels of users you can recruit.)
3. Users attempt to do the tasks set by the site owner.
In the most popular and effective form of usability testing, test-takers record their screen and voice, while sharing their thoughts out loud as they use the site. In other types of usability testing, feedback may only be written afterwards.
A short questionnaire typically follows screen-and-audio-recording type tests. Questions focus on problems encountered using the site, possible solutions, and how the tester would have behaved had it not been a test.
4. You review the user feedback to understand what the users were trying to do and why. Make note of the most common difficulties.
5. The site owner makes changes to the website to solve the problems discovered from the recordings.
Note: For ease of reference, I’m going to use “the moderator,” “site owner” and “you” interchangeably, but the roles can be shared between team members as you see fit.
Let’s see each step of the testing process in more depth.
Step 1: Deciding On And Writing Tasks For Users
The first step is figuring out and writing what you want your users to try to do. You need to describe outcomes for users to achieve, and avoid (to the extent possible) being too explicit in how to do a task, avoid mentioning the names of particular links to click (e.g. the task reads “learn pricing,” when the site has a “Pricing” link).
A good task description is, “Get in touch with us,” not “Click ‘contact’ and fill out the form.”
Photo credit sxc
Start by asking, what is the purpose of the site? Then work backwards and ask yourself what steps are needed to get a visitor to accomplish those steps.
The following example illustrates how this works at a coarse level, but you can go more granular and test sub-elements within each step.
Ecommerce example: Let’s imagine we’re usability testing a running shoe retailer.
6 – Purpose of the site: The site’s purpose is to sell running shoes.
5- Previous step: To sell merchandise, users need to checkout.
4 – Previous step: To checkout, they need to have added something to cart.
3 – Previous step: To have added something to cart, they need to have found a product that adequately meets their needs.
2 – Previous step: To find that product, the users must sort through the site’s products easily.
1 – First step: Before sorting, visitors need to be convinced not to bounce with a credible appearance that’s relevant to their intent in visiting.
For step 1, since users are coming to your site to do the test, they won’t bounce on their own. To find out if your site is credible and relevant, use a five second test task description:
Ask people to look at the page for five seconds, look away, then share what they remember. Does “what people remember” match the landing page’s traffic sources, such as search keywords and ad copy? Is the design credible or does it make people flee?
For steps 2-6, a regular usability test with screen and voice recording would work fine.
Good: “Find a pair of Reebok running shoes and buy them. Find answers to any questions that come to mind.” This leaves discretion to users to browse for Reebok running shoes the way they naturally would. This allows them the choice to browse with search or clicks, refine their options as usual… in sum, to be themselves, which is what we want to see.
Bad: “Click the Reebok running shoes button in the lower left sidebar, click on a pair of shoes, add them to cart, and fill out the checkout forms.”
Step 2: Recruiting Users
(image credit Vintage Military Ads)
This is where most web pros aspiring to run usability tests give up. Frankly, recruiting testers is hard.
First, it’s not always obvious who your audience is. Who are these “representative users” anyways?
To answer that, build personas (easier), and seek keyword-level demographics (harder). If you’ve never worked in the field, you can always email the owners of existing sites and ask them; if they’re competitors, look at
people offering the service in a different city or language.
Second, how do you go about soliciting people to test the site? Why would they care or bother? Where do you find them? How do you approach them?
Commonly, usability testing will pay testers for their time. With remote usability testing, paypal payments or online gift certificates (ex.: Amazon.com) are common. Others are interested in helping just to be helpful or because they’re friends.
Once you know who your representative users are, you need to find them. To do that, first try the free way and solicit amongst family, friends and contacts by email and phone.
After exhausting this pool or if your contacts just are too different from representative users (you’re starting a hispanic dating site without knowing anyone hispanic), you can run demographically or professionally targeted ads. Places to advertise include Facebook, PlentyOfFish, LinkedIn, MyAds (demographics powered by MySpace) and supposedly the Google Display Network (formerly the Content Network).
Originally, Ethnio was offered at a rate of a few hundred dollars per person because it was an offline, labour-intensive recruiting process. Today Ethnio is a software tool with a free trial that just requires you to copy-paste some code to get started. It intercepts visitors as they come to the site, asking them to participate in your test. The downside is that the test needs to consist of a survey or Usabilla click test, which helps but isn’t as useful as getting them to record their screen and voice.
Another easy way out of recruiting is buy a remote usability testing service with a panel of users.
Tools in this category that will have users record their screen and spoken thoughts include TryMyUI , UserFeel -whose panel includes testers in the UK and Greece- and UserTesting.com. (UI means User Interface, such as the part of the website customers interact with by clicking and typing.)
On a lower-tech level there is Feedback Army, which surveys visitors after they try using your site. While it doesn’t record audio or video, it’s also only $15 for 10 users to provide their feedback.
Note that these panels are only appropriate for sites geared towards a general audience. Sites requiring knowledge of particular jargon (e.g. SEO) won’t find representative users here, since user selection criteria are limited to demographics and tech-savvy (“technographics”), and don’t break down by vertical.
A word on “representative users”
Finally, while it’s ideal to get representative users, it’s also fine to get users who are less representative and grade on a curve, proportionately to how closely they match your audience. (Hattip Steve Krug)
Step 3: Running the Test
Once recruited, it’s up to users to follow instructions and do the test. This either happens at agreed-upon times (moderated testing), or at the user’s discretion (unmoderated usability testing).
With moderated testing, the moderator either sets up a computer with screen and audio recording software before the test, or instructs users to use browser-based software to record their screens and voices.
With unmoderated testing, the user is responsible for ensuring the audio and video recording software is on and recording, at the right level.
Note: Both of these comments assume a regular usability test, as opposed to a limited or partial test aimed at discovering how users interact with particular aspects of the site. This includes mouse-movement tracking and click measurement tests, for example, as well as visual analysis tools like FiveSecondTest or predictive gazeplot-and-eye-tracking tool Feng-Gui.
A gazeplot generated on Amazon’s mobile homepage by Feng-Gui.
Tip: It’s important to make sure that both video and audio are recording at the start, and that the sound levels are high enough, to avoid wasting time and money.
What Your Instructions Need To Cover
Once the audio and video are on and at the right level, there are three things your instructions must cover:
a. Asking the user to say what hes doing and why out loud, constantly. To quote TryMyUI:
“Clearly say exactly what you are thinking as you are thinking it. We are interested in your impressions, expectations, and the motivations for your actions. Don’t edit your thoughts as you navigate the website! Simply say exactly what you are thinking at each step.”
Steve Krug’s excellent book on usability testing, Rocket Surgery Made Easy, emphasizes that you should make clear it’s the website being tested, not the user. They can’t make mistakes here!
This serves the highly important purpose of preventing self-censorship, as alluded to in TryMyUI’s instructions.
b. Providing a starting URL for the user. With in-person testing, you’ll have this pre-loaded.
c. Provide a written list of tasks. Don’t just provide it orally, because you want to ensure the same phrasing each time.
If users get distracted and go off to do their own thing, the moderator gently encourages them back to the task at hand.
Step 4: Reviewing The Feedback
Once the tests have been run, what’s left is data – not actionable insight. To turn that information into actionable recommendations, the moderator needs to review the collected feedback.
His purpose is to look for patterns in what gave users difficulty.
By focusing on problems common to more than one user, you ensure that you get the most bang for your buck in making changes and fixing the site. This is also why running a test with one user is insufficient.
Conversely, to save time and money, you shouldn’t test with more than 5 users per round of testing. By the fourth or fifth user you’ll already know the main issues the site has and having the same problems highlighted another half dozen times isn’t productive.
The problems the moderator will discover can usually be categorized by WiderFunnel’s classic LIFT model of conversion optimization:
- Value Proposition: What’s in it for the user. This is the core which the other factors act upon.
- Anxiety: How design -security reassurances, layout- and content -answering questions, social proof- affect trustworthiness. Reduce anxiety to lift the conversion plane.
- Distraction: The page features too many eye-catching elements. Focus is drawn away from the page’s main purpose. Reduce distraction for more lift.
- Relevance: The degree to which the page matches users’ expectations before arriving. Increase relevance for more lift.
- Clarity: Do the design and content work in synergy to explain the value proposition?
- Urgency: Do users have a reason to act now?
To return to our earlier example of the online running shoe retailer, here are some things you would look for in your analysis.
First, as mentioned earlier, is the landing page’s credibility and relevance to traffic sources. Is the site secure? Is the graphic design professional? For relevance, you can show the traffic source (ad, link or search listing) briefly before taking users to the landing page. Within the five second test, you can ask if the landing page matches the expectations the traffic source created. – Relevance, Anxiety, Clarity -
Visitors need to drill down and find a product that suits their needs. Can they browse and search their way through your categories to find an adequate fit? How effective are your refinement options? When clicking through to a subcategory or product detail page, does the content there match expectations? – Relevance, Clarity -
Third, visitors need to get sufficient information from the product detail page. Does the information answer visitors’ questions? Does it convey benefits? How scannable is the information – bullet points or paragraphs? Where’s the refunds policy? Are the shoes in stock? How much is shipping? – Value Proposition, Urgency, Anxiety-
Users also need to be able to add to cart easily, and then modify the contents of the cart. Is the add to cart button noticeable and clearly a button? Are the cart buttons clearly labelled and laid out in a hierarchy reflecting their importance? Does the site need all this information the checkout requests? – Clarity, Distraction, Urgency, Anxiety -
Step 5: Implement The Changes
Again, follow Steve Krug’s excellent advice: Change the least possible to solve the problems.
For example, instead of redesigning a page’s graphics to reduce distraction, comment out some of the graphics. Test again and see if that solves things. To save time on such visual makeovers, you can just use a visual testing tool like Feng-Gui or Five Second Test, mentioned earlier.
And don’t wait for the “upcoming redesign”… 99% of the time it’ll go live months after promised. In the meantime, you’ve wasted loads of traffic.
Miscellaneous Testing Tips and FAQ
How do I test…
… information architecture (aka the organization of content on a site into logical groups)? Use card sorting. Do users look for content where you expected? Do they click the right links to drill down to the content you want them to find? (Image credit Revium.)
… how people make their way through a model of the site? Try wireframe creation software that (i) lets you interact with the wireframe by clicking the navigation and going to the wireframes for the relevant pages and (ii) allows you to share your wireframes online, so testers can access them. Some wireframe tools are designed to only be used in a desktop environment, so read carefully to avoid that.
Remote Testing vs In-Person Testing Pros and Cons
Remote testing is
- Cheaper: Don’t need to pay for travel, premises, computer equipment, food
- Less work to organize
- Easier to recruit for because there’s no geographic restriction
- Commonly paid for via Paypal or gift certificates
In-Person testing is
- More personal, you can get body language
- A more compelling opportunity for the whole team to witness the testing simultaneously and debrief, which can get things moving faster
- Commonly paid for in cash
- Instantaneous feedback; you view the data as it comes in, instead of waiting until later
“Moderated Testing” vs “Unmoderated Testing” Pros and Cons
- Is significantly less likely to encounter errors requiring retakes, such as setup issues with hardware or software, or testers veering off topic
- Yields richer, more useful data
- Tends to have representative users
- Sometimes yields obnoxious or useless responses, like FiveSecondTest and Feedback Army testers more interested in accumulating credits or micropayments than being helpful.
- Costs less on an individual test basis, because there is no moderator, or because it typically involves testing narrow interaction aspects such as click tests or visual feedback
- Is commonly associated with narrow aspects of interaction
- Tests of particular aspects may or may not have representative users
- Can save time when run through a service with a panel of testers, such that recruitment effort is negligible. This advantage makes it an easy entry-point/stepping stone for beginners to enter the world of usability testing.
Additional testing tools references:
There are many tools available for usability testing specific aspects of a website, or the whole shebang.
Kyle Soucy of Usable Interface put together a helpful, comprehensive overview of these varied tools.
Concluding Tip: Start as early in the web design process as possible
- Talk about the idea with friends and family and see what are the most common questions/objections raised, so you can answer them.
- Create user interface mockups in Photoshop and do the 5 second test (with or without the site): namely, ask people to look for a few seconds, then look away and ask what they recall.
- Put early design mockups through Feng-GUI as a sanity check – are there too many visually ‘loud’ areas?
- Imperfect testing is better than none. As the French say, “the best is the enemy of the good [enough].”