The Beginner’s Guide To Remote Usability Testing: Increase Conversion In 30 Days With No Experience

Author: Gab Goldenberg

Want to learn usability testing? I promise that any complete beginner who applies the lessons in this guide thoroughly will see a higher conversion rate in 30 days, guaranteed!

Q: What is usability testing? What is remote usability testing?

A: Usability testing is the process of having people interact with a website and provide feedback on it. The purpose of this testing is to find out why users are using the site as they are.

The value in understanding why users behave a certain way is that you get actionable insight, which raw data on what users are doing doesn’t tell you. You can see what’s tripping up users and fix it!

Note: Usability testing is also used offline for testing products, but for our purposes we’ll stick to website usability testing.

Remote usability testing is a method of usability testing where the users testing the website are in a different location than the person giving the test. The test giver, known as the moderator, sets up the test and provides instructions on what to do.

For now, let’s see how usability testing works. Later we’ll address specific issues like remote vs in-person usability testing, moderated vs unmoderated testing, and what tools to use for specific tasks.

How Usability Testing Works

Photo credit sxc

ladder

There are 5 steps to running a usability test.

1. First, you decides on tasks you want people to do on your website, such as search for a product and add to cart.

2. You recruit users, ideally ones representative of the site’s audience for the test. Typically, 3-5 testers are used in each round of testing, after which changes are made and another round of testing begins.

(If you can’t recruit users, there are companies who do that and usability testing tools with panels of users you can recruit.)

3. Users attempt to do the tasks set by the site owner.

In the most popular and effective form of usability testing, test-takers record their screen and voice, while sharing their thoughts out loud as they use the site. In other types of usability testing, feedback may only be written afterwards.

A short questionnaire typically follows screen-and-audio-recording type tests. Questions focus on problems encountered using the site, possible solutions, and how the tester would have behaved had it not been a test.

4. You review the user feedback to understand what the users were trying to do and why. Make note of the most common difficulties.

5. The site owner makes changes to the website to solve the problems discovered from the recordings.

Note: For ease of reference, I’m going to use “the moderator,” “site owner” and “you” interchangeably, but the roles can be shared between team members as you see fit.

Let’s see each step of the testing process in more depth.

Step 1: Deciding On And Writing Tasks For Users

check list

The first step is figuring out and writing what you want your users to try to do. You need to describe outcomes for users to achieve, and avoid (to the extent possible) being too explicit in how to do a task, avoid mentioning the names of particular links to click (e.g. the task reads “learn pricing,” when the site has a “Pricing” link).

A good task description is, “Get in touch with us,” not “Click ‘contact’ and fill out the form.”

Photo credit sxc

Start by asking, what is the purpose of the site? Then work backwards and ask yourself what steps are needed to get a visitor to accomplish those steps.

The following example illustrates how this works at a coarse level, but you can go more granular and test sub-elements within each step.

Ecommerce example: Let’s imagine we’re usability testing a running shoe retailer.

6 – Purpose of the site: The site’s purpose is to sell running shoes.

5- Previous step: To sell merchandise, users need to checkout.

4 – Previous step: To checkout, they need to have added something to cart.

3 – Previous step: To have added something to cart, they need to have found a product that adequately meets their needs.

2 – Previous step: To find that product, the users must sort through the site’s products easily.

1 – First step: Before sorting, visitors need to be convinced not to bounce with a credible appearance that’s relevant to their intent in visiting.

For step 1, since users are coming to your site to do the test, they won’t bounce on their own. To find out if your site is credible and relevant, use a five second test task description:

Ask people to look at the page for five seconds, look away, then share what they remember. Does “what people remember” match the landing page’s traffic sources, such as search keywords and ad copy? Is the design credible or does it make people flee?

For steps 2-6, a regular usability test with screen and voice recording would work fine.

Task description

Good: “Find a pair of Reebok running shoes and buy them. Find answers to any questions that come to mind.” This leaves discretion to users to browse for Reebok running shoes the way they naturally would. This allows them the choice to browse with search or clicks, refine their options as usual… in sum, to be themselves, which is what we want to see.

Bad: “Click the Reebok running shoes button in the lower left sidebar, click on a pair of shoes, add them to cart, and fill out the checkout forms.”

Step 2: Recruiting Users

Army Recruiting Ad: Wanted: More Men Like Mike

(image credit Vintage Military Ads)

This is where most web pros aspiring to run usability tests give up. Frankly, recruiting testers is hard.

First, it’s not always obvious who your audience is. Who are these “representative users” anyways?

To answer that, build personas (easier), and seek keyword-level demographics (harder). If you’ve never worked in the field, you can always email the owners of existing sites and ask them; if they’re competitors, look at
people offering the service in a different city or language.

Second, how do you go about soliciting people to test the site? Why would they care or bother? Where do you find them? How do you approach them?

Commonly, usability testing will pay testers for their time. With remote usability testing, paypal payments or online gift certificates (ex.: Amazon.com) are common. Others are interested in helping just to be helpful or because they’re friends.

Once you know who your representative users are, you need to find them. To do that, first try the free way and solicit amongst family, friends and contacts by email and phone.

After exhausting this pool or if your contacts just are too different from representative users (you’re starting a hispanic dating site without knowing anyone hispanic), you can run demographically or professionally targeted ads. Places to advertise include Facebook, PlentyOfFish, LinkedIn, MyAds (demographics powered by MySpace) and supposedly the Google Display Network (formerly the Content Network).

Another option that recruits directly from your users is Bolt|PetersEthnio recruiting service.

Originally, Ethnio was offered at a rate of a few hundred dollars per person because it was an offline, labour-intensive recruiting process. Today Ethnio is a software tool with a free trial that just requires you to copy-paste some code to get started. It intercepts visitors as they come to the site, asking them to participate in your test. The downside is that the test needs to consist of a survey or Usabilla click test, which helps but isn’t as useful as getting them to record their screen and voice.

Another easy way out of recruiting is buy a remote usability testing service with a panel of users.

TryMyUI logoTools in this category that will have users record their screen and spoken thoughts include TryMyUI , UserFeel -whose panel includes testers in the UK and Greece- and UserTesting.com. (UI means User Interface, such as the part of the website customers interact with by clicking and typing.)UserTesting logo

On a lower-tech level there is Feedback Army, which surveys visitors after they try using your site. While it doesn’t record audio or video, it’s also only $15 for 10 users to provide their feedback.

Note that these panels are only appropriate for sites geared towards a general audience. Sites requiring knowledge of particular jargon (e.g. SEO) won’t find representative users here, since user selection criteria are limited to demographics and tech-savvy (“technographics”), and don’t break down by vertical.

A word on “representative users”

Finally, while it’s ideal to get representative users, it’s also fine to get users who are less representative and grade on a curve, proportionately to how closely they match your audience. (Hattip Steve Krug)

Step 3: Running the Test

Once recruited, it’s up to users to follow instructions and do the test. This either happens at agreed-upon times (moderated testing), or at the user’s discretion (unmoderated usability testing).

Test Preparation

With moderated testing, the moderator either sets up a computer with screen and audio recording software before the test, or instructs users to use browser-based software to record their screens and voices.

With unmoderated testing, the user is responsible for ensuring the audio and video recording software is on and recording, at the right level.

Note: Both of these comments assume a regular usability test, as opposed to a limited or partial test aimed at discovering how users interact with particular aspects of the site. This includes mouse-movement tracking and click measurement tests, for example, as well as visual analysis tools like FiveSecondTest or predictive gazeplot-and-eye-tracking tool Feng-Gui.

A gazeplot generated on Amazon’s mobile homepage by Feng-Gui.

Tip: It’s important to make sure that both video and audio are recording at the start, and that the sound levels are high enough, to avoid wasting time and money.

What Your Instructions  Need To Cover

Once the audio and video are on and at the right level, there are three things your instructions must cover:

a. Asking the user to say what hes doing and why out loud, constantly. To quote TryMyUI:

“Clearly say exactly what you are thinking as you are thinking it. We are interested in your impressions, expectations, and the motivations for your actions. Don’t edit your thoughts as you navigate the website! Simply say exactly what you are thinking at each step.”

Steve Krug’s excellent book on usability testing, Rocket Surgery Made Easy, emphasizes that you should make clear it’s the website being tested, not the user. They can’t make mistakes here!

This serves the highly important purpose of preventing self-censorship, as alluded to in TryMyUI’s instructions.

Gab with Rocket Surgery Made Easy and Don't Make Me Think, by Steve Krug

b. Providing a starting URL for the user. With in-person testing, you’ll have this pre-loaded.

c. Provide a written list of tasks. Don’t just provide it orally, because you want to ensure the same phrasing each time.

If users get distracted and go off to do their own thing, the moderator gently encourages them back to the task at hand.

Step 4: Reviewing The Feedback

Once the tests have been run, what’s left is data – not actionable insight. To turn that information into actionable recommendations, the moderator needs to review the collected feedback.

His purpose is to look for patterns in what gave users difficulty.

By focusing on problems common to more than one user, you ensure that you get the most bang for your buck in making changes and fixing the site. This is also why running a test with one user is insufficient.

Conversely, to save time and money, you shouldn’t test with more than 5 users per round of testing. By the fourth or fifth user you’ll already know the main issues the site has and having the same problems highlighted another half dozen times isn’t productive.

The problems the moderator will discover can usually be categorized by WiderFunnel’s classic LIFT model of conversion optimization:

LIFT Model of CRO

  • Value Proposition: What’s in it for the user. This is the core which the other factors act upon.
  • Anxiety: How design -security reassurances, layout- and content -answering questions, social proof- affect trustworthiness. Reduce anxiety to lift the conversion plane.

  • Distraction: The page features too many eye-catching elements. Focus is drawn away from the page’s main purpose. Reduce distraction for more lift.

  • Relevance: The degree to which the page matches users’ expectations before arriving. Increase relevance for more lift.

  • Clarity: Do the design and content work in synergy to explain the value proposition?

  • Urgency: Do users have a reason to act now?

To return to our earlier example of the online running shoe retailer, here are some things you would look for in your analysis.

First, as mentioned earlier, is the landing page’s credibility and relevance to traffic sources. Is the site secure? Is the graphic design professional? For relevance, you can show the traffic source (ad, link or search listing) briefly before taking users to the landing page. Within the five second test, you can ask if the landing page matches the expectations the traffic source created.  – Relevance, Anxiety, Clarity -

Visitors need to drill down and find a product that suits their needs. Can they browse and search their way through your categories to find an adequate fit? How effective are your refinement options? When clicking through to a subcategory or product detail page, does the content there match expectations? – Relevance, Clarity -

Third, visitors need to get sufficient information from the product detail page. Does the information answer visitors’ questions? Does it convey benefits? How scannable is the information – bullet points or paragraphs? Where’s the refunds policy? Are the shoes in stock? How much is shipping? – Value Proposition, Urgency, Anxiety-

Users also need to be able to add to cart easily, and then modify the contents of the cart. Is the add to cart button noticeable and clearly a button? Are the cart buttons clearly labelled and laid out in a hierarchy reflecting their importance? Does the site need all this information the checkout requests? – Clarity, Distraction, Urgency, Anxiety -

Step 5: Implement The Changes

Again, follow Steve Krug’s excellent advice: Change the least possible to solve the problems.

For example, instead of redesigning a page’s graphics to reduce distraction, comment out some of the graphics. Test again and see if that solves things. To save time on such visual makeovers, you can just use a visual testing tool like Feng-Gui or Five Second Test, mentioned earlier.

And don’t wait for the “upcoming redesign”… 99% of the time it’ll go live months after promised. In the meantime, you’ve wasted loads of traffic.

Miscellaneous Testing Tips and FAQ

How do I test…

… information architecture (aka the organization of content on a site into logical groups)? Use card sorting. Do users look for content where you expected? Do they click the right links to drill down to the content you want them to find? (Image credit Revium.)

… where people would click next? Look at click based tools like CrazyEgg or Usabilla.

… how people make their way through a model of the site? Try wireframe creation software that (i) lets you interact with the wireframe by clicking the navigation and going to the wireframes for the relevant pages and (ii) allows you to share your wireframes online, so testers can access them. Some wireframe tools are designed to only be used in a desktop environment, so read carefully to avoid that.

See

Remote Testing vs In-Person Testing Pros and Cons

Remote testing is

- Cheaper: Don’t need to pay for travel, premises, computer equipment, food
- Less work to organize
- Easier to recruit for because there’s no geographic restriction
- Commonly paid for via Paypal or gift certificates

In-Person testing is

- More personal, you can get body language
- A more compelling opportunity for the whole team to witness the testing simultaneously and debrief, which can get things moving faster
- Commonly paid for in cash
- Instantaneous feedback; you view the data as it comes in, instead of waiting until later

“Moderated Testing” vs “Unmoderated Testing” Pros and Cons

Moderated testing:

- Is significantly less likely to encounter errors requiring retakes, such as setup issues with hardware or software, or testers veering off topic
- Yields richer, more useful data
- Tends to have representative users

Unmoderated testing:

- Sometimes yields obnoxious or useless responses, like FiveSecondTest and Feedback Army testers more interested in accumulating credits or micropayments than being helpful.
- Costs less on an individual test basis, because there is no moderator, or because it typically involves testing narrow interaction aspects such as click tests or visual feedback
- Is commonly associated with narrow aspects of interaction
- Tests of particular aspects may or may not have representative users
- Can save time when run through a service with a panel of testers, such that recruitment effort is negligible. This advantage makes it an easy entry-point/stepping stone for beginners to enter the world of usability testing.

Additional testing tools references:

There are many tools available for usability testing specific aspects of a website, or the whole shebang.

Kyle Soucy of Usable Interface put together a helpful, comprehensive overview of these varied tools.

Concluding Tip: Start as early in the web design process as possible

- Talk about the idea with friends and family and see what are the most common questions/objections raised, so you can answer them.

- Create user interface mockups in Photoshop and do the 5 second test (with or without the site): namely, ask people to look for a few seconds, then look away and ask what they recall.

- Put early design mockups through Feng-GUI as a sanity check – are there too many visually ‘loud’ areas?

- Imperfect testing is better than none. As the French say, “the best is the enemy of the good [enough].”

Further reading:

Review of testing service UserTesting.com
Sensible – Steve Krug’s site
Usability Post
RemoteUsability.com/tools

http://okcancel.com/archives/article/2006/07/guide-to-remote-usability-testing.html

http://www.boxesandarrows.com/view/remote_online_usability_testing_why_how_and_when_to_use_it

Sidebar Story

Comments

  1. Great stuff Gab. Loved the idea of using POF and Facebook ads to find testers. Never thought of that. Also, the formatting in this article kicks ass. There is a ton of content there but it's easy to take in.

    Comment by Andrew Breen - November 29, 2011 @ 1:08pm
  2. Glad you liked the ad idea and formatting. I was worried about it initially...

    Comment by Gabriel Goldenberg - December 6, 2011 @ 6:32am
  3. All great points and in my opinion Usability Testing is critical. I use UserTesting.com to see what real website users really think while they use my websites. I am always amazed at what I learn.

    Comment by Jim - December 6, 2011 @ 7:31pm
  4. Thanks for this article. We begun working on CRO on our website, and your advice has just come at the right time.

    Comment by DirectAsia - December 8, 2011 @ 5:17am
  5. All great points and in my opinion Usability Testing is critical. I use UserTesting.com to see what real website users really think while they use my websites. I am always amazed at what I learn Totally agree with you !

    Comment by Eimantas E Sakalauskas - December 11, 2011 @ 6:39pm
  6. Excellent points & summary Found this article via a link from SEO Moz & it is really useful * As a business owner it is very easy to concentrate on generating traffic to your site, with very little time spent on dealing with the conversions side in this much depth * One of the main problems we face is that our customers are usually elderly (due to nature of product) and they don't respond well to certain interactive elements in a webpage. We have found that tables & PDF supporting documents that can be printed off work best {edit from gab:}: You guys can see how it works at Creative Fireplaces - thanks for your great comment Raj!

    Comment by Raj - December 15, 2011 @ 8:46pm
  7. hey Raj, Thanks a lot for sharing that useful conversion tip - I'd never heard of that before. Do you guys do most of your business over the phone too then? Cheers Gab

    Comment by Gabriel Goldenberg - December 19, 2011 @ 12:54pm
  8. Hey Gab We have a unique product in some senses, however I think anyone selling home and garden related furniture and fixtures would probably be in the same position as us: Phone calls are certainly key to converting inquiries into orders. However to make the online side of marketing work profitably you still need to have a high enough ratio of customer being enticed into ordering online without any direct sales staff intervention Despite what many people think, the best method for an older age group is to have an abundance of readily available printed brochures and supporting documents that can be posted to customers. There is a sense or reassurance about receiving professionally printed physical catalogs that can be touched. I suppose we have never had any concerns about spending £2.50 ($4) per copy on printed brochure, as we have done this for over 22 years. However, it is allot of money to spend as minimum print runs are quite high & printed brochures have a low life span as new products are developed * Another problem with driving conversions for all home furnishings is that customers want a ‘discounted price’ (less than RRP or High street) and yet also want to be able to look and touch. This is hard to achieve and portray in many ways. We specialize in marble fireplaces which are very difficult to photograph as well, because the surface is reflective and colored (our photographers often compare it to trying to take a picture of a mirror with a specific color). Hence, having supporting literature and samples is very useful ... a ‘good mix of online and traditional channels of marketing’

    Comment by Raj - December 20, 2011 @ 10:12pm
  9. Great info! I like your point about starting the usability early within the design stage. Doing that will keep things more efficient and effective for clients in the long run.

    Comment by Jack Michael Napier - December 22, 2011 @ 11:54am
  10. Of course - easier to build than retrofit!

    Comment by Gabriel Goldenberg - December 25, 2011 @ 12:34pm
  11. Wow, fascinating insight into your business there Raj! Teaches a lot about discovering your audience's goals and catering to them! One thing sounds a bit self-contradictory though. You write that " Phone calls are certainly key to converting inquiries into orders. However to make the online side of marketing work profitably you still need to have a high enough ratio of customer being enticed into ordering online without any direct sales staff intervention" That sounds like phone orders aren't profitable then; in which case, why do them at all? Can you clarify the role phone calls play for you guys or what you meant?

    Comment by Gabriel Goldenberg - December 25, 2011 @ 12:37pm

Leave a Reply