Part 1: Introduction to usability testing
Part 2: Preparing your usability test
Part 3: Running your usability test
Part 4: Usability test evaluation
In person or Online
Personal testing
Testing in person has many advantages. By meeting the respondent face-to-face, you can shake their hand and establish a friendly atmosphere. Furthermore, testing in person eliminates the possibility of technical problems. If you are using an application prototype or a website for testing, make sure that your prototype is ready, properly set up and that the process being tested does not have any crucial flaws that would make testing impossible.
Testing in person also allows you to notice and evaluate the respondent's immediate reactions, as you will be able to witness their body language as they take the test. You can then more easily figure out whether they like a certain feature, are nervous or are losing patience during the testing process.
Online testing
Today’s world and technological development allow us to run usability tests in an online environment. This type of testing is possible, but requires extra preparation.
Compared to testing in-person, online testing is prone to running into technical issues. There can be problems with audio, with screen sharing, prototype sharing, or the respondent could also face issues operating special software during the test. In order to prevent these issues, make sure that you have set up your online test properly and have ensured it is working correctly before beginning of the test. If you made sure to troubleshoot technical issues in advance of the testing, troubles that may occur during the testing would likely be on the respondent’s side.
It is highly recommended that you leave at least 15 minutes in between tests, just in case you have to deal with any of the aforementioned issues. We encountered a delay of 30 minutes before, but this is an exception rather than the norm.
The disadvantage of online testing compared to testing in person is the delay of response time. If you conduct a follow-up call with the respondent, it’s important to ask questions slower than if you were in person. The connection can sometimes be delayed by a few seconds and you need to take the delay into account, so you can avoid interrupting the respondent or losing any feedback.
How many people should be in the room with the respondent?
It is completely acceptable to have just one member of the project delivery team in the room with the respondent. Other members can observe the testing from a distance. Having more people around the respondent could create an uncomfortable atmosphere from being outnumbered. If you are asking questions about sensitive topics, having too many team members in the room could result in respondents closing off from you and not being willing to engage in the discussion.
In case there are additional people involved in testing, the rule of thumb is that the person who leads the test should also be the one who assigns tasks and asks prepared questions. This person will occasionally, or at the end of the test, invite other participants to ask their questions. Do not in any case allow participants to talk over each other, particularly if they are interested in the subject or have unique ideas. This would complicate the test and lead the participants’ attention away from where it should be focused. Ultimately, you could end up not having enough time to reach all the goals you set.
Taking Notes
The person leading the test usually does not have enough time and space to record all the feedback from the respondent. Therefore, it is beneficial when somebody else is responsible for taking notes. If there are test observers, give them a whiteboard, post-its and markers. You can divide the whiteboard into three sections: „issues – no issues – suggestions“ and let them add post-its to these sections. This way, the observers can take notes for you.
The same process can be applied in the digital sphere. A good tool for note-taking is Miro, where you can set up the environment in advance. You can assign a set of relevant screens from the prototype for each task and then let observers add post-its to these screens. Do not forget to ensure in advance that observers will have access to this tool.
Whether you are at the testing session by yourself or you have assistants, we recommend making a video recording of the testing. A video recording of the respondent will help so that you can later evaluate both facial and physical reactions. An additional video recording of the prototype will also help you to see where the respondent was clicking and which task they were reacting to. You can access these recordings at a later time and fill in more comments and notes as well.
Sometimes, you may need to show your respondents’ reactions to your stakeholders. You can use the video recordings to make a short clip of the biggest difficulties respondents experienced during the test, which you can later discuss with your project delivery team and stakeholders.
Do Not Skip the Introduction
Make sure to warmly welcome your respondents and offer them some refreshments – this is really important! Allow them enough time to calm down and relax. During both personal and online testing, always start with easier questions. For example, where is the respondent from, what is their occupation, how was the traffic, what do they like etc. so they can get used to interacting with you. Before you start with the testing itself, explain the main procedures of the testing clearly and slowly. Tell the respondent how the test will proceed, ask them for permission to record the test and let them sign the consent form. Express your gratitude for their time and give them the promised remuneration (if you are using remuneration as a motivation tool) in exchange for the signed document confirming they have received their reward. It is better to take care of all the forms and financial aspects right after the respondents have arrived so that you will not need to worry about this later on.
Before you start assigning tasks to the respondent, inform them that the goal of the test is to find deficiencies and problems with your prototype, not to test their abilities to respond to the given scenario. Reassure them that if they get stuck during the test, it does not mean they are to blame but that the product designer is.
How Long the Test Should Take
Each respondent should spend no more than 45–60 minutes on the test. After this time frame, respondents start to feel tired and they tend to lose focus.
How to Ensure the Best Possible Feedback
At the very beginning of the test, assign the respondent with a complex task that will check the whole pathway you aim to test. Ideally, the respondent should complete this task without your interference. Through this task, you should be able to tell if the respondent is able to complete the process by themself. You receive the most optimal feedback when you do not interfere with the respondent at all. This way, you’ll be able to assess if there is a part of the process that is confusing or unclear to your respondents.
How Long You Should Wait for the Respondent’s Reaction
Some users take a really long time before they make up their mind about how they really feel about the product. If you interrupt them during the process, you risk losing valuable feedback. For you, it can feel like eternity, but be patient and wait just a few additional seconds. A good researcher should be able to patiently wait in silence to give the respondent enough time to decide what they thought of the test. By going into testing with this in mind, you can gain the patience required for effective results through practice.
Even when the respondent asks about certain features, it is better to remain silent. The common saying applies here: „Silence is golden, speaking ruins testing.“ Most importantly, do not explain the feature you are testing to the respondent and do not justify its purpose just because you personally like this feature. The purpose of the test is not to sell the website or an application to the respondent, but rather to gain valuable information from them. If you can turn silence into your ally, you will gain constructive comments from your respondents.
What if the Respondent Does not Know How to Proceed or is Curious About the Next Steps?
It is essential not to offer direction to your respondents. If the respondent asks you how to proceed, it is the best to simply repeat the question. E. g. when the respondent asks „Is this button for completing my transaction“? You may reply: „Where do you think this button will take you?“ You can also transform the question into a situation in which the respondent would be by themselves: „What would you do in this situation if you were at home alone and you saw this button?“ The respondent’s answer can reveal frustration or shortcomings of the test. You may get a response saying that the respondent would have left the website or would have closed the application. This is an important result that you could have potentially lost if you had explained the function of the button.
