How to record qualitative observations

Known for his paradoxical witticisms, baseball player Yogi Berra once said, “You can observe a lot by just watching.” This is a good rule of thumb for user research: if you’re talking, interviewing or questioning, you’re probably not watching. But there’s a problem with that word “watching” — it suggests a passive process, like the state you’re in when you watch television. In contrast, if you’re a user researcher, watching is an active process: you watch, you record what you see, and then you make use of your observations. Now, this all sounds quite straightforward. But what exactly is a qualitative observation? What’s the best way to record it? And afterwards, when you have a bunch of recorded observations, all potentially useful, how do you go about the complex task of analyzing them?

What is an observation?

Imagine you’ve picked up a researcher’s lab book, and have opened it to a page containing her notes from a field visit to call centre representatives (CSRs). You see a list of ten items you presume are her recorded observations. But there’s a difference between making notes and making observations. To illustrate this point, here’s a quick pop quiz: which of these notes are observations?

1. CSR’s monitor is covered in sticky notes listing system shortcuts, URLs and username/passwords.
2. The ambient noise level is quite high; I have to speak loudly to ask a question.
3. It seems unprofessional to me when a CSR can’t deal with a call and they need to transfer it to a supervisor.
4. The online documentation would be easier if it was printed out and placed in a ring binder.
5. The CSR says to me: “Most of the customers are wound up before they even speak to me. They know their problem probably won’t get solved, so they shoot the messenger.”
6. The sales process takes an inordinately long time. If I was a customer, I’d have hung up long ago.
7. Remove the CAPTCHA from the sales system: at a stroke, you could save each CSR time and frustration.
8. CSR cuts customer contact details from two different screens and pastes to another.
9. Not allowed to leave her desk, other than 30 minutes for lunch and 2 x 10min breaks am/pm.
10. Since every CSR seems to have a smartphone, why not allow people to manage their time with a mobile app?

Think you can tell the difference between notes and observations? Here’s the golden rule: an observation is a record of something you see or hear. These are qualitative observations — high quality data that provides you with real first hand insights into what your users are doing, feeling, and thinking. Novice researchers often treat their opinions on what they like or dislike as observations. But your opinions aren’t observations — and neither are suggestions for new features or insights on how to fix usability issues. Don’t try to interpret the things you observe or fit things into a solution. That comes later. The best observations come from watching real users do real tasks, and being as true as possible to the concrete details of what you see. You can either do this in the wild (with a field visit) or simulate actual use in a lab environment (with a usability test).

How to record an observation

There are two main approaches for recording observations: note-taking and behavioural coding. Note-taking is the simplest and the one I recommend, especially if you’re new to user research. As you watch the user, write down each observation on a sticky note. You should have a separate sticky note for each observation you make. The second approach — behavioral coding — is a way of doing rudimentary analysis as you observe. You assign a predetermined code to each observation. For example, if the observation relates to using support, you might classify this with the code “H” for Help. And if the user complains about the system, you might code this with “N” for Negative Comment (a similar system to the Bullet Journal technique). This approach allows you to filter observations by behavior in your analysis within a spreadsheet. Software packages such as Techsmith’s Morae and Noldus’s Observer XT will also help you code behaviors. However, behavioral coding adds complexity to the task, because as you’re noting observations you’re classifying them at the same time — which may mean you miss something else. So I would start with simple note taking and move onto behavioral coding only if you think it will add more value to your research.

How to analyze observations

If you used sticky notes to make observations, you’ll have a head start when it comes to doing analysis. If you made your observations some other way, transfer them to sticky notes for this step. Stick all the observations on a wall and announce the purpose of the study to make sure everyone is on the same page. (For example, “We want to find the top problems with the checkout flow” or “We’re trying to uncover the user needs for this service”). Then, working as a team, group the sticky notes in ways that make sense. For example, you could group observations that refer to the same part of the system, or by a common task, or by where they occur in the workflow. What you’re doing here is interpreting your observations as a team to gain a shared understanding of the observations. Once the groupings are finalized, give each one a label so you can easily refer to it. The final step is to prioritize the groups. The most important groups are usually obvious but if there’s disagreement you can dot vote to identify the key groups. Now you can get to work designing or fixing the system.

Now you try

Try practicing observation next time you’re amongst a group of people. What behaviors are you observing? What’s curious, odd or unusual? What’s taken for granted? What tools or devices are people using? Are people alone or in groups? What are people touching or looking at? Are their facial expressions relaxed or frustrated? How would you code the behaviors? Answers to pop quiz: 1, 2, 5, 8 and 9 are observations; 3, 4, 6, 7 and 10 are not.

Further Reading:

• Qualitative research methods – Our blog article explains the different types of qualitative research methods you can use, and how to use Reframer to conduct your own research.
• UX Design: Interviewing and analyzing users – Dan Szuc reveals how to take better notes when you’re collecting qualitative observations during your research.
• Observation methods – This article by Sam McLeod from Simply Psychology explains the different types of observation methods and the strengths and weaknesses for each method.