How to Get Feedback Over Time with Diary Studies

| June 25, 2014
Sign up to get weekly resources, and receive your FREE bonus eBook.
Thank you!

Get ready for some great content coming to your inbox from the team at UserTesting!

DiaryStudy header-01

Diary studies are a form of longitudinal research (research that takes place over a long period of time with the same participants). Typically, users self-report their activities at regular intervals to create a log of their activities, thoughts, and frustrations. It is a very useful approach for capturing organic feedback on activities that are repetitive, long, or unpredictable.

Research questions that involve habits, prolonged processes, or perception change over time (likelihood to use, learnability, etc) are prime candidates for diary studies. As with all longitudinal studies, time is of the essence. If you can get a reasonable snapshot of behavior in an hour or a day, diary studies may not be right for your questions.

While diary studies are more complicated than ordinary usability studies, they can be done using the UserTesting platform with the help of our expert Research Team.

When to use a diary study

Here are some examples of situations that might lend themselves well to diary studies:

  • Tracking how users complete a long process that is likely to take several days, such as shopping for a large/expensive item or looking for a new house or apartment.

  • Discovering what motivates users to initiate certain actions, such as using a search engine or looking up recipes.

  • Determining how an app or website factors into regular habits, such as fitness or fertility trackers.

  • Assessing retention and/or learnability; for example, seeing whether your users stay engaged with a game after a certain period of time.

Diary studies are also great for capturing information that you might not normally think to do on the UserTesting platform. For example, you could recruit users from our panel and ask them to review a physical product like a fitness tracker, or you could ask them to play a Wii game for several days. (Our Research Team would jump at the chance to help you with any kind of study like this!)

Similarly, you can use a diary study to discover habits and motivations that users might not be able to give accurate, nuanced, or complete enough answers about during a normal recording session.

Diary studies vs. iterative testing

Do not confuse diary studies with iterative research.

Many UX practitioners test the same product consistently and often (as you should!), and may even use the same participants if they so choose. This type of testing provides a snapshot of the product.

Diary studies, on the other hand, provide a look at what prolonged exposure to a product or process is like for the user.

This can, of course, include usability issues, but these issues are typically of a broader scope – problems that affect or derail the overall process, rather than small annoyances or design preferences.

A loose guideline is that diary studies are good for “how” questions (How do people use this? How does it factor into their habits?), while regular usability sessions are good for “why” and “what” questions (What is broken on my site? Why don’t users like this feature?).

Complex but compelling

While diary studies are a bit more time-consuming than regular usability testing, their uses are diverse and fascinating. They provide a clear slice-of-life image of your users, their needs, motivations, and reactions.

For example, the Mozilla Foundation used diary studies to track Firefox users’ habits with regard to saving web content and returning to it later.

You can also use this method to stay on the edge of changing technologies. In 2002, researchers at the University of Colorado used a diary study to examine the learning curve associated with mobile phone adoption.

And if you caught our recent webinar with Janelle Estes of Nielsen Norman Group, you saw a very cool use: tracking users’ multi-channel experience when booking a flight. The complex, time-consuming, and often frustrating nature of booking travel made it an excellent candidate for a diary study.

I think this is the right method for me! Now what?

Diary studies have one VERY important thing in common with many other kinds of testing: make sure you take the time and care to set it up correctly!

Please note: Diary studies are fairly complex and are made easier if you have direct contact with your participants. If you’re a UserTesting Enterprise client, you’ll want to get in touch with your Client Success Manager to make this possible.

Recruitment

Of course, you can recruit the participants you need from the UserTesting panel! Remember to tell users that they will be participating in a long-term study, and that they will need to work on it almost every day.

Also remember that participants will expect a higher compensation than normal, so be ready for that! In a recent diary study, I offered my participants $100 upon successful completion of about a week of diaries.

Testing

The instructions you send to participants must be explicit. Tell them how often you expect them to keep a diary, how long the diary should be, and whether you expect images, screenshots, survey responses, etc.

Try not to make the instructions too complicated, and be very clear!

It is often helpful if you provide a list of questions you want participants to answer or keep in mind as they complete their diary. Because of how organic and flexible diary studies are, users may take your question list as a suggestion rather than a strict plan, so if there are questions that you MUST have answered, make those explicit as well.

Here is an example of a good set of diary study instructions:

We would like to see how you use the YouTube app on the Xbox. For the next 5 days, whenever you use the YouTube app, make a diary about your experience. It does not matter how long you spend on the app or what you watch.

For each session, please answer the following questions:

– How long did you use the app?

– What did you search for?

In addition, here are some questions for you to think about as you write your diaries:

– How was your overall experience?

– What prompted you to use the app?

– How did your experience compare to using the YouTube website?

Implementation

There are many ways for testers to share their insights with you as they complete the diaries. The medium you use is largely preferential, but it should be something that can be shared quickly, keeps everything in the same place, and doesn’t require too much of a learning curve.

I’ve used email in the past (users emailed me every few days with updates). This was easy to use, but a little sloppy when it came time to compile notes. Other studies have used Evernote, SMS, Twitter, survey tools like SurveyMonkey or Google Drive, and even voicemails! You could even create a custom site and use a tool like Posterus to let users upload their thoughts.

Whatever you choose, try piloting your tool with a friend or fellow researcher to see if there are any pitfalls. You want participants exploring your product, not stressing out about how to take notes!

Summaries

This last step is optional, but it can be helpful if you don’t have too many participants.

Once all your diaries are complete, you can schedule moderated interviews with the testers in order to discuss their overall experience. This will give them the chance to provide a summary of what they did, and it will allow you to clarify any details or ask any questions that arose during the study.

Let’s get started!

    If you haven’t gotten the chance to try them yet, diary studies are really fun to do, and they result in insights that are genuinely hard to capture otherwise.

    If you’re a UserTesting Enterprise client, get in touch with your Client Success Manager to kick off a new diary study. Our team is happy to help, and we think it’s well worth the time investment!

    You can also download this handy checklist to help you plan your diary study.