How to Conduct a Longitudinal Study with UserTesting

| December 12, 2013
Sign up to get weekly resources, and receive your FREE bonus eBook.
Thank you!

Get ready for some great content coming to your inbox from the team at UserTesting!

Today’s post is from Aidan Bryant, one of our very own Senior UX Researchers. Enjoy!


Longitudinal Study Calendar image

Remote, unmoderated usability testing is like a mini cupcake. It’s small, delicious, bite-size, and provides just what you need – instant gratification! It can be used to quickly serve up some profound insights in small, manageable portions.

As easy and convenient as single-serving research can be, it doesn’t mean that you have to limit yourself to just that. If you’re finding that what you really want is a double-layer chocolate ganache cake, UserTesting can help you conduct what’s known as a longitudinal study – a rich, dense, time-representative collection of user-focused insights (BYO ganache).

Longitudinal studies allow you to capture a great deal of data over a series of time. In science or sociology, longitudinal studies involve checking back with the same subject over a period of weeks, months, or even years. (Fun fact: the world’s longest longitudinal study has been running since 1921 – that’s 93 years and counting!)

Here are some examples of when you might want to do a longitudinal study:

  • You want to see how users’ enjoyment of your iPad game changes as they continue to play it for several days.
  • Your website is for a service or process that people don’t typically complete in one sitting (for example, buying insurance or comparing colleges), and you want to see how people go through their process.
  • You know your app has a steep learning curve, but you want to see if it becomes easier over time.

What are the benefits of longitudinal studies?

Longitudinal studies are GREAT for capturing information that is difficult to assess from a single 15 or 20-minute session. Therefore, they are well suited for any research that involves long-term behaviors and attitudes, such as decision making, learnability, and attitudinal changes over time. They are also great for exploratory user research.

Another benefit of extended-length studies is that you get to follow a single person through their journey with your site or app. You can see everything (or almost everything) they do from beginning to end. Have you ever done a usability test and thought to yourself, “Well she seems to love my site right now, but will she come back to it tomorrow?” This is how you get to find out!

You also get to see if user reactions are a fluke or anomaly. You might have seen studies where your user just seems unhappy about everything, but how do you know they’re only unhappy with your site? Maybe they just failed a test or got their car towed, so they’re going to be more prone to bitter feedback and violent outbursts.

However, if you get to track their activity over a period of days, you’ll learn if this feedback is normal for them, if it changes, or if it stays at the same level of ire or frustration throughout.

Okay, but how do I do it?

Easy! All you have to do is run a series of 15-20 minute tests (let’s say… one a day for a week) with the same set of testers. So if you want a sample of 5 testers, each of those testers will take one test per day for you.

First of all, it’s important to have a very clear test structure in mind. What will your participants be doing in each test? Will it be the same process over and over, or will they be doing something new every time?

Make sure you let your testers know what they’re getting into! When you release your first test, include a requirement telling them that they will be expected to take a specific number of followup tests, and that the test will last a specific amount of time.

Requirements

Be sure to add important participant requirements and add tester usernames to the ‘re-use your favorite tester’ field so they receive subsequent tests.

Once you have your testers successfully recruited, use the “re-use your favorite tester” option to add them to follow-up tests.

(Enterprise clients: Your friends in our Research team can help you post and maintain longitudinal studies.)

Helpful tips

    • As always, let your research question dictate your plan. Do you want to see if your game is still fun after a week of daily play? Or maybe your store has a Twitter account and you want to see if users reference your account while making a purchase. It’s okay to have more than one question, but be very clear about what you expect to get out of the research.
    • Make sure you have your research plan clearly laid out BEFORE you start recruiting. That said, it’s okay to make small changes to the plan as you get more results. People can be unpredictable, so flexibility is key.
    • Be prepared for some participant dropoff. Most of our testers are happy to help you get the results you want, but life happens, and longitudinal studies typically have a dropoff rate of 10%. Our Support and Research teams can help you make sure your participants stay engaged and hungry for more tests.
    • Be creative. Longitudinal tests can be used to answer SO many kinds of questions. There’s really no limit!

    Longitudinal studies, just like a fancy cake, take more time and energy to conduct, but like any beautiful confection, are extremely enriching and rewarding when they’re finished.