Recruiting Your Target Audience for UX Testing with Screener Questions

| January 29, 2015
Sign up to get weekly resources, and receive your FREE bonus eBook.
Thank you!

Get ready for some great content coming to your inbox from the team at UserTesting!

screener-header

Users are at the heart of what we do at UserTesting and in the UX industry.

They tell us what needs they have so we can fill them.

They inform our designs.

They identify strengths and weaknesses within our products.

In a perfect world, usability testing involves users who wholeheartedly represent your site or app’s target market; they have the easiest time adopting the right mindset, they speak the right lingo, and they have the best insight into an ideal experience.

But whether or not you need to recruit exactly the right target user depends on the nature of your test.

When should you test with your exact target market?

Many UX thought leaders encourage researchers not to be too granular about the users included in their studies. After all, a vast majority of products should be clear  and intuitive enough that ANYONE can figure them out: a grandma should be able to buy her grandsons some sports gear online without getting confused, and a 6-year-old kid should be able to check a weather app to make sure his soccer game won’t get rained out.

Of course, there are many circumstances in which researchers need to capture insights from a particular type of user—like if you’re working on a Human Resources tool, you need to test with HR professionals because they’re the only ones who would know whether the tool could be helpful to them as they work.

If you’re in one of those circumstances, and you’re testing remotely, then you need to use screener questions—multiple-choice questions that can either eliminate users from taking part in your study or give them access to it.

Here’s an example of a screener question designed to capture plus-sized users:

"Your dress size is between: 0-6; 7-13; 14-21; 22-28; None of the above; Prefer not to say"

Only users who select the third or fourth option will be allowed to proceed to the test.

Screeners may seem easy, but it’s often quite challenging to craft screeners properly. If a user misunderstands the questions, you could end up with test participants who don’t fit the needs of your study.

UserTesting’s Research Team knows firsthand how important (and challenging) it is to write solid screener questions, and we have a wealth of experience in capturing the correct users. Below are some guidelines and examples that can help you get just the right user for your next remote, unmoderated user test.

Guidelines

Many of the guidelines for writing good screener questions are the same as the guidelines for writing great Multiple Choice questions:

  • Always provide a “None of the above”, “I don’t know” or “Other” option just in case you’ve forgotten to include an answer that applies to the user or the user is confused by the question. This is especially important to include in screeners, because if users don’t have this option and pick an answer at random, they might end up in your study accidentally.

  • Provide clear and distinct answers that don’t overlap each other so that (for example) a size 6 doesn’t have to decide if they want to be grouped with the 0-6 sizes, or the 6-12 sizes.

  • Avoid asking leading questions or yes/no questions because users will be inclined to give you the answer they think you want instead of the one that really applies to them. We find that instead of asking direct questions, instructing users to select the option that most closely applies to them, followed by a list of statements, is the most neutral way to phrase most screeners. This method ensures that users will answer honestly because it’s less obvious what answer is desired.

How to check that you screener is capturing the right users:

If you need someone with a particular background (like a medical degree) or someone who is going through a particular experience (like shopping for a new car), we recommend that in addition to screeners, you use the first task of your test to verify this:

“You indicated in the screener questions that you are currently shopping for a new car. Please describe what kind of car you are looking for, where you have looked so far, etc.”

Sometimes, just listening to a user describe their experience can let you know if they’re really the right fit.

A few months ago, I was running a study that required business professionals who are involved with the purchase of IT software for companies with more than 5,000 employees. One of the test participants, when asked to describe their role, turned out to be a cashier at a GameStop who once helped decide what kind of telephone headset to use at his store. While his response to the screener question was honest, he certainly wasn’t someone who knew how to research IT software solutions for a huge company! Having the user explain his background prevented my findings from getting skewed—and taught me to be more careful with how I phrase my screeners.

Examples

Screeners based on familiarity with a product

One of the most common kinds of screener questions that researchers use is capturing users’ level of familiarity with a product or a brand. Sometimes they need fresh users to test out a new tutorial for their app. Other times they are looking for insight from their most frequent users.

Whatever the case, you don’t want to ask point-blank if users fit the mold; people are naturally inclined to say yes, just to please you! Instead, ask users to indicate their familiarity, and then define the different levels of familiarity, like in this example:

"Please indicate your level of experience with Evernote: Absolutely no experience; Novice (30 notes or fewer, use it sporadically, forget it's there); Intermediate (60 notes or fewer, use it somewhat regularly, keep it in mind when needed); Expert (More than 60 notes, use it frequently, use it on all devices); None of the above"

This screener makes it easy for users to tell which category they belong in.

As you can see in the example, it’s helpful to define the Novice, Intermediate, and Expert level by a few elements: overall content levels, usage levels, and consideration levels. This keeps a user from overestimating their experience level with your product and accidentally skewing your results.

Screeners based on frequency of use

Similar rules apply for the related—and equally popular—frequency-of-use screener. A researcher may be interested in finding users who regularly check the weather on their phones, or users who used to play a game, but have given up on it over time.

"Please indicate how frequently you play FarmVille: Often (At least 5 days per week); Somewhat often (1-3 days per week); Occasionally (A few times a month); Rarely (Once a month or less); Sporadically (Used to play it often, but not anymore); Never; None of the above

As with experience levels, it’s important to define frequency in solid terms, not just “rarely”, “sometimes”, “often”, etc.

Another common screener related to frequency of use might have to do with how recently a user has participated in a certain activity. For example, many e-commerce product researchers prefer to hear from users who purchase items online fairly often, and many travel product researchers want to hear from those who are planning a trip within the next year.

In those cases, it may be wise to create two screeners: one to confirm that they purchase items online/have an upcoming trip, and then a follow-up screener to determine time frames.

What will be your household's next big expenditure? New house; New car; Vacation; New apartment; Electronic equipment; Other; None

This question will confirm that the user is planning on going on a vacation soon.

When are you planning to go on this vacation? 0-3 months; 4-6 months; 7 months-1 year; 1 year or more; Other; None of the above

This question filters for users who are planning their trip within four months to a year.

Screeners based on industry or occupation

Another occasion when multiple screeners might be needed to reveal a single characteristic would be when you need users within a particular occupation.

For example, a massage therapy retailer might want to hear from people in the massage therapy industry.

Obviously, massage therapy is a very specific profession, and it would be hard to come up with an exhaustive list of options inside of one screener question. But you also want to avoid asking a yes/no question, so you might start by listing broader professional categories, including Health (which would encompass massage therapy), and then in a follow-up screener, have users indicate the role they occupy within the Health industry. An example is below:

"Please indicate the industry in which you are currently employed/studying: Technical/Web/Engineering; Education/Consulting/Legal; Retail/Service/Sales; Government/Construction/Finance; Healthcare/Beauty/Massage Therapy; Other

The first question gives broad categories to keep from overwhelming the user.

"In which of the three industries do you primarily work? Healthcare; Beauty; Massage Therapy; None of the above"

The second question narrows it down to specific industries.

Screeners that deal with personal information

The last type of screener that the UserTesting Research Team relies on frequently involves users providing sensitive information, such as their income, their race, their Facebook profile, or their body type.

For example, there are apps that help couples who are attempting to conceive a baby determine peaks in fertility. Obviously, reproductive health is not a topic that everyone is comfortable discussing in a user test, so it’s important to forewarn users of that, and only accept users who are willing to be open about this personal information. The screener would probably look something like this:

"This test will require you to speak openly about your sexual habits and physical processes. Only select "I Agree" if you are willing to discuss this information: I agree; I do not agree; Other/None of the above"

This screener gives users a heads-up that the test will deal with information that not everyone is comfortable discussing.

Get started with screeners

    The above examples only scratch the surface of ways that researchers capture their ideal users! Often these are used in some kind of combination, like screening for men who frequently track their fantasy football data, but have never heard of Rotowire. When you use your imagination (or enlist the help of the UserTesting team), the recruiting possibilities are endless.

    Shaping screeners can be a challenge, so we encourage you to always perform a dry run, in part to ensure that your screeners are effective!

    And if you’re a UserTesting Pro client, you can reach out to your Client Success Manager about creating the ideal screeners for your product.