Tomer Sharon

several types of questions you should avoid in an experience sampling study because they will never help you uncover needs:
Questions about opinions: Asking for someone’s opinion about something several times a day or week is useless. Opinions don’t change five times a day or week, and there’s no point in asking for them that much. Examples might include:
• What do you think about hiring a Web developer for updating your website?
• Should links be blue or black?
Questions that speak to “average” behavior: A common trap people who phrase questions for experience sampling fall into is asking them in a way that vaguely refers to a time frame about which the question is asked. For example, when you ask, “What frustrates you most about boarding a train?” you are setting yourself up for failure. Let’s assume the person who tries to answer this question has boarded trains 300 times during her lifetime, 50 of them in the past year, one last week. This person will probably not remember all of the frustrations about boarding a train and will try to come up with an answer that “averages” the ones she does remember (maybe the last five). She will also try to satisfy you with the answer and give you a real interesting one with a nice anecdote

anecdote. The answer will probably not represent many real frustrations very well. The best way to avoid this trap is to ask about the last time the behavior happened. This way, her memory is still fresh, and it is less likely the participant will pick and choose an answer she thinks will satisfy you. Here are some examples for questions that ask people to “average” their behavior:
• What frustrates you the most when you board a plane?
• How do you spend time while you wait in a long line?
Questions that are too general: While taking a broad approach to learning from people is usually a good thing to do, asking a general question in experience sampling is going to force you to deal with a lot of noise. If you are interested in one aspect of a topic, ask about that aspect rather than asking about the entire topic. For example, if you decided that the scope of your research was uncovering needs related to finding a parking spot, don’t ask, “What annoys you about driving in a city?” Instead, ask, “What was the most frustrating thing that happened to you the last time you were looking for a parking space?” Here are some more examples of questions that are too general:
• What frustrated you recently?
• How did you decide which smartphone to purchase?

anecdote. The answer will probably not represent many real frustrations very well. The best way to avoid this trap is to ask about the last time the behavior happened. This way, her memory is still fresh, and it is less likely the participant will pick and choose an answer she thinks will satisfy you. Here are some examples for questions that ask people to “average” their behavior:
• What frustrates you the most when you board a plane?
• How do you spend time while you wait in a long line?
Questions that are too general: While taking a broad approach to learning from people is usually a good thing to do, asking a general question in experience sampling is going to force you to deal with a lot of noise. If you are interested in one aspect of a topic, ask about that aspect rather than asking about the entire topic. For example, if you decided that the scope of your research was uncovering needs related to finding a parking spot, don’t ask, “What annoys you about driving in a city?” Instead, ask, “What was the most frustrating thing that happened to you the last time you were looking for a parking space?” Here are some more examples of questions that are too general:
• What frustrated you recently?
• How did you decide which smartphone to purchase?
• What is email good for

• What is email good for?
Yes/no questions: Experience sampling is a research method that integrates qualitative, rich data with quantitative, numerical data. Asking a yes/no question eliminates the qualitative aspect of the study to a point where you’ll have nothing actionable to do when you see the results. If you ask, “Did you update your website this morning?” and 78% of the answers are yes, then what are you going to do next? What did you learn about user needs? Which pain points did you uncover? Here are some more bad examples:
• Did you buy milk today?
• Was your bus late this morning?
• Do you like your boss?
Quantitative questions: Similar to yes/no questions, quantitative ones are also not going to be very helpful. A number, an average, or a percentage tells you nothing about unmet needs, missing features, painful problems, or joyful delights. It’s just a number. Here are some examples:
• How many emails did you receive in the past hour?
• What time did you wake up this morning?
• How many items did you purchase the last time you went grocery shopping

infamous question “What do people need?”
To sum it up, after you phrase your question, make sure that it:
• Asks about repeated behavior.
• Does not ask about opinions.
• Does not ask to “average” a behavior.
• Is not too general but very specific.
• Is not a yes/no question.
• Is not a quantitative question in which the answer is a number.

Better Question
1
How did you create your shopping list the last time you went to the grocery store?
Yes
Asks about repeated behavior and is open-ended enough for getting valuable responses.
-
2
What is the reason you use your mobile phone while grocery shopping?
No
Phrased as if it is asked once, not repeatedly. Also, it does not meet a most likely situation where a phone is used multiple times for different purposes during grocery shopping.
The last time you went grocery shopping, what was the reason you used your phone to help you shop?
3
How often did you check the time when you recently went grocery shopping?
No
Asks for a number. Numbers will not