11/05/2013

But what are you really asking me?: The challenge of asking research participants the right questions

In my previous entry, I invited autistic adults to participate in a survey.  I wanted to understand:
  • What special interests do they have? (I wanted to get an idea of what special interests are particularly common among autistic adults.  It's easier to keep any questions I ask consistent across participants, compare people's responses to each other, and recruit the neurotypical comparison group if I recruit people with a few specific special interests).
  • What language abilities do they have?  What communication methods do they feel comfortable using? (Speaking? Typing or writing? Sign language? Gesture? PECS? Other AAC?  This will determine the format in which I ask participants to respond).   
  • What can I do to make it easier to respond to questions?  (It's hard to specify what I mean by this in advance, precisely because I don't know what the answers will be.  But an example might be not making eye contact, or sitting next to a participant rather than facing them). 
 Designing this survey turned out to be even more helpful than I had predicted, and not for the reasons I'd expected.  

Although I deliberated exhaustively over every word of each question and each answer choice to make sure they were as clear as possible, my volunteers still found the survey confusing.  I was particularly surprised that one person asked what I meant by a "special interest."  It never occurred to me this term would need explanation, as it's widely used in the autism community by autistic people, parents of autistic kids, and professionals.  It turns out to be difficult to bridge that neurological communication gap, even with a desire to listen and an effort to make myself clear.

Interestingly, my autistic participants actually told me they didn't understand what I was asking them.  The same number of participants who expressed confusion also skipped many of the survey questions, suggesting that they chose not to answer questions they didn't understand.  As a researcher, I appreciated that they chose not to give me data that didn't actually reflect what they thought.

Autistic people may not be unique in their difficulty deciphering what researchers want from them.  In my very limited experience, neurotypicals also sometimes get confused, but they respond very differently.  

I've seen a surprisingly large number of confused participants because I've administered (or seen other people administer) surveys that ask arcane questions--about number properties, object identity, or the nature of objects and substances.  

In cases where I've suspected non-autistic participants were confused, they often didn't ask the experimenters questions, although sometimes their written responses indicated uncertainty.  Whether or not they expressed their confusion, they still attempted to respond.

Sometimes, they appeared to be using their own definition of the terms used or the questions asked.  For example, in one class, I designed a survey asking people if various materials were "objects" or "substances."  These terms are used in several ways in real life; thus, each person provided a different definition of "object" or "substance."  Alternatively, participants may have been responding to what they guessed the researcher might have meant.

I think neurotypical participants answer questions they don't understand because they want to help researchers collect the amount of data they need.  They may also be embarrassed to admit their confusion.  Lastly, such participants may not realize that data from confused people can be hard to interpret.  For example: did my participants really see gravel as more like an object and water as more like a substance in everyday life?  Or was that simply their interpretation of the specific questions I asked them?

Thus, both neurotypical and autistic participants sometimes have trouble figuring out what experimenters want from them.  Autistic people might be more likely to say when they're confused, and decline to answer questions they can't address with confidence.  Thus, researchers know the challenges involved with interviewing autistic people (and thus primarily research those with the highest verbal skills, who are easiest to interview), but might underestimate the difficulties of interviewing neurotypicals.

It's not surprising that even neurotypical people will be confused sometimes, because researchers have to do a difficult balancing act: making their questions as clear as possible without "leading" participants to a particular answer they wouldn't otherwise choose.  It's like designing a math problem that gives all the information needed to solve the problem without actually giving away the answer. 


If you've participated in a psychology survey or interview, how have you responded to confusing questions?  If you've designed them, how do your participants typically respond when confused?  And how do you try to make your questions easier to understand, without "leading" participants to the answer you want?


---
By the way: I updated the title page to explain what I mean by "special interest," among other things.  Hopefully, the survey questions will now make more sense.  I may not have covered everything, as there are still character limits. Check out the new and improved version here:https://www.surveymonkey.com/s/F3CFRZL

Thank you for participating!