Recently, a friend of mine got recruited to participate in some market research. He was pretty excited and was happy to sit down with me to chat about the research. We didn’t discuss anything about what the product was so as not to violate the NDA, but I wanted to know what his experience was like as a participant. I was absolutely shocked by what he told me. He described poor procedures, sloppy work, and what I would consider to be unethical practices.
His story made me wonder how much their client knew about what was going on. Did the client even know what questions to ask when they were sold this research? They likely didn’t. They likely told the firm what they wanted and assumed the firm would do it right. If you were buying a house, you wouldn’t do so without asking questions or hiring an inspector, would you? You have the Realtor on your side pointing out things you should ask about. I’m going to talk a little about the things he shared with me and the questions you should ask when considering engaging with a research firm.
Recruitment
My friend was recruited by phone after completing an online survey that popped up while he was browsing a particular product. He spoke with the recruiter briefly before they got into the screener. My friend, it turned out, worked in a similar industry. I’m not sure that they were direct competitors, but one could certainly make the argument that there could be a conflict of interest. This brings me to the first question you should ask a research firm.
“Who would you recommend be excluded as a research participant?”
Most reputable firms will have a list of professions they’ll regularly exclude. This list should include professionals like UX and marketing professionals, members of the media, government officials, and nonprofits. They should also suggest excluding competitors. This is to protect both you and the potential participants. If you include a competitor’s employee and make a change or adjustment based on their feedback, it's possible that participant is commenting based on what they know from their product, which could potentially place you in a precarious position (not to mention that you’re hoping the participant complies with the NDA and doesn’t talk about your product when they get back to work).
As the recruiter started to go through the screener, all was going well. My friend was answering questions, and they got to a particular question when the recruiter said, “Alright, this is the question that eliminates a lot of folks, so just so you know, if you answer ‘no’ to this, I can’t schedule you, and you won’t get the incentive.” He was coaching my friend on how to answer! Now, the honest answer to that question was “yes,” so that’s how he answered, but I was shocked when he told me this. This brings me to the next question you should ask about recruitment:
“What is the plan if we struggle to get people who meet the recruitment criteria?”
Any company should have a good answer here. Even if they don’t expect to run into recruiting problems, they should have a few ideas in the bag. The solution may be as simple as having several recruiting resources. Another possibility is an answer like, “If we run into trouble, we may have to come back and talk about the possibility of opening up a few of the criteria that would have minimal impact on the data.”
Whatever the solution is, you, as the client, deserve a voice in it. Don’t let anyone tell you, “Oh, don’t worry about it; we’ll fill the spots.”
NDAs and other paperwork
My friend arrived at the designated location at his appointment time and was ushered to a desk where he presented his ID. The person there greeted him and checked his name against the list. They then presented him with a form to sign. He was offered no explanation as to what he was given, and the font was so small that he couldn’t really read it. The way he described this reminded me of the Terms of Service that no one reads when you sign in somewhere on the computer. Before he could even begin to glance it over, the person began explaining what he would be doing and then started talking over him to the next person in line, asking for their ID. The message was clear: “Don’t stand here and read this; just sign it, and I’m not going to explain it to you.”
He ran into a similar encounter as he was leaving. He walked up to the table to turn over his name badge and was presented with a stack of papers. This was a stack of packets all turned to the last page, which was just a signature page. The person handed him a pen and said, “Please sign this NDA.” NDAs are important, and you need to understand how they will be covered with participants.
“How do you handle NDAs and ensure that participants understand what they can and can’t talk about?”
Ask this question. It’s incredibly important. Participants should see and sign the NDA before the session starts. They should be given ample time to read it unrushed and have a chance to ask questions. Additionally, someone should verbally review with them the most important pieces. Explicitly explain what a participant needs to understand and get verbal confirmation that they understand. Ideally, this is videotaped at the start of the session.
The actual research
My friend sat down and was informed that he would be completing a survey. He was given very few instructions beyond being told to follow the steps on the iPad in front of him. He was also told that they all would be using the same visual references for the survey on the screens in front of them. They were told that they would go through sections one at a time, together as a group, and couldn’t move to the next section until the entire group was finished, “so don’t be a slow poke.” I was completely shocked! I confirmed with my friend that this was a direct quote. The researchers actually said this. A statement like this is only going to fluster people, make them feel like they need to rush and encourage them not to give thoughtful answers.
This was just one of several things my friend described that made me suspect the moderator was not an actual researcher but a temporary worker hired just for this event to give out the tablets and read the instructions. This leads me to my next question:
“Who will be conducting the actual research and interacting with the participants? Can I meet them? What is their background?”
You’re paying for this research. While the project is still being scoped, a firm might not exactly know which researcher they’ll assign to your project, but they should be able to tell about the experience and background of their senior researchers who will likely be involved. You should have an opportunity to meet these people prior to the research beginning and raise any concerns you might have. If temp workers are utilized for in-person events (for things like reception, check-in, and checkout), ask what they’ll be responsible for.
My friend then told me that during the survey itself, he kept laughing out loud. The sentences were often not worded correctly, and several words were misspelled over and over again. He told me it felt like he was checking over his 4th grader’s English homework!
“What is your quality control process like? Who reviews materials before participants see them? Will I have an opportunity to review surveys and discussion guides?”
At Key Lime Interactive, every research artifact goes through at least three sets of eyes. A lead researcher, a support researcher, and a layer of executive oversight will review it. That’s all before we present it to the client for their review and sign-off. Typos happen, but with that many people looking at things, it's a rare day that something gets through.
Additionally, this is ultimately your research. You, as the client, have a right, and I would argue an obligation, to review surveys and discussion guides before the research begins. Ask why questions are worded in certain ways. Ask about how these questions help answer your research objectives. You’re a stakeholder in the research; you have a right to understand how it works.
At Key Lime Interactive, we hold our research to the highest standard. Participants will not feel rushed, will be given an opportunity to express their thoughts fully, and will understand their obligations under the NDA. Most importantly, while we’ll do all the heavy lifting, you, as the client, will understand what we are doing, why we are doing it, and how it’s going to help you get the information you need to reduce risk and make the decisions you need to make with actionable recommendations, contact us for information on how we can help you meet your business goals.