So you have defined your market research objectives, written your questions, and now your stakeholders are breathing down your neck wanting the results. You are in a hurry to get this survey underway, but wait; before you do you need to test your survey with a few respondents in your target audience. This is known as a pilot.
You should pilot your questionnaire using the methodology you intend to use for your survey in order to find any last-minute errors. If you find an error after you have finished the fieldwork it is too late to do anything about it. You may have to discard some or of all the data if it is not accurate, and potentially start again from scratch, and that could be a very costly exercise.
Here are five mistakes you will avoid by piloting your questionnaire.
1. Question wording that consumers do not understand.
Sometimes we have spent so much time on our questionnaire, or we are so close to our industry that the lingo or jargon is second nature to us, so second nature that when we are designing our survey, we automatically use it in our question wording.
By doing three or four pilot interviews you will soon find out if respondents don’t know what you are talking about if they keep asking for clarification (in a telephone interview or face to face), or they just don’t answer the questions in an online questionnaire.
2. Questions that do not follow a logical sequence.
Sometimes questions need to follow a logical sequence. For example, if you were asking a respondent what they like or dislike about your product, then it should come after a question like “have you ever used product X?”
3. Routing or skip logic that does not work
This can happen regardless of whether you are doing an online, face to face or telephone survey. You have been through so many changes of the questionnaire you your skip logic is wrong.
So for example if a respondent has said they like your product, but then get skipped incorrectly to a question that asks them why they did not like your product, then you will have irritated respondents, not to mention unusable data.
4. Overlapping Questions
I was running a customer satisfaction study for a large company once and two of the questions we asked were.
- On a scale of zero – ten “how likely would you be to recommend us to your family and friends, based on the service you received today”
Followed by
Why did you give that rating?
Then my stakeholder came and asked if she could include a question – “And what is the one thing we can improve”
I tried to tell her that because it is an open end response on a telephone survey the increased time it would take for an interviewer to ask that question and type in the answer would increase the cost of the survey significantly. Which is a waste of money given that the answer to the question:
“And what is the one thing we can improve”
Would be the exact opposite to
“Why did you give that rating?”
For example – If a respondent says “I rated you a 4 because I had to spend 45 minutes of my lunch hour queuing in your branch” The answer to what is the one thing we can improve, is likely to be “Increase the number of staff on at lunch time or reduce the wait times”
After many “discussions” I suggested to my stakeholder that we pilot the additional question for a month and see what the responses would be. Guess what the answers to
“And what is the one thing we can improve”
Was the exact opposite to:
“Why did you give that rating?”
Not to mention a few irritated customers who had not only been delayed in the branches they had also been delayed on the phone as well. We discontinued with her additional question after the pilot.
5. Finding out if there are any “Other” responses that you may have missed.
Sometimes when you are writing closed end questions you need to think of every possible response that respondents may come up with to put into your answer list. This is much cheaper than leaving the question open and then having to pay for someone’s time to code and analyse responses. However, sometimes it is not easy to think of every possible response, and therefore you need to add an ‘other’ specify to your answer list. After the pilot you can review the answers in the other specify space to see if there is anything else you can add to your answer list.
Once you have completed the pilot, checked the results and you are sure that all your skips are working you should have caught all the small errors that could cause big headaches later. Better to have to discard 4 or 5 surveys after the pilot than all 400 or 500 surveys after you have completed your survey.
