Today’s Environment…
With some degree of regularity, clients approach Just The Facts Research wanting to conduct mail survey research studies. In other instances, clients have been conducting their own in-house mail research and now seek our assistance in execution. Is mail survey research still viable into today’s heavy direct mail, promotional environment or has it become “Dinosaur Research?” You decide...
In the past seven years, in particular, our firm has seen a huge decline in the requests for mail survey research, in favor of telephone studies. This article will look at various sides of the decision of whether to utilize mail surveys versus telephone studies.
The first question we always ask, when approached about conducting mail survey research is, “What is your organization’s objective and are there other ways it could be more effectively or easily be accomplished... and yet still achieve equal or better results?”
The biggest challenge with mail surveys is the highly competitive direct mail and promotional mail environment that marketers face today. Just think about your own experience at the office, and the daily barrage of materials that come across your desk. How many promotional, advertising or solicitations do you typically receive in an average day, an average week? Lots, and lots! Now how many do you actually open, how many never get viewed at all... they're just tossed into the circular file!
Much of the mail survey materials we see, are not very eye-catching or appealing on first reviewing by the receiver. As a result, the tendency of recipients is to say, “here’s more junk mail, I don’t need to read it or I don’t have time to read it.”
When this happens, as a researcher, you’re losing a significant portion of your potential, broad-based response base to the survey. So the very first issue marketers and researchers are confronted with is, “Are you willing to invest an appropriate amount of your budget into really well-designed research materials.” Or, is your company’s expectation “We can just run these off the copy machine and send out a few thousand, and this will be an adequate response?”
This design issue is an important first step, because the quality of target response and its representativeness across your target base is critical. So then ask yourself, is our organization willing to invest in effective presentation of our research… without an “affirmative response”, we would suggest not even doing the research.
Pitfalls to Mail Survey Research…
The presentation of mail surveys is only the beginning. Assuming you do have the “most well-designed and presented printed materials”, there are many other hurdles still lurking in the background with respect to mail surveys.
Response Bias
We touched on this above, but even with the best designed materials, the level and type of response can be extremely biased, polarized or misleading. This is true because in mail surveys, the researcher has absolutely no control over who in fact, is responding to the survey.
Representative Bias
In addition to “Response Bias”, how representative of the target sample can a mail survey ever be? Because the researcher can never control or select who will answer the survey, its representativeness is highly questionable.
The responses received to a mail survey can never be gauged as to their representative nature. Truly representative research should provide a balanced response from a broad spectrum of the research target. But what happens instead, is that because respondents are “self-selecting”, the researcher never knows if he or she has a good cross-section of the target sample.
As an example, we have seen instances where mail surveys have been “wildly positive, absolutely glowing”, in the respondents’ replies on everything from new products and services to customer satisfaction issues. But the results are highly misleading, because they are not representative. In other cases, the responses are mostly negative… indicating greater dissatisfaction than truly exists.
Accuracy of Response
Another issue with mail surveys is their accuracy. Because respondents are alone when filling out the questionnaire, individual questions can be open to a wide range of interpretation. The wording of a question may strike two persons with distinctly different meanings. And so, without anyone to clarify the meaning, the answers are given based on their own impressions and interpretation.
Timeliness of Response
As noted above, we are all inundated with mailings and direct mail materials. So just capturing a respondent’s few precious moments of time to complete a self-administered questionnaire is a challenge in itself. Many persons receiving the survey will be “turned off” just by looking at it, and perceiving it will require too much time… so it gets tossed.
Turnaround Response
Then there are those respondents who are well intentioned, and are open to taking the survey, but just never get around to it, or keep putting it off… longer, and longer. Typically, in mail surveys, the bulk of response will occur in the first 2-5 weeks after the “drop”… after that, responses can dribble in over many weeks or even months.
The optimal mail surveys, have some component of follow-up telephone execution to propel higher response rates... or some will also do multiple waves of mailings. But neither of these “stop-gap”measures deals with the other inherent problems or biases of mail surveys. Plus, if you’re going to add a follow-up phone component, why not just do it right to begin with?
Phone Surveys… The Best Alternative?
In our experience with mail surveys versus telephone studies over the last several years, we have found the latter (phone) to be significantly preferred. Why is this…
First, the representativeness of the study is addressed all the way through the research. WE SELECT the respondents in phone research as opposed to the self-selection, which potentially brings on such flawed results as are common in mail surveys. Phone research ensures that a broad cross-section is selected and responding to the study, so that the most representative response is obtained.
Second, we control the time factor with a phone study. If the respondent contacted declines to participate in the study, we know this instantly and replace them with an equally appropriate target candidate. There’s no waiting or wondering about if, and when, the “mail respondent" will in fact, participate.
In phone studies, the accuracy of response is very high. On the few occasions where there is confusion over the wording to a question or its meaning, the phone interviewer can immediately respond and clarify any misunderstandings.
Additionally, completion of phone studies can be executed in a much quicker period of time than can mail surveys. Not uncommonly, we receive client requests to execute a mail survey in a 4-5 week period… GOOD LUCK… it simply can’t effectively be done!
What About Researching Visuals With Phone Research?
Even visual ideas, concepts or products can be researched through a combination methodology we utilize at Just The Facts, Inc. We know how to combine research tactics to optimize both the timing of a project, and the need for getting response to visual stimulus materials.
There’s Just No Comparison!!
So, in concluding this discussion, it is our considered opinion that there’s just no comparison between phone research and mail survey… phone survey wins hands down nearly every time.
And from a cost stand point, while phone research can be a bit more expensive, this isn’t necessarily so. This is especially true when one takes into account the time factor and the required investment for well designed printed materials; the latter is critical in order to have EVEN A CHANCE of acceptable levels of mail readership. So in the final analysis, phone studies are actually a much better value and investment of the client's research dollars.
