We were recently reminded of the old research axiom that Bad Research Methodology Rarely Produces Good Results… only “tainted results”. So why do some companies or their research providers serve up misleading or in-exact survey methodology? We’ll take a look at some of the key issues in this article.
Our firm is quite frequently approached by a client this has already developed a research plan or even a questionnaire and wants us to execute it for them. Many of these prospective methodologies are well designed or at least provide a good springboard for starting, but others offer the potential for financial or strategic disaster for the client firm.
What Are The Research Goals?
The primary issue in any research, but especially survey research is what is it you want to accomplish, what are the key issues at hand, and how will having answers to them help your firm address or resolve strategic business questions at hand? This is the starting point for any research. Without some sense about the direction, “any road will take you there…”
Obtain Management’s Perspective
One area that is critical is obtaining management’s perspective on the research to be conducted. What are their expectations, what do they hope it will accomplish, are their desires realistic? We find the most important aspect is management’s expectations… if they don’t understand the limitations and potentials of the research, this can create problems when the report arrives on their desk.
Enter Into A Partnership With Your Research Provider
View your research provider as your partner. Don’t hold back on any information that will optimize the development, direction, methodology or results. Unless the research provider fully understands where your company is coming from and their perspective, a lot of key issues can be lost or misunderstood in the process. So be up-front and don’t hold back anything that will be important to creating the best possible research.
Establish And Share The Budget
Another extremely important issue is budget. First of all, be realistic with your supplier… is there enough money in the budget to do the job right? Where is the budget coming from, is it from the research department, marketing or some combination of budgets. The latter case can prove vexing at times, especially when the various parties have different agendas or goals in mind.
It’s important to share with your research provider what budget is available or at the least what range-of-dollars are available to conduct the research. Sometimes clients are asking for bids among two or three providers and they think by not sharing the budget with the research provider, they’ll somehow get a better price or better deal. While this thinking is understandable, it has at least two disadvantages…
The first disadvantage is when asked to quote on a project, if the supplier doesn’t have a sense of the budget available, they can’t develop the optimal plan for that budget. The last thing providers want to do is create a research plan that the client falls in love with, but it’s just not in the cards, budget-wise.
A second disadvantage is the potential fallout for the client. This is a waste of your valuable time, and sometimes it makes you look bad to management by coming back with an unrealistic methodology.
As one recent client recently shared with us, “the last time we selected a research provider on a bid basis, we got exactly what we paid for… poor quality results. The research was done badly and they couldn’t stay on schedule. We’ll never do that again!” As a quality provider of survey research, Just The Facts, Inc. finds that optimal research results are achieved by just being open about budget expenditures. This way the research provider can develop the best research plan possible to meet the client’s needs, goals and available budget.
Poor Research Design
Example #1, Misleading Scaling Usage
Recently, a client-designed survey was brought to our attention, which was conducted via email to their customers. The intent was to measure satisfaction with their customer telephone service department. As we reviewed the research there were several problems that are illustrative of poor research design and as a result, misleading results.
The approach utilized by the client was to send emails to customers on a quarterly basis and solicit answers to a set of developed questions. They then tracked the responses on a quarter by quarter basis. The results were tabulated and then shared in client newsletters and corporate communications citing how much their service had improved during the past year.
There were several problems with this research. First was the selection process, none of the respondents were selected on a random basis, all responses were self-selecting, which meant the bias could be strongly positive or negative, but certainly not representative… and yet it was presented in customer communications as being representative.
Second, was the issue of scaling. The client chose a 4-point scale where the measurement descriptors were as follows: 4 = excellent, 3 = very good, 2 = good and 1 = poor. As can readily be seen, the deck was strongly stacked in favor of a positive response. With only one negative option, where does the respondent go if they feel the client’s telephone customer service is either average or fair, but not poor or good?
Now, while this may be a clever way of forcing positive response, it certainly isn’t good or accurate research… it’s highly misleading. The results proved favorable for the firm as they were able to demonstrate an increase in customer satisfaction over four quarters from a 3.5 out of 4.0 to a 3.8 out of 4.0, nearly an excellent rating. We should state at this point that these results are in deed false, we know this because of our prior knowledge that the firm’s service level had been very poor. They could not have possibly improved service levels so quickly.
A better and more accurate method of scaling would have been either a 5-point scale (5 = excellent, 4 = good, 3 = average, 2 = fair, 1 = poor) or a forced 4-point ( 4 = excellent, 3 = good, 2 = fair and 1 = poor). Either of these scales would have provided a more honest reflection of how customers felt about the company’s service.
Another issue that came up was use of an entirely wrong scale for the question involved. For example, the company wanted to know if customers had obtained the needed answers to their questions, so they asked, “Did you get the answer you were looking for?” They then used the same scaling measure as was used on other questions (4 = excellent, 3 = very good, 2 = good, 1 = poor). The problem is this was not a “poor, good, very good, excellent” type of question! The scaling should been something like, “yes, I did get the answer being sought; no, I didn’t, or I received a partial response”, or something similar to these descriptors.
Example #2, Faulty Sampling Technique
In a second instance, a client came to us asking that we replicate a survey procedure, which they had been conducting at various customer events around the United States. Basically, the procedure the client was using involved handing out follow-up cards at each event, retrieving them back before the respondents left the event, and then completing 20 satisfaction phone interviews per event.
In total, the client conducted 40-50 events annually in various locations around the country. This meant the total sample ranged from 800-1,000 for all events. Typically, each event would draw several thousand attendees (3-5,000). The respondent cards distributed at the event asked for name, phone, address and if they would participate in a follow-up survey. Anywhere from 50-100 respondent cards would be retrieved for each event.
As the reader may have already recognized, there are numerous problems with the methodology employed. First, there was no attempt to obtain a broad base of respondents at each event. The cards were handed out at the end of the event as attendees were all scrambling to leave. Further, the returned cards were all subject to a significant self-selection bias. Many attendees declined to fill out the cards, or did not turn them in, or gave false information (e.g. wrong phone numbers). Our client felt they were doing well if they got back even 25-30 cards per event. Their personnel handing out the cards were untrained in common research techniques and “timid” about retrieving the cards from those filling them out.
The client was satisfied with having only 20 completed phone interviews from each event. They also analyzed the overall satisfaction and need for change of each event based on the results of just these 20 interviews.
Major problems are inherent with the client’s design and execution of this research. First, because they wanted to obtain a satisfaction reading from each event, a call completion rate of only 20 interviews per event is much too small to make any reasonable conclusions about the respondents’ satisfaction. The client should have had responses from at least 75-100 total respondents per event, more if they wanted to read sub-segments.
Second, there was no attempt to gain a breadth of response from different audiences in attendance (e.g. age, male/female, ethnicity, child/teen/adult, and other factors). Any of these could have had tremendous impact on the event format ratings and satisfaction.
Third, because the respondents were opting-in or out, the self-selection bias is a huge problem. The client has absolutely no idea if the 20 respondents per event are reflective of the total audience or not. This means any attempt to read this small sample is probably highly misleading.
Fourth, even if the client were reading all 40-50 events combined, where respondents totaled 800-1,000, the responses would still be unrepresentative due to the sampling technique. Further, each of the events had different aspects to their program format and presentation, so measuring one event against another or in total, could not be done effectively with the current methodology.
Final Perspectives On Bad Research
As these examples illustrate, anyone can conduct research but in the end, what do the results really mean. Where ineffective techniques or poor methodology has been used, the results can be highly misleading, and the research really not worth doing.
Is doing a little bit of cheap research, better than doing none at all? The answer is, if you are seeking representative opinion, then cheap research can hurt you more than help you. If on the other hand, you are seeking exploratory-qualitative research to gain some understanding of respondents’ beliefs, then you need to recognize the research for what it is.
The major problem we see with some client’s techniques, is confusing what type of research they are doing, (representative versus non-representative), and then using the latter to make important, costly decisions which should be based on representative methodology.
