Evaluating the Strengths and Weaknesses of Various Data Collection Methodologies

A commonly encountered question when planning a survey concerns the mode of data collection: should respondents be interviewed in person, by telephone, by mail, or through an online questionnaire? In the end, each mode presents its own unique set of advantages and disadvantages. Some of Responsive Management’s major observations regarding the advantages and disadvantages of the basic four modes of survey data collection are summarized below:

Survey Mode	Advantage	Disadvantage
Telephone	One of the fastest data collection methods Along with mail, typically achieves the most representative sample (telephone ownership is near universal) Power of persuasion through interaction with live telephone interviewer Easy for person to respond with minimal effort Higher response rates because of multiple callbacks Accuracy through Computer-Assisted Telephone Interviewing (CATI) software Ensures all survey questions are answered Ensures appropriate respondent is contacted and answering the survey Can provide anonymity Eco-friendly	Costs are moderate to high CATI software required Technical expertise required Answering machines / screening can impact response rates Limitations on survey length For gen. pop. surveys, best sampling requires purchase of the sample For gen. pop. surveys, sample must include both landline and cellular Area codes for cell phones reflect area of purchase and not necessarily residence For program participants, sample needs to be representative of all participants Use of graphic or visual aids not possible Interviewers must be hired or trained
Mail	Provides excellent coverage (address-based sampling) and allows for the most representative sample Does not require large staff Respondent convenience (can choose time and place to answer survey) Can use graphics or visual aids Flexibility in types of questions that can be asked (ranking of multiple items, tabulations, etc.) Ability to include incentives to assist in increasing response rates	Responses are usually biased toward males in general population studies Beware of illiteracy issues: 43% of U.S. adults read only at the most basic level or are illiterate Lower coverage rates for named respondents Response rates depend on group Multiple mailings required (costly) Obtaining a sufficient response rate can take 6 weeks or longer Cannot verify correct respondent Cannot guarantee that respondent will fill out survey properly Not possible to probe or clarify Many questions often left blank Data entry costs Possibility of data entry error Management can be labor intensive No power of persuasion—many times only those with vested interest choose to respond Not environmentally friendly
On-Site Intercept	Best for personal interaction, including probing and clarifying of questions Allows in-depth exploration of issues Allows for observation of nonverbal responses/cues Greater tolerance for survey length Almost any type of question can be asked Can use graphics or visual aids Power of persuasion through interaction with live interviewer: high response rates Ensures all survey questions are answered Housing units or location intercepts can be used to obtain sample (no list needs to be purchased)	Very costly Very time-consuming Geographical limitations Possible interviewer bias Possible interpretation bias Possible acquiescence bias Respondents lose feeling of anonymity May have no second chance to convert refusal into completed survey May not be representative sample May be more difficult to hire and train interviewers (travel may be required)
Web / Online	Anonymity One of the fastest data collection methods Costs are low Computerized data collection Can use graphics or visual aids Flexibility in types of questions that can be asked (ranking of multiple items, tabulations, etc.) For closed populations with known email or internet access (i.e., complete coverage through email addresses or internet access), online surveys are a good option	Everybody thinks they’re an expert For open-ended general population surveys, data are not representative (non-probability sampling) For open-ended online surveys, results may be manipulated by people sharing the link to likeminded individuals Cannot verify correct respondent Populations less likely to have online access (e.g., rural residents) are excluded Online panels usually consist of “professional survey takers” Difficult to calculate response rate Difficult to assess nonresponse bias Cannot guarantee respondent understands questions No power of persuasion—many times only those with vested interest choose to respond (self-selection)

Responsive Management has also published peer-reviewed papers on survey methodologies, including “The Fallacy of Online Surveys: No Data Are Better Than Bad Data”.

For all projects, Responsive Management works with clients to customize the study methodology according to the goals and objectives of the research.