Many of the questions we get about online research deal with response quality and/or quality of participants online versus face to face methods. The Advertising Research Foundation (ARF) hosted the Online Research Quality Council to discuss issues surrounding respondent quality. The group just completed a wide-ranging study of online participants utilizing 100,000 participants from 17 different panels.
The following are three findings of interest from the study. More about the study can be found on the ARF website at: www.thearf.org.
The study results address a number of critical questions, including:
- Is there a small group of “professional respondents” on everyone’s panel, doing it for the money, and gaming the system rather than providing thoughtful answers? The answer is a resounding “no”. A small proportion of people are on more than one panel, and the panelist pool is not small, in fact comparable or better than mail panels in their heyday. People who are on multiple panels and who take numerous surveys in a month are, on average, better respondents.
- What drives good survey-taking behavior? The underlying driver is length of survey. Shorter surveys produce fewer “bad respondents”. The optimal number of surveys taken is higher than most expected, and those who are motivated by wanting to share their opinions rather than being in it for the cash gifts also tend to give more thoughtful, consistent answers.
- Are people taking the same survey more than once? The potential exists, although it is less than initially reported. The industry must develop operational approaches to ensure that a survey is not taken more than once.