ARF: Industry is not served with several research quality standards

Already in September 2006 - almost 3 years ago - the Dutch NOPVO initiative was born; an industry-wide study to investigate panel effects across all Dutch online Panels.

Early June, the ARF's Online Research Quality Council presented detailed findings from a similar US based research-on-research project regarding online data quality, called “Foundations of Quality” (FoQ). The ARF FoQ study has results compiled across 17 online panel providers, it represents around 75% of the online panel sample available in the United States.

The Background

Increasingly, buyers of market research ask the question about panel effects. To what extend have the different panel strategies an effect on the data that is collected and who are the respondents that participate in such panels? There are a number of similar and comparable findings in both studies, topics include:

• Effects of multi-panel membership on survey results;
• Effects of respondent motivations and engagement on survey results; and,
• Connections between proposed or commonly used metrics and data quality.

Let’s take a look on how do these findings cast out on some strongly held believes on panels particularly regarding professional respondents:

Professional Respondents

The idea behind the concept of professional respondents originates from the assumption that online research really is coming from a very small number of people who respond to online surveys for the money, the points, the rewards; they figured out how to game the system. The ARF results show that this is not at all the case.

The picture that emerged from the findings wasn’t what many researchers would have expected: not only wore most respondents not members of more than one panel, but the so called professionals – the ones who are doing most of the surveys - were actually the ones giving the most thoughtful and reliable answers. This conclusion from the ARF confirms what was found in the Dutch NOPVO study over two years ago.

But for the ARF, duplication and professional respondents is not the biggest issue here, the ARF findings highlight the fact that researchers should pay attention to other questions too.

Sample Source

The number one thing that buyers and suppliers should be talking about right now is that panels are not interchangeable. According to the ARF, buyers need to start having conversations with suppliers about the sample sources that they use, which is not a conversation they’re having today.

In a recent Research podcast, ARF’s Joel Rubinson explained how

“…operations people within the sample suppliers need to start monitoring and managing how they source sample for a given study. Not just based on sample availability and productivity, but also based on data consistency..."

This – according to Joel Rubinson – should be the number one area that needs to be attended to, to be able to establish comparability across studies.

90 Day Deadline

The ARF has several more sets of results from this study to release in the coming weeks and has given itself 90 days to come up with recommendations on metrics, business practices, definitions and training. The ARF seems to be taking this self-imposed 90-day deadline seriously.

People are out of patience and out of time and the ARF believes it should come up with solutions or chaos may occure when people find their own proprietary solutions. This is such an interraletd ecosystem, having individual solutions where one buyer has an own approach and another one found another solution will simply no serve the research industry.