The Problem with Surveys

Paul Neto tries to keep things simple on his blog. Last Monday, he posted his vision on why today's market research surveys are broken. Visit the original here.

He takes us through a short history of surveys and research methods ever since central location became en vogue in the late 1950s and ends with the introduction of a couple of MR firms who are getting some recognition for doing innovative things today.

He ends his post by saying that ... online research is a sector that has been growing incredibly fast... ...Though, due to lack of innovation there are many concerns in it's direction. Some major firms are even starting to pull away from using online research due to the rise of professional survey takers, poor methodologies, weak panels and questionable sampling techniques.

Nearly 10 years ago, we were all trying to justify online research to clients. Today just about everyone is online because it's the place to be. The next generation of research is not far around the corner. It really only takes a little innovation to lead to big changes...

I do applaud his post and agree with almost everything he says. I recommend you visit the original here. But I wonder: how should we be preparing ourselves as an industry? The answer I believe must be found somewhere in the shift which will undoubtedly happen over the next 12 years: we will move away from measuring "claimed behaviour" (the whats, whens, wheres and how much) toward analysing "real behaviour". The only thing left in the near future that we may still need to ask for is the why. All the other Ws can be measured simply because (research) data will become a commodity now that all new technologies today are based upon the concept of at least having huge databases.

So on one hand we should have more "web 2.0" type of communication with respondents: really engage them by using survey methods which facilitate creativity, collaboration and sharing of experiences and information on one hand. On the other hand we should facilitate those companies owning the databases to make data accessible and transform digits into data into information into knowledge into insights...

Drop me a line, I am interested in hearing your thoughts!

Online Market Research—Are You Getting What You are Paying For?

This is particularly to those readers in Cincinnati...:

I have a weekly google-alert on "online market research" and this morning it contained an announcement of the Cincinnati chapter of the American Marketing Association.

Next Wednesday, they have a meeting to discuss the strengths, weaknesses, and challenges of the online research methodology and how to get good results.

Online research has made it quicker and easier to get customer input. But the group questions itself: are we always getting what we want and need?

• Will today’s consumers who won’t sit through at 30 second commercial stay engaged for a 20-30 minute questionnaire?
• Will the target customer we need to talk to choose to participate in our survey?
• If a research request interrupts what you personally are doing, do you stop and participate?
• Does the incentive just engage consumers or introduce bias? How long will today’s consumer stay engaged in an online study?
• How do we make sure we are talking to the right people about the right questions to get the right results?

They encourage members and non-members from client companies, research companies, and our local universities to come join us to provide additional perspective to this lively discussion. With both Procter & Gamble and GE in that city, I hope they'll join too.

Should some reader of this blog have an opportunity to go there: let me know what it is all about, I would be very interested in hearing how the meeting was and what the outcomes may be. And I am sure, so are many of the readers of this blog...!

Click here if you have an opportunity to join, it's only 10US$ for non-members!

When: Wednesday,May 28th, 20087:45 a.m. - 9:00 a.m.

Where: Web Media Tools18. W. 7th Street, 3rd FloorDowntown

Cost: Members - Free! Non-Members - $10

MarketTools promises to clean up panels

According to an article on Research Live, the US based MarketTools launched TrueSample, a system designed to ensure better online panels by eliminating fraudulent respondents.

The system follows a three-step process:

1. Checks of prospective panellists to ensure they are who they claim to be by verifying the information they provide against what MarketTools calls “extensive databases with objectively validated consumer demographics”. Tests have seen as many as 20% of people turned away from the panel at this stage, the company said.
2. Digital Computer Fingerprint. This fingerprint allows panellists to be cross-checked against other panel members and other panels that have signed up to use the service, stamping out duplicate respondents in studies that utilise multiple panels, and preventing respondents from taking the same survey more than once. Use of TrueSample to date has seen around 3% of respondents fail this test at the outset of each survey, the company said.
3. Data validation technology should spot fraudulent behaviour by correlating survey completion time and response patterns. Activity flagged as fraudulent includes the usual: speeding, straight-lining and ‘satisficing’, but in addition those found to have engaged in such activity will have their computers’ "digital fingerprints" blacklisted.

TrueSample would be available in the US this quarter and was on track for a release in the Europe during the third quarter. It is already in use on MarketTools’ own ZoomPanel and on panels operated by partners Luth Research and Greenfield Online. The TrueSample launch comes within a month of a similar product release from Peanut Labs, called OptimusID.

Why do some panel sources cost more than others?

I've addressed the issue of panels and panel providers before. But I cannot emphasize enough that you should check and understand what type of online sample you're purchasing for your online survey. Online Access Panels from companies like SSI, GMI, ToLuna, CIAO/Greenfield, but especially also from niche market panel providers vary greatly in quality and this affects price. You should screen the panel sources you use and check them to see if they meet your minimum expectations of panel quality.

• Non marketing panel – so you avoid education effects from advertising campaigns influencing the results of research
• Diversely recruited panel – that sourced respondents are derived from a lot of different locations and in different ways to ensure an attitudinal spread in the panel population
• No excessively high rewards – as this tends to result in panellists that are money/discount sensitive, which in turn can affect their research responses
• Actively seek to exclude professional respondents – both through their recruiting (not using professional respondent sites) and through active management (screening and deleting straight lining respondents)
• Well managed to avoid sampling biases – regularly cleaned and scrubbed to remove non active panellists, consistently updated and profiled, panellists are not over-used, sample eliminations are available to avoid education effects from past research participation

As a general rule panels that include a combination of online and offline recruiting are more costly to develop and tend to attract a premium price in the market. This is because they minimise the biases associated with online recruiting alone.

Three Essential Panel Quality Checks

In GMI's latest email Newsletter, three essential panel quality checks are given: a handy guide for any user of online panel as a source of market research. In fact, the three checks are probably more comprehensive than using the more complete, yet bulky ESOMAR 25 questions ESOMAR has defined for online research buyers on panel quality. On the other hand, I think that GMI document is covering all the notional basic checks regarding panel / data quality, but I think I'll dedicate a futre post on that topic... Here are their three checks: CHECK 1: PANEL RECRUITMENT AND INCENTIVES

• Ensure minimum response rates. Your panel provider should require their recruitment partners to provide willing, active panelists who will secure a minimum response rate for your research studies.
• Pay only for good panelists. Your panel provider should not compensate their recruitment partners for fraudulent panelist registrations. They should have systems in place to track which recruiting partners provide fraudulent panelists, and strictly enforce contractual measures to demand a refund of the recruiting fees paid.
• Review panelist redemption files. Your panel provider should regularly create and thoroughly review panelist redemption files to detect any suspicious member account activity.
• Engage panelists with profiling surveys. Your panel provider should regularly revamp each of their profiling surveys, and provide an incentive to their panelists for completing each one. This not only contributes to keeping panel profiling information up-to-date for future research studies, but also helps continuously engage the panel.

CHECK 2: PANEL REGISTRATION

• Enforce panelist login restrictions. Your panel provider should require their panelists to log in using their email address, and create a strong password with a particular type of / minimum set of characters.
• Prevent panelists from changing personal information during registration. Your panel provider should not allow their panelists to change some of the personal information they provide during the registration process. This will prevent fraudulent panelists from creating accounts in multiple geographic regions with varying demographic attributes.
• Store and compare account creation and changes. Your panel provider should be in a position to record a snapshot of every account created upon registration, and store that information in a secure database. Subsequent account changes made by panelists to their personal information should prompt another snapshot, so information can be compared between the two steps.

CHECK 3: OVERALL PANEL HEALTH

• Strictly enforce panelist account removal policy. Your panel provider should decrease the rating of panel members who have incurred an infraction, and remove those fraudulent panelists from the panel after a set number of infractions. Block fraudulent panelists without warning. Your panel provider should delete those panelists who act in an egregious manner from the panel, and block them from opening a subsequent account.
• Remove suspicious email addresses, domains and IPs. Your panel provider should permanently block a panelist suspected to use multiple email addresses, domains or IPs from accessing the panel.
• Detect speedsters. Your panel provider should have checks in place to filter out responses provided by speedsters – panelists who complete a survey faster than the established minimum time threshold.
• Keep bots and scripts at bay. Your panel provider should have technology solutions in place to prevent bots, scripts or other programs from creating or editing panelist accounts. This technology should also be used during account registration, account login and before a panelist submits changes to the personal information stored in their account.

Tagging as a new online research tool

As you would know by now, this blog is all about new methods to do research. It's about how to get rich data from your consumers in a digital world. Only recently I cae across a pilot presented by Ruigrok Netpanel in late 2007. This Dutch research firm is running an interesting pilot where participants can tag products. According to Ruigrok, tagging is not only a way to organise information, it can also be a powerful tool to get feedback from consumers on products. A first demo of a ‘tagging research tool’ was presented early November 2007 at the Dutch Marketing Information Event. This could be a new way of measuring web-site effectiveness or new product concepts. Online panel research today mostly is a static one-way method of asking questions in a survey that most of the times originated from a face-to-face questionnaire. Web 2.0 is all about sharing of information and one way of doing so is by tagging.

Inspired by 2.0 application Fleck, Ruigrok decided to do a pilot with tagging. When the company conducts tagging research, panellists receive an invitation to make leave digital notes. These notes can have a positive or negative emotion. The idea is simple, participants can tag for example websites by placing colored tags and comment on it. This can be done alone or in a group process, moderated by a researcher.

Using heatmaps ('tagclouds') the tool will show what elements of tested concepts trigger the most attention.

A first pilot showed that the data was comparable with their more traditional quantative and qualitative methods. The new research tool showed that this method could get open response, quick feedback, easy to analyze, visual attractive and fun for the participants. But above all, it seems to be an interesting new tool that is fun to do: an experience itself.

Quality Initiatives of Four Research Associations

SSi's blog has a comprehensive post on the major initiatives to increase data quality of online results, the post summarises the objectives of four initiatives and it outlines the differences between the following initiatives:

• The ARF
• CASRO
• ISO 20252
• ESOMAR

Click here to see the post.

Recruiting and Incentivising using Social Networks

Last week, Ray Poynter's blog had a post on Peanut Labs introducing their cooperation with Facebook. This week, the US based AdvertisingAge Magazine published an online article featuring Peanut Labs in which it addresses the growing relationship between online research and using social networking websites as sample sources. Some companies started to use social network sites like FaceBook as a sample source for recruiting panellists or for real-time sampling: recruiting "on the fly". Virtual Currency to Reduce Professional Respondents Their article of December 17th, "Web Research Could Gain Clout Via Virtual Currency" a new start-up company Peanut Labs is introduced. Peanut Labs is an online market research network backed by the investors of Skype, Yahoo and Delicious. The company recruits members of social-networking sites to take surveys. By paying their respondents in the virtual currencies of their respective sites, Peanut Labs is producing response rates of 29% in the US. According to their CEO Murtaza Hussain, they offer a solution for some of today's issues in the market research industry. Offering cash for incentives has proven a real problem for the industry, helping breed professional respondents and incentives for people to cheat so they can qualify for cash payments, next to this it gets expensive to pay hundreds or thousands of people real money to take each survey. Because virtual currency is considerably cheaper than the real stuff, research firms and marketers can afford to shell it out liberally, even to people who don't make it through a survey's screening questions or just to keep people involved with a panel between surveys. Real-Time sampling and Recruiting via Social Networks Some US research companies now started to recruit on Facebook: Focus Groups, but also some of the bigger MR firms offer a registration process to participate in research. The registration is short and easy to complete and right under the Facebook application page. Next to that, some companies also start using real-time sampling (or "river sampling") through Facebook by asking survey qualifying questions to see if the respondent matches any open surveys. The great advantage of Real-Time sampling is that is enables to recruit site visitors from a network of prescreened websites who are willing to participate in a survey that interests them, but who may not be willing to join an online research panel. These potential respondents are randomly assigned to an open survey based on their responses to screening questions. With social-media audiences skewing towards the young and unmarried the relevance of using these sources to recruit and incentivise may be limited for many projects, yet with the growing popularity of these sites, as well as the industry's battle in search of good online research quality it may turn out to be part of the future of online research.