Help my data doesn't align...!

Have you ever been there...? You did everything right: you convinced your client that online data collection is a valid method for the study, he seems ready to move his -- let's say -- offline tracker to an online method. You've prosed a parallel study to identify changes directly resulting from a shift in data collection method and to you will assess these findings when considering a move to online research. And now the data between the offline and online research does not line up. What to do? Here are four steps that will help you and your client...:

1) Understanding the type of difference

The first step in dealing with the different results is to look for the type of differences:

• Type 1 difference: shift in results but no fundamental change in the overall conclusions and recommendations.
• Type 2 difference: shift in the results and a fundamental change in the overall conclusions and recommendations (e.g. different concept ranking, significant results in brand awareness between brands, etc). So as a first step we must check if the conclusions and recommendations differ between the offline and online results. Remember it's all about you supporting the client in their business decisions, and if the decision will be the same, the difference in data collection method does not matter, right?

2) Explain the difference

You now have to interpret the differences found: in case of Type 1 differences you should comfort the user of the research that several reasons exist to explain the differences, but they are relatively little and did not influence business decisions. In case of Type 2 differences: we should distinguish between those type 2 results that we can easily "live with": they are not key-indicators and are not hugely important for business decisions, and those type 2 results that really differ and influence business decisions. Once we divided the Type 2 differences into two groups, we can move to the next step:

3) Which data is closer to reality?

For those Type 2 differences that are really key for the project, we must now check which results (the offline or offline results) are closer to reality: which results are actually a better representation of the market? This could be a combination of variables: general ones such as age, gender and education or region, or more specific ones to the category: category use in the P3M. Perhaps you have actual market shares to compare? Perhaps sales figures? Is none of this available, well, together with the client you can evaluate which figures are more likely to be representing reality.

4) Calibration of results

We will now have to decide how we are going to calibrate the results between the two methods. Generally speaking, the approach towards existing data is the following:

1. before, or during the transition from offline to online data, parallel testing is done to measure discrepancies caused by the change of method.
2. For a limited amount of time (couple of months, up to one or two years, depending on the old data available) the new online data will be weighted towards the old existing offline data for those variables where the parallel test proved a change in results.
3. Once enough online data exists, once your client feels comfortable and has gotten used to the different new level of the online data, the above exercise will be done again, this time weighting the offline historic data towards the new online data. This should allow for the historic offline data to be still used in the future.

Drop me a line to let me know if the above is helpfull !