ComScore: 'Our Panel Is Bigger'

ComScore (SCOR) and Nielsen are both market research and audience measurement businesses. They have a "panel" of independent internet users whose activities they monitor (with permission of course) and based on the internet usage of this panel, they publish broader internet measures like unique visitors, visits, page views, time spent etc. for most of the highly trafficked web sites on the internet. They also work with top internet companies to sell their portfolio of digital media measurement services that allows their clients to determine how their site is doing, where they get their traffic from, who they lose their traffic to, how their competitors are doing in terms of online traffic and so on.

Nielsen, in a recent press release titled “Nielsen Launches Largest, Most Representative Online Audience Measurement Panel in U.S.” stated that their panel of Internet users numbered more than 230,000, bigger than any rival’s panel. This sparked a quick reaction from ComScore whose Chief Research Officer Josh Chasin responded that his company's panel was bigger with 300,000 users. "Our panel is larger!", he claimed. Well, not exactly, but you get my point right?

As for the stocks, SCOR has been an interesting stock, but I am not a buyer. In fact, I am skeptical of these two companies using a sample size of just 300,000 or so to project the behaviour of 100 million internet users. To me, their sample size is not statistically significant, but unless someone else can measure a broader audience, ComScore remains the official word on internet usage statistics.

Also, companies like Hitwise and Compete are nipping at the heals of ComScore with their own rival panels and technology, which proves that this business is not a very unique one and has little to no barriers to entry.

Full Disclosure: I do not own SCOR but my position can change anytime without notice.

Comments

[rrtzmd:]

...no disrespect intended but the statement "their sample size is not statistically significant" is just plain silly...a sample's size CAN'T, under any circumstances known to statisticians, EVER be "statistically significant"...it might be "inadequate" for whatever it is you're trying to measure; but NEVER "statistically significant"...

Jul 16 10:01 AM

[User 283977:]

Sample size is not equivalent with the concept of sample quality. Sample size is related to the precision of the mean estimate. The larger the sample size, the higher the precision. Increases in precision decrease rapidly in a non-linear fashion with sample size. So sample sizes in excess of 2,000 are seldom worth the cost with respect to gains In measurement precision. So why are the panels so large? Because specific web site visitation in the general population is often low. On top of that, researchers want to break their samples into marketing related sub-groups based on gender / age / income. The more ways you want to cut your data, the higher the needed sample size.

The fundamental criteria of sampling quality concerns the ability of the results to generalize to the underlying population. All commercial panel designs should be accompanied with periodic generalization checks against results from freshly recruited randomly selected samples. Without this quality metric you don't know if the panel generalizes to the underlying population.

Another fundamental issue with panel designs concerns panel wear-out. The more times a panel member participates in a research survey, the more likely the panel member's attitudes/ opinions, and behaviours are likely to be altered by answering multiple surveys. Hence, the more times you measure something, the higher the probability that respondents will be changed by the measurement process. A social science analogue to the Heisenberg uncertainty principle in Physics. If you are buying panel research you should be asking questions about how the research supplier "refreshes" their panel. In other words what are the criteria for removing a panellist? In addition, you should be able to obtain information about how frequently the panellists in your research survey participated in other research surveys.

Unfortunately, quality is a hard sell in the research industry. It’s a hard sell because a large proportion of the people buying market research services don’t have sufficient knowledge to recognize quality. Most research services are priced as commodities, lowest bidder gets the contract. To people that don't know what they are doing, large panel sizes are equivalent to 'quality'. This is why the research vendors use 'panel size' as a primary sales proposition. If you think this is bad, you should see what passes for quality analysis in market research studies. One of the reasons why management makes so many poor decisions is because the basis for those decisions, their market research, is substantially flawed.

Jul 16 02:50 PM

[ciao29:]

I have four comments:
1. comScore doesn't use a survey to measure the Internet activities of its panelists. It's all done passively and electronically, with no workload required of the panelists. So, all the discussion about panel wearout is irrelevant.
2. Any statistician will tell you that a sample of 300,000 is more than adequate to accurately measure a population of 100 million
3. comScore's estimates of e-commerce sales have closely matched the Government's Dept of Commerce numbers to within a few percent for a period of 8 years. So the reperesentivity of the comScore sample also seems to be beyond question
4. Hitwise and Compete are hardly "nipping at the heals (sic) of comScore". They're thousands of miles behind. For example, Hitwise isn't even capable of producing audience data that show the absolute number of people visiting a web site. It's all percentage of visits. That's useless for media planning purposes.

Jul 17 07:17 AM

[User 283977:]

User 449588
You are correct that PART of the panels tracking is automated and non-reactive. However, a major part of the panel research business model concerns asking panellists to complete custom surveys. Surveys consisting of hundreds of questions. Surveys that measure the attitudes, and opinions of those panellists. This part of the survey process is quite reactive and is subject to panel wear-out.

Panel wear-out has nothing to do with workload. Panel wear-out is about the effect of completing a REACTIVE survey on the respondents attitudes and opinions. The more surveys a respondent completes, the less likely that respondent is representative of the general underlying population that has not completed a survey. In other words, the act of completing multiple research surveys modifies respondents.

With respect to your second point, a statistician will tell you that a RANDOM sample of 300,000 is more than adequate. Sample size does NOT correct for selection bias. In addition, randomly selecting a typical survey of 2,000 panel respondents from a pool of 300,000 NON-RANDOM selected panel members is not equivalent to randomly selecting 2,000 respondents from the general population. If the selection pool is biased, random selections from that pool will reflect the bias.

You mention that comScore's estimates of e-commerce sales track the Government's to within a few percentage points for a period of eight years. You offer this as prima facie proof that the comscore sample is indeed a representative sample. Lets further examine that reasoning… Suppose the comscore panel was biased towards people that were heavy internet users. Heavy internet uses tend to purchase more frequently on the internet. Hence, if comscore's panel was biased towards heavy internet users, it would be expected to somewhat accurately track annual estimates of total e-commerce sales. In fact, the stronger the bias towards heavy internet users, the more accurately the panel would track actual total e-commerce sales. Hence, tracking annual e-commerce sales is a necessary, but not sufficient proof of whether the panel is representative.

Does it matter if a panel is not representative of the underlying population? It matters if the bias has a potential influence on the basic research question. For example, if a panel is biased towards heavy internet users, and a researcher was interested in a total population sales estimate than the researcher would want to weight the data by the true proportion of heavy, medium, and light internet users in the population. Where do those weighting parameters come from? They should come from annually conducted, randomly selected calibration samples. Calibration samples that allow researchers to understand the nature of the bias in their primary research panels and allow them to accurately portray the exact nature of existent biases to their research customers and provide information that allows those research customers to partially correct panel results to better reflect the underlying population. This is the missing link in most commercial panel studies.

Jul 17 12:50 PM

[Gregman2:]

It would be interesting to see more detail as to Comscore's panel behavior relative to other samplings pulled from various sources, including Neilsen. Analytics, it seems, requires looking at numerous other slices of the pie to get a real picture. Thanks for this.

Jul 17 03:07 PM