How They Measure
In the UK the main TV audience measurement system (BARB) has approximately 5000 sampling boxes in 5000 households, and each represents somewhere between 5000 and 10000 viewers. They call this group of people a “panel.” This panel is meant to be demographically and geographically (and supposedly “perfectly”) representative across the entire populous of the area being sampled.
Obviously, in reality, even with the best intentions a sample size of this scale is prone to an incredible skew and errors that really would bring into question the validation of the data were it in any other context.
I did reach out to BARB in the UK to see if they had any comments about some of my cynicism. Despite several attempts to communicate with them, they did not reply.
The Discernible Inaccuracies
So where do these problems take us? For example, the current promotional video on the home page of the BARB website describes the panel as having insight beyond that of logging and page impression type analytics so commonly found in website statistical analysis, and further indicates that the panel system gives them a completely accurate picture of how long and who is watching the TV program. They claim that this is something not possible with online statistics. The examples they give indicate that they have the ability to tell if a teenage girl was watching a program “with her friends” and that is something that cannot be done by the sort of online measurement systems that exist today. They also make light of the fact that estimation and extrapolation by three orders of magnitude is not in some way highly inaccurate when compared with the one-for-one data collection that is possible in the so-called Big Data scenarios represented by online media, despite the fact that their panel systems rely on human user input very often, and that data set is then collected in exactly the same way - online - as machine generated data that is inherent within streaming systems.