Assessing and Measuring Culture
The idea of culture having a causal link to company performance makes the idea of measuring and analyzing organizational culture very important. You are not going to successfully change or manage something that you cannot define, observe, and assess; in other words, something you cannot measure. Researchers have responded by developing instruments and techniques for measuring culture and its impact on an organization's effectiveness.
Qualitative vs. Quantitative Measurement of Culture
As anyone who knows me or has heard me speak publicly can tell you, I have a problem with the information security field's use of the term qualitative. In InfoSec, referring to data as qualitative implies that the data should be considered subjective and less reliable, as opposed to quantitative data, which are seen to be more objective and trustworthy. This creates all sorts of challenges for security professionals who are trying to measure the results of their activities.
Our industry's bias toward numbers limits the measures and approaches we can use. It also encourages us to engage in "statistical alchemy,” which is the process by which we take things that are not quantitative and assign numbers to them in an attempt to make them appear more rigorous. What we end up with is not only an attempt to compare apples to oranges, but a formula by which apples are multiplied by oranges, then weighted using a system of bagels. In other words, nonsense dressed up as science.
I regularly see security teams get into trouble statistically, usually when they feel the need to create metrics that will impress senior management. Asking individual members of the security team whether risks and costs are high, medium, or low is a staple of information security risk assessments. The resulting red, yellow, and green heat maps can come across to some audiences as simplistic, because they usually are. But changing high, medium, and low to a range between 1 and 100 (or corresponding arbitrary financial figures) doesn't make a measurement quantitative. It just means that you are asking for an opinion expressed as a number rather than a word. You're still getting people's opinions about the truth rather than actually measuring what's true. But this nevertheless allows many security teams to claim that they have stopped collecting fuzzy “qualitative” data in their assessments in favor of those that are more quantitative.
In the social sciences, including fields like anthropology and sociology, where culture can be of primary interest, qualitative data means something very different. Simply put, data are qualitative when you cannot easily count them. Good examples include a story told during a staff meeting, the transcript of a response to an open-ended interview question, a video recording of a sales meeting, or the photograph from your last team-building event. My example of the security team's opinions regarding risk is another example of qualitative data. Qualitative data are empirical, meaning you can observe them. They just don't immediately lend themselves to statistical analysis, assuming that's what you want to do. But is statistical analysis the only way we can obtain truth or knowledge? When your significant other, or your child, tells you they love you, do you insist on verifying that assertion through a two-tailed t-test or linear regression? Do our favorite movies and novels speak to us because we appreciate that they follow a verifiable Gaussian probability distribution? Clearly, numbers can't tell us everything that is worth knowing.
