The Validity of Findings
Assuming, however, that the instruments and data are valid, we can ask whether the findings and conclusions derived from the data are valid. Asian Americans get higher average scores on the math part of the SATs (scholastic aptitude tests) than do other ethnic groups in the United States—581 versus 515 for all ethnic groups combined (College Board 2008). Suppose that the SAT math test is a valid instrument for measuring the general math ability of 18 year olds in the United States. Is it valid to conclude that ‘‘Asians are better at math’’ than other people are? No, it isn’t. That conclusion can only be reached by invoking an unfounded, racist assumption about the influence of certain genes—particularly genes responsible for epicanthic eye folds—on the ability of people to do math.
Reliability refers to whether or not you get the same answer by using an instrument to measure something more than once. If you insert a thermometer into boiling water at sea level, it should register 212 Fahrenheit each and every time. “Instruments” can be things like thermometers and scales, or they can be questions that you ask people.
Like all other kinds of instruments, some questions are more reliable for retrieving information than others. If you ask 10 people ‘‘Do the ancestors take revenge on people who don’t worship them?’’ don’t expect to get the same answer from everyone. ‘‘How many brothers and sisters do you have?’’ is a pretty reliable instrument (you almost always get the same response when you ask a person that question a second time as you get the first time), but ‘‘How much is your parents’ house worth?’’ is much less reliable. And ‘‘How old were you when you were toilet trained?’’ is just futile.
Precision is about the number of decimal points in a measurement. When you stand on an old-fashioned scale, the spring is compressed. As the spring compresses, it moves a pointer to a number that signifies how much weight is being put on the scale. Let’s say that you really, truly weigh 156.625 pounds, to the nearest thousandth of a pound.
If you have a predigital bathroom scale like mine, there are five little marks between each pound reading; that is, the scale registers weight in fifths of a pound. In terms of precision, then, your scale is somewhat limited. The best it could possibly do would be to announce that you weigh ‘‘somewhere between 156.6 and 156.8 pounds, and closer to the former figure than to the latter.’’ In this case, you might not be too concerned about the error introduced by lack of precision.
Whether you care or not depends on the needs you have for the data. If you are concerned about losing weight, then you’re probably not going to worry too much about the fact that your scale is only precise to the nearest fifth of a pound. But if you’re measuring the weights of pharmaceuticals, and someone’s life depends on your getting the precise amounts into a compound, that’s another matter.
Finally, accuracy. Assume that you are satisfied with the level of precision of the scale. What if the spring was not calibrated correctly (there was an error at the factory where the scale was built, or last week your overweight house guest bent the spring a little too much) and the scale was off? Now we have the following interesting situation: The data from this instrument are valid (it has already been determined that the scale is measuring weight—exactly what you think it’s measuring); they are reliable (you get the same answer every time you step on it); and they are precise enough for your purposes. But they are not accurate. What next?
You could see if the scale were always inaccurate in the same way. You could stand on it 10 times in a row, without eating or doing exercise in between. That way, you’d be measuring the same thing 10 different times with the same instrument. If the reading was always the same, then the instrument would at least be reliable, even though it wasn’t accurate. Suppose it turned out that your scale was always incorrectly lower by 5 pounds. This is called systematic bias. Then, a simple correction formula would be all you’d need to feel confident that the data from the instrument were pretty close to the truth. The formula would be:
true weight = your scale weight + 5 pounds
The scale might be off in more complicated ways, however. It might be that for every 10 pounds of weight put on the scale, an additional half-pound correction has to be made. Then the recalibration formula would be
true weight = (your scale weight) + (scale weight/10)(.5) or
(your scale weight)(1.05)
That is, take the scale weight, divide by 10, multiply by half a pound, and add the result to the reading on your scale.
If an instrument is not precise enough for what you want to do with the data, then you simply have to build a more precise one. There is no way out. If it is precise enough for your research and reliable, but inaccurate in known ways, then a formula can be applied to correct for the inaccuracy.
The real problem is when instruments are inaccurate in unknown ways. The bad news is that this happens a lot. If you ask people how long it takes them to drive to work, they’ll tell you. If you ask people what they ate for breakfast, they’ll tell you that, too. Answers to both questions maybe dead on target, or they may bear no useful resemblance to the truth. The good news is that informant accuracy is one of the methodological questions that social scientists have been investigating for years and on which real progress continues to be made (Further Reading: informant accuracy).