Systematic Sources of Error

Each source of error diminishes validity and reliability, attenuates potential effect, and reduces statistical power. Seven general sources of data error/ bias and systematic variation have been identified. Possible sources of error should be anticipated and plans developed to avoid or minimize them. Measurement error will be produced from any and all of these sources:

• • Chance variation
• • Participant instructions
• • The instrument
• • The data collector
• • The respondent
• • The environment
• • Data management errors.

Common Biases to Valid and Reliable Measurement

All evaluations in which data are collected are subject to some forms of bias. Although controlling all sources of bias is impossible, minimizing error from each major source in an evaluation is critical. The literature has identified 12 common biases in human measurement: ^{[1] [2] [3] [4]}

• 5. Interviewer effects: Interviewer characteristics, for example, age, gender, or dress, may affect the receptivity and answers of the respondent.
• 6. Changes in the instrument: When an instrument is used more than once, a learning effect is possible. Interviewers may become more proficient or tired of conducting an interview.
• 7. Population restrictions: The method of data collection may impose restrictions on the population to which the results can be generalized, for example, telephone interviews require phones.
• 8. Population stability over time: An instrument administered at different times may not be collecting the same data on different populations.
• 9. Population stability over areas: The same way of collecting data in two different geographic areas may assess different types of people.
• 10. Content restriction: Only a limited range of data can be reported by each method. Self-report questionnaires cannot be used to study cognitive mechanisms of short-term to long-term memory. Observational data cannot be used to study relationships among for values.
• 11. Stability of content over time: If a program restricts a study to naturally occurring behavior, the content of the studied phenomenon may vary over time.
• 12. Stability of content over an area: A program may not be uniform in content.

• [1] Subject effect: People who are aware that they are being measuredmay respond in atypical ways.
• [2] Role selection: Awareness of being measured may influence peopleto play a special role.
• [3] Measurement as a change agent: Measurement affects a person’sbehavior.
• [4] Response sets: People respond to questionnaires and interviews inpredictable ways that have little or nothing to do with the questionsposed, for example, answering yes or no to most or all questions orgiving a socially desirable response.