Sample Bias and Random Assignments to Groups

Propensity Score Matching

If we are doing a quasi-experimental design, that means that we were unable to randomly assign research participants to either the experimental or comparison groups. This could result in a problem with sample bias, as our groups might differ on some key characteristics. We might be able to address this by matching individuals in the experimental group with similar people in the comparison group. This can be done manually, but another approach is to engage in propensity score matching (PSM). PSM allows researchers to match individuals or cases from different groups to each other based on theoretically relevant variables. Jennings, Fridell, Lynch, Jetelina, and Reingle Gonzalez (2017) utilized propensity score matching when they were unable to randomly assign police officers to either wear body-worn cameras or serve as a control group member without a camera. Instead, they had 60 officers who were wearing the cameras and used propensity score matching to find 60 officers in the same jurisdiction who were not wearing cameras who also matched the experimental group officers on sex, race/ethnicity, age, years of law enforcement experience, and number of physical-response-to-resistance incidents in the previous 12 months. The PSM procedure made the two groups much more comparable, and while there is still the possibility of sample bias due to other variables that were not considered, this matching technique substantially enhanced the validity and generalizability of the results.

A Note About Randomization into Groups for Experimental Designs

Random assignment is important as it can help increase our chances of selecting groups that are equal in many important respects. Taking this step, however, is not a guarantee against sample bias if the pool of potential participants is somehow different from the general population. In Chapter 3,1 discussed the Zimbardo Stanford Prison Experiment that had to be discontinued after just six days due to the aggressive behavior of the young men who were randomly selected to act as corrections officers. As I noted in Chapter 3, one reason to be concerned about the validity and generalizability of these results is that there is evidence that Zimbardo encouraged the same aggressive behavior that he later found so noteworthy. Another problem with the study was his sample selection.

To find participants, Zimbardo put an ad in a local newspaper calling for “volunteers for a study of the psychological effects of prison life.” Once he gathered a pool of volunteers, the research team conducted “diagnostic interviews and personality tests to eliminate candidates with psychological problems, medical disabilities, or a history of crime or drug abuse,” giving Zimbardo a sample of “an average group of healthy, intelligent, middle-class males” (Zimbardo, 2020). What Zimbardo either did not consider or did not discuss in his research findings was the strong possibility of sample bias resulting from his recruitment approach. Carnahan and McFarland (2007) wanted to test this idea, so they replicated Zimbardo’s recruitment plan and then compared it to another recruitment effort for a generic psychological test. Carnahan and McFarland sent out two different ads to recruit research participants. The first had the same wording as Zimbardo’s solicitation, just with a higher offer of pay to account for inflation. The second ad also called for paid participants for a “psychological study” but left out any further details. The researchers administered personality tests to the volunteers for each of the advertised studies, and they found that the people who expressed interest in the prison study differed from those who volunteered for the generic psychological study. Specifically, those interested in the prison study had higher scores indicating greater degrees of aggressiveness, authoritarianism, narcissism, social dominance, and Machiavellianism. It appears that people who are attracted to research in which they could role-play being part of the prison system have some personality characteristics that make them not necessarily representative of the typical college-age male population. While Zimbardo did randomly assign students to the prisoner condition and the correctional officer condition, the manner in which he acquired the volunteers prior to that randomization introduced sample bias. I note this because I noticed that my students tend to see the word “randomization” and immediately conclude that there cannot be any problems with the sample, and that is not always the case. We must consider how the participant pool being randomized was selected.

< Prev   CONTENTS   Source   Next >