Data Analysis for Qualitative Data

Questions to Consider

  • • How can qualitative data be collected and used in the evaluation of ASCA National Model programs?
  • • What are the methods for collecting qualitative evaluation data?
  • • What are the methods for analyzing qualitative data in evaluations?
  • • What are the strategies for ensuring the trustworthiness of qualitative evaluations?
  • • What are the major considerations for using qualitative data in culturally responsive evaluations?

Vignette

Ellen, a counselor working in a middle school, implemented and evaluated a classroombased anti-bullying curriculum. In addition to using the closed-ended student surveys provided by the curriculum developer, Ellen decided to interview the 4th-and 5th-grade teachers to ask, “Are there any unanticipated negative consequences associated with the implementation of the curriculum?” She developed a short, structured interview guide, interviewed each of the teachers, and analyzed the recorded data using thematic content analysis. A major theme that emerged was that the teachers felt reluctant to address bullying issues themselves when she was not present because they were unsure of what to do and did not want to be inconsistent with what she was teaching the students. In response, Ellen circulated readings on the anti-bullying curriculum to the teachers, organized a short orientation to the curriculum for them, and asked them to stay in the classroom and observe her while she was teaching the anti-bullying lessons. Subsequent evaluations indicated that these changes enhanced student learning, attitude change, and behavior change.

Qualitative Evaluation Data in the ASCA National Model

Qualitative evaluation is often insufficient to document the outcomes of a program or its associated activities and interventions. However, qualitative evaluation approaches are necessary to document the human impact of the program on participants and to identify the ways in which the program can be improved (Patton, 2015). Qualitative data reflect the subjective experiences, opinions, and judgments of the program’s participants, implementers, and stakeholders and provides important insights into the operation, limitations, and ways to improve the program.

The ASCA National Model (ASCA, 2012) does not emphasize the use of qualitative data in evaluating school counseling programs and their activities. This may be because the ASCA National Model is focused on using evaluation to produce persuasive evidence of the impact of school counseling programs using quantitative data that are of particular interest to school administrators (e.g., student attendance rates, graduation rates, and disciplinary referral rates). While this is understandable, failing to use qualitative approaches to their best advantage will limit the ability of an evaluation to generate information needed to improve programs.

Qualitative evaluation requires that data be collected and analyzed in a methodical fashion. Qualitative data can be collected in a wide variety of ways (e.g., documents, observations, interviews). Qualitative data can be analyzed in a number of ways such as, by thematic content analysis (Braun & Clarke, 2006), interpretative phenomenological analysis (Smith, Flowers, & Larkin, 2009), and grounded theory analysis (Glaser & Strauss, 1967). Below we will describe the most useful and straightforward ways to collect and analyze qualitative data in the evaluation of a school counseling program. Additional useful information on qualitative evaluation methods can be found in both Patton (2015) and Rallis and Rossman (2017).

How to Collect Qualitative Data

School counselors have a particular advantage regarding qualitative data collection. School counselors know how to suspend judgment, ask open-ended questions, probe for understanding, and avoid asking leading questions that constrain and bias clients’ responses. School counselors are experts at individual interviews. Within group interview settings, school counselors are skilled at creating a safe environment, drawing people out, and helping them add to and build on each other’s understandings. All these skills are necessary in constructing qualitative data collection instruments and strategies and in collecting qualitative data.

Open-Ended Survey Questions

Using open-ended items in a survey is the easiest way to collect important qualitative data. Open-ended items require a written response from survey participants while closed-ended items only require the participant to select from among a set of pre-determined responses. Open-ended items can be added to a closed-ended item survey in order to create the opportunity for participants to add insights that cannot be anticipated beforehand by the school counselor. For example, at the end of a closed-ended stakeholder satisfaction survey, an item might ask, “In your opinion how could the school counselors work more effectively with parents?” Alternatively, an entire survey may be comprised of only open-ended questions. A major advantage of open-ended survey questions is that they efficiently generate a written record of a large number of responses that represent the data upon which subsequent qualitative analyses are based.

All open-ended survey items should be aligned with an evaluation question. To avoid confusion, each survey item should address only one issue. Open-ended survey items should be written in simple straightforward language, avoiding jargon, to ensure that everyone will understand the question in the same way and has sufficient experience and knowledge to formulate an informed response to the question. In addition, all items should be screened to make sure that they do not inadvertently bias participants’ responses. It is advisable to invite a small group of stakeholders to pre-review these items, to observe them while they answer the questions, and to interview them in order to identify needed modifications and revisions. Table 8.1 provides a sample of open-ended survey questions, problems with each question, and a possible revision.

 
Source
< Prev   CONTENTS   Source   Next >