Reporting Qualitative and Quantitative Methods

As advocated in this book, school counseling program evaluations can employ both qualitative and quantitative methods to collect the kind of data needed to answer the evaluation questions. Qualitative methods used in school counseling program evaluation could include, for example, observation or video of prevention program operation, examination of archival records that detail responsive services, or open-ended and semi-structured interviews that ask stakeholders about the quality of services. Quantitative methods include, for example, the statistical analysis of school profile data, student behavioral issues, and student achievement test scores. There is broad agreement among scholars and evaluators that the collection and integration of qualitative and quantitative data generally provide a more comprehensive and valid picture of the program. Torres, Preskill, and Pointek (2005) provide several recommendations when working to integrate qualitative and quantitative methods into evaluation communication and reporting. Three recommendations that we think are important for reporting evaluation of school counseling programs are presented here. One, obtain the help of individuals knowledgeable and skilled with each type of data collection method. By seeking help, no type of data will be shortchanged, and presentation of both qualitative and quantitative data will more likely be supported and done well. This is important for the credibility of the findings and will increase the likelihood of evaluation use. Two, develop a framework that will guide the presentation of qualitative and quantitative data. The most obvious is to use the evaluation questions as the guide. The appropriate data are then presented that answer a particular evaluation question. Three, determine the type of data that will be most appealing to a particular stakeholder group or audience and determine whether this type of data should be presented first. The idea behind this recommendation is to gain the group’s attention and hold their attention so that they will seriously consider the findings. The other data could then be presented to complement or bolster the first presentation of data.

Evaluation Communication Forums

Schools and school districts have conventional, routine meetings that can be used to communicate about the evaluation findings of the school counseling program. These include school counselor meetings, school staff meetings, school district administrator meetings, school board meetings, and parentteacher organization meetings. Take advantage of this built-in organizational meeting structure to communicate widely and purposefully about the evaluation findings. Each meeting is discussed below with an eye toward identifying the type of evaluation use the communication is attempting to achieve.

Meetings with School Counselors

Meetings with all school counselors in the school district are typically conducted on a routine basis, often monthly. These meetings are used to discuss a variety of issues important to school counselors and could be used to brainstorm strategies for addressing problems of one kind or another. These meetings are real possibilities for discussing the results of an evaluation of the school counseling program, either an individual school program or the school district as a whole.

To get the most out of these meetings, work with school counseling meeting facilitators to get time in the meeting to discuss the evaluation findings. Perhaps more than one meeting will be needed. Make sure that key aspects of the report are discussed (e.g., recommendations), rather than the report as a whole. And make sure that school counselors have the report with sufficient time to read it, with a clear understanding of what the discussion will entail and the desired outcome.

Whether the evaluation was focused on formative or summative evaluation, discussions could and should occur among school counselors so that they have processed the results and understand the implications. With respect to type of evaluation use, and the meeting and stakeholder group involved (i.e., school counselors), we suggest that instrumental, conceptual, and or process use could all be possibilities. Work to frame the evaluation discussion outcomes to foster one or more of these types of use.

School Staff Meetings

School staff meetings typically occur on a monthly basis. The school principal is likely responsible for the meeting agenda. Work with the principal to obtain time in the meeting to discuss the results. Again, make sure that school staff have a copy of the evaluation report or document well ahead of the meeting so that they have sufficient time to read and formulate thoughts they might have about the findings. Productive discussion at the meeting is more likely as a result. Types of evaluation use that are likely with school staff meetings include conceptual and process use primarily, and persuasive and symbolic use secondarily.

District Administrator Meetings

School district administrator meetings is yet another routine, likely monthly meeting that can be used as a forum to communicate school counseling evaluation findings. These meetings include school building and district-wide administrators. The school district superintendent is likely responsible for the agenda. Work accordingly to obtain meeting time to discuss the evaluation findings and make sure all participants have the necessary material ahead of time with a clear idea of the outcome of the discussion. As this meeting maintains all key decision-makers in the school district, conceptual, persuasive, and symbolic use are likely evaluation use outcomes from this meeting. Reduced to its essentials, communication at this meeting should include how the evaluation was conducted, the findings, and how recommendations will be addressed. In short, as school counselors, use this time to showcase transparency and accountability, and a willingness to act on the results. This strategy will help to obtain and keep support from key school district decision-makers.

School Board Meetings

School board meetings also likely occur monthly and have elected community members to the board. Most all school district administrators attend this meeting. As with school district administrator meetings, transparency and accountability are central to communicating about the school counseling evaluation findings. Work with the superintendent’s office or appropriate individual to obtain time to make your evaluation presentation at the school board meeting. Ensure that all participants have appropriate materials and know the outcomes for the evaluation communication and discussion. In addition, given that school district administrators participate in school board meetings, ensure that all know ahead of time that the school counseling program evaluation will be discussed and the desired outcomes from the discussion. Reach out to administrators with an overview of the presentation so that they have an opportunity to consider concerns they might have about the program evaluation before the meeting. Conceptual, persuasive, and symbolic use are primary evaluation uses in reporting to school boards.

BOX 9.3

Ensure that meeting participants have advance copies of materials beforehand so they have sufficient time to read them and are not surprised by what is presented in the meeting.

Parent-Teacher Organization Meetings

Parent-teacher organization (PTO) meetings occur monthly and may occur more regularly. Since PTOs are composed of parents/guardians from individual schools, evaluation findings that address ways to improve the school counseling program and services will likely be of most interest. Meeting preparation recommendations that were articulated earlier also apply to PTO

Table 9.3 Primary evaluation uses for diff erent types of meetings

Type of meeting

Evaluation use

Instrumental use

Conceptual Persuasive

use use

Process use

Symbolic use

School counselors

School staff

District administrators

School board

PTO

V

y/

y/

meetings. Conceptual, persuasive, and symbolic use are primary evaluation uses in this forum.

Of course, there are other meeting forums that could be used to communicate about the school counseling evaluation findings. Schools and school districts often have community meetings for different purposes and could be used to communicate about the evaluation findings. One-on-one or small-group meetings could also be established with school personnel or community members. Conceptually, there is no limit to the meeting possibilities. Table 9.3 provides the kinds of evaluation use that each type of meeting is best suited. The assessment captured in the table is based on the primary evaluation use for the meeting. Depending on context, any meeting or forum could be used to foster or address any type of evaluation use.

Reporting Results, Using Findings, and Culturally Responsive Evaluation

Reaching stakeholders from different cultural groups with your school counseling evaluation so that they read and react to it is central to conducting effective evaluation. For this to occur, stakeholders must view the evaluation as credible, responsive to their needs and concerns, and culturally relevant. Three recommendations are offered. One, as discussed in Chapter 5, a strategy for fostering a culturally responsive evaluation is to obtain from each cultural group the precise evaluation questions they think are important. While precise evaluation questions may not be articulated, as a school counselor, listen to the intent of what is said and work to develop evaluation questions that represent their concerns. Once the evaluation is conducted, communicate to them the evaluation questions they posed and show how the evaluation answered these questions. Two, if cultural groups in your school speak a language other than English, acquire the services of an individual who can present the evaluation in their language. Three, if certain types of data are important for a particular cultural group, include these data upfront in the evaluation. Storytelling for example, is important to some indigenous groups in Australia, Canada, New Zealand, and the US. Thus, including this type of data and doing so early inthe communication about the evaluation will help to report on evaluation in a sensitive and responsive manner.

Summary

Effectively reporting and communicating evaluation findings is central to fostering evaluation use. Understanding the different types of evaluation use that can be fostered, the kind of use different school counseling stakeholder groups would most likely employ, and choosing reporting and communication strategies that supports and motivates the stakeholder desired use is part of the professional art and craft of doing evaluation work. The number of available reporting tools, especially technology-based tools, continues to grow. Any one of these tools could be used effectively, given the context. And it is not the tool itself that embodies effective reporting and communication. It is how the tool is used. Thus, being thoughtful about how and why particular tools are to be used is important to fostering evaluation use. Incorporating both qualitative and quantitative data collection methods will help to provide a more comprehensive picture of the school counseling program, as well as add validity and credibility to the evaluation. Using the evaluation questions developed by various cultural stakeholder groups and connecting findings to these questions will help to ensure a culturally responsive evaluation, an evaluation that will foster evaluation use.

References

American School Counselor Association (2012). The ASCA national model: A framework for school counseling programs (3rd ed.). Alexandria, VA: Author.

Christie, C. A. (2007). Reported influence on decision makers’ actions: An empirical examination. American Journal of Evaluation, 28(1), 8-25.

Dimmitt, C., Carey, J. C., & Hatch, T. (2007). Evidence-based school counseling: Making a difference with data-driven practices. Thousand Oaks, CA: Corwin Press.

Mayne, J. (2014). Issues in enhancing evaluation use. In M. L. Loud, &J. Mayne (Eds.), Enhancing evaluation use: Insights from internal evaluation units (pp. 1—14). Thousand Oaks, CA: Sage.

Torres, R. T., Preskill, H., & Piontek, M. E. (2005). Evaluation strategies for communicating and reporting (2nd ed.). Thousand Oaks, CA: Sage.

Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2010). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Corwin Press.

Young, A. & Kaffenberger, C. (2009). Making data work (2nd ed.). Alexandria, VA: American School Counselor Association.

 
Source
< Prev   CONTENTS   Source   Next >