I: Applications

Applications

The chapters in Part I describe and discuss the details involved in some specific applications of methods and principles of transformative action research. In Chapters 2-4,1 describe and discuss specific uses in community organizations, which are also quite applicable to schools, colleges, public agencies, and neighborhood grassroots groups. These uses include program evaluations, community needs assessments, developing pilot projects, the innovative notion of community-based think tanks, and ways to improve one’s organization, or simply one’s own daily efforts. In Chapter 5, I discuss how the qualities of transformative action research intertwine with other dynamics of learning and human development, including learner-centered open education, ego development, and expert knowledge. Then, in Chapter 6, I examine the application of transformative action research ideas and methods to critically reflect on the need to inquire into the decision-making challenges and dilemmas in the midst of the traumatic, rapidly unfolding uncertainties of the COVID-19 pandemic.

Program evaluations

PROGRAM EVALUATIONS

Evaluations in community agencies

(Revised and updated from an earlier paper with Dr. Terry Lunsford.)

Most people who work in community organizations, schools, and colleges have heard a lot about evaluation. Many funding sources, especially foundations and federal and local agencies, now routinely demand that a group asking for a continuation of funding provide them with an evaluation, and this is sometimes required even for a first-time request. Programs that are parts of larger organizations also are being evaluated frequently, these days, as funds grow smaller and the competition for them grows hotter, especially in public, social programs, and educational innovations.

For these reasons, the most common version of evaluation is a one-shot assessment of the program’s or organization’s “performance,” sometimes on a pre-set list of brief, statistical perfonnance indicators, which are assumed to define in working terms its goals, expected outcomes and worthwhile efforts. The group or program is typically faced with a too-near deadline, evidence of unfamiliar measures of perfonnance that they have to show they are meeting, and consequently, the need to hire someone who does “evaluation research” for a living. Alternatively, the group may have to give the job of doing the evaluation on short notice to someone already working within the organization. The evaluations produced in this way range from the ingenious to the ridiculous, and still, many do obtain some useful information. However, the insights from these spur-of-the-moment evaluations often have little or nothing to do with what the people in the organization, or the people it is supposed to serve, think is good work, or what they care about in their lives.

With a transformative approach to action research, however, it is possible to start somewhere else in building a good evaluation process for a community agency or innovative program, one that will help you learn about what you are doing, make some judgments about how good that is, and communicate with your staff and constituents about these things, and about what the problems are. Here are some suggestions about ways to start and use such a process, intended as starting points. Most of you have ideas and experiences of your own about evaluations, and these will provide a good starting place for any evaluation.

Let’s assume that you are the person responsible for evaluation in your organization. An advantage to involving others in working with you to do an evaluation that makes sense to all of you on the inside of the organization, more than just conforming to “outside” demands, is that you can get everyone off the defensive. Having to prove that you meet an outside set of criteria usually is like proving that you’ve stopped beating up on your spouse; it’s a loaded question that’s being asked. Funding and other external agencies have their own reasons for wanting to know how many bodies you’ve processed, or how your “cost-benefit ratios” compare to some other groups that they consider “good ones.” These kinds of indicators make the funders look “objective,” even though their criteria may not fit your organization’s activities very well. Such standards help funders and other outsiders to answer the questions of people to whom they feel accountable (like their own Board members, or higher administrators) about why they’re giving you money or “official approval,” instead of some other group. In this way, funders, for example, get what look like definite, fair, impartial criteria for making their judgments on the spending of scarce resources. But such questions may have very little to do with what your community group or innovative program is trying to do, and they may distort the reality so much that its real value and distinctive benefits become unrecognizable or unseen, if your evaluation stands or falls by answering only the questions asked by others. So, one important place to start is by figuring out what questions do make sense for you.

Using statements of goals and planning strategies, and other ways of being very systematic in the ways you talk about your programs and services, and their outcomes, are about as unpopular with most agency and school staff as are evaluations, these days. However, in figuring out the right questions, it’s often better to start, not with a list of questions, but with what your organization or program is actually doing. You might well begin by laying a descriptive base for your evaluation. Just set out to describe the activities you do, and why, and maybe how each of them got started, and something of how they fit together. If you do that, it can get an extremely rich, colorful description, going far beyond pre-set ideas and sterile concepts such as “categories of services.”

You will almost certainly have to pare down your earliest descriptions, to make them readable and not too long. The point is, you can start by describing what your organization does as it is actually experienced by the people who are doing the work and helping to get things done, and by those who benefit from the organization’s activities. Leave articulating the formal goals and analytic categories until later. If you start with them, they could kill most of your ideas about what really goes on and push you back into the traditional kind of “evaluation.” In merely adopting other people’s standards and concepts, you may fail to learn more about what you’re doing well, and why, what your organization’s shortcomings and challenges might be, and even whether you might gain insights that could lead you toward some different directions that are more consistent with your deeply felt values and commitments.

You’re likely to find that not everyone in the group really knows what is going on, and that different people see what is happening differently. So, you can begin to build a kind of descriptive mosaic, consisting of these varied views, with a lot of pieces fitted loosely together, but not in a tight, “logical” scheme. This less coherent, evolving mosaic will probably be more like real life than a contrived effort to begin with an artificially drawn picture that is not so accurate. You’ll have to figure out whom in the organization, and on the edges of it, to ask for their views—including likely, staff members, clients, Board members, potential clients, other informed observers, and maybe members of other similar or connected groups with whom you work regularly. It’s a good idea to get views from a range of different people who have different relationships to the agency, including those especially with strong, informed opinions. You should record much of what these people say, because you won’t know, at the start, which of the things they’ve said will prove most useful, and will make good parts in a full description—once you start to investigate further the picture they’re painting, and then go on to write your sense of everything later.

It’s okay to ask people what they think the organization is doing, and why it does it that way, what they think is good and bad about it, and what else you should be doing, and why they say those things. As you get into it, a natural flow will probably come. You can ask about “goals” if you want to—but ask each person what goals they think the organization is pursuing, and what goals they think it should be pursuing. Their answers, and the other descriptions, will tell you a lot about what the group’s operational goals are; that is, what its actions indicate, as against what people say it is doing. The people with whom you talk, either informally and/or in more formal interviews can tell you what they see to be the organization’s important constituencies—workers, clients, supporters, and others— and what those people think the organization should be about. The standards they use, in talking about the organization, will tell you a lot about how they think about the organization, and its priorities. You will probably learn a lot you didn’t know about what people really care about, as against what they think is the answer you want. The more open and conversational you can be about these discussions, the more you’re likely to get insights and observations that matter. Both group discussions, where people spark each other’s thoughts, and individual, private discussions can be helpful in laying this descriptive base.

As you build up your base of information, you may find you are naturally working backwards from this description of what has been the group’s values (goals, ideals) and what, at the moment, various people in the group think should be the values and guiding ideas. Further, over time, you’ll have a lot more information on which to make such judgments, than you did at the start. You’ll probably find that you have learned a lot that you didn’t know about things that you thought you were on top of You are likely to realize that you now have thoughts, insights, and feelings about the agency’s activities that hadn’t ever occurred to you before. This is a big dividend that often results from this kind of open, descriptive research, even before you get to the explicitly evaluative part.

After a while, you may want to ask people in your organization’s subgroups to write evaluations of it, and to make a practice of doing this every so often. In the process, they can set the standards they use, and use their own words to describe the good and the bad. That’ll make it easier for them to contribute to the evaluation, and you may find they think about the whole idea of doing “an evaluation” in new ways. Furthermore, you will almost certainly get a lot of rich material that isn’t pre-digested by you, or unintentionally “parroted” by others because they think they know what they “should” say.

This can build toward a real evaluation that can make a difference—a difference that is likely to be meaningful to you, and to others in the organization, and to those who may benefit from what the organization does. Whether your organization is a community service agency, a school, or a neighborhood group, you can mobilize broad participation involving many people in a potentially valuable evaluation, not just by dutifully performing a perfunctory task. You have the opportunity to critically reflect on what you actually are, and aren’t, doing, examine some new options, and then decide if you want to initiate some creative improvisations.

Ideally, you, and others, will come to see “evaluation” as something that can, and should, be done continually. For example, it’s perhaps valuable to think in terms of a doing a continual series of evaluations, none of which has to be the “perfect” evaluation. As you do some evaluations, whether large or small, it may be a good idea to pull such evaluations together, periodically, and to share them with the important constituencies of the organization. Share only a digest or highlighted summary of what you learned, if you like, but make it an honest one, and a good basis for a series of discussions of what each evaluation means, and why. For one thing, that will tend to make people think more deeply about the criteria of judgment that you should be using in evaluating your group’s work. It may make people feel a lot better about what they are doing, and this often happens. It may bring to light some hard criticisms, and some tough issues about how and where the organization is going next. In any case, it will usually be an important educational process in self-evaluation for the group and can be a major step forward in improving the ways you operate, and a very substantive basis for group planning. I’m not saying do this mindlessly, setting up a mutual put down session, or stirring up submerged enmities in the group. Still, taking a chance, in the right context with this kind of frank, full information, can be a powerful builder of self-confidence and cooperation, if the group is basically a healthy one.

From such written descriptions and evaluations, from your interviews and group discussions with important constituencies, and from your own efforts in pulling these together, you should be able to formulate your own standards, acceptable to the necessary constituencies, of what the organization is doing right, what its problems are, what priorities make sense for the coming period, and what the organization means to the people who put effort into it, and/or depend on it, or who hope to benefit from it. In these ways, you can write a better, one-shot evaluation than most deadline-driven, reactive “studies” will allow. You can combine this richer description, which fits your own values and goals, with whatever, additional, needed formal goal statements, outcomes, and indicators of effectiveness that your funding sources, or other external agencies require of you.

That is, don’t let the formally stated requirements imposed by others, along with their expected indicators of success, take the initiative away from you. Make it clear that your richer version of the organization is the more real one and that any others, while they may be reasonable from other points of view, are “outsiders’” versions. No need to be defensive here, either—-just keep on the affirmative, and speak your mind with confidence and full facts. In that way, you really put your best foot forward, showing that you care about what your organization is doing, and that you aren’t afraid to look at it. This can be much more impressive to an external source than any hasty, on-demand reaction to its categories and criteria, no matter how expertly you try to manipulate “the data” within the frameworks determined by others.

This means, of course, that evaluation should start well ahead of any deadline that you will have to meet from the outside. Ideally, you should aim to build evaluation into your organization’s operations, as an ongoing thing. Make it okay to talk openly about the values and the shortcomings of the agency—not in a carping, negative way, but constructively, trying to be helpful with one another, to learn, and still willing to do some self-criticism as well. If periodic, written evaluations are accepted in your agency, as an expected thing, and if people know that they will have a chance to get their voices heard, they may be a lot more willing to make commitments to the group, to do work, and to care about the community that you are a part of. And, when an outside demand comes, you’ll be prepared. Not because you have started far ahead of the deadline, but because evaluation is a part of your organization’s life, not a semi-fraudulent response, done at the last minute, to keep someone else off your back. That may feel a lot better, too.