Generating Multiple Objectives and Multiple Performance Measures for a Decision Situation

Making decisions about the design and operation of criminal justice systems requires the consideration of multiple, conflicting performance measures. For example, in a decision by a state as to whether to privatize its prisons, performance measures related to cost, security, effectiveness of programs for rehabilitation, quality of health care, among others, needs to be considered.

In the vernacular of decision analysis, these performance measures are called attributes. An attribute is associated with an objective and can be thought of as a measure of that respective objective. For example, an objective that one might consider in a decision involving privatization of prisons, mentioned in the previous paragraph, would be to maximize the effectiveness of rehabilitation programs. An attribute that one might consider for this objective would be the percentage of prisoners who are rearrested within three years of release from prison. (Note that we want to maximize the objective and minimize the attribute value in this case.) An objective can always be stated as a phrase that starts with one of the following words: maximize, minimize, or optimize.

In many situations, the set of objectives and associated attributes for a decision situation can be presented in the form of a hierarchy, with the more general objectives (e.g., optimize well-being of prisoners) toward the top of the hierarchy and the more specific objectives toward the bottom of the hierarchy (e.g., minimize recidivism).

An Illustrative Example: Determining Objectives and Performance Measures for Setting Up a Syringe Exchange Program

Consider a decision situation involving the institution of a syringe exchange program (SEP) for an urban area. (These programs are also called needle exchange programs (NEPs), or syringe service programs (SSPs).) Through the use of storefronts, mobile vans, pharmacies, outreach programs, and syringe vending machines, these programs allow intravenous drug users to (1) exchange used syringes for clean syringes; (2) obtain condoms, cookers, purified water, and bleach; and (3) obtain counseling services and educational material for their drug addiction problems.

The basic idea is that intravenous drug users, especially those who are homeless or plagued with poverty, are not overly concerned with the obtainment of clean syringes when they inject themselves. As a result, they are at much greater risk for contracting the human immunodeficiency virus (HIV), the hepatitis В virus (HBV), or the hepatitis C virus (HCV). HIV causes AIDS. For example, in 2013 in the United States, 3,096 of the 47,352 diagnoses of HIV infection were attributed to injection drug use. In addition to the obvious problems associated with the health of the individuals involved, there are the economic problems associated with treating the diseases. For example, it has been estimated that the lifetime treatment for HIV is $379,668 in 2010 dollars (Access to clean syringes, n.d.). Much of this cost is paid with public funds. See Syringe Service Programs (SSPs) FAQ (2019) for additional information about these programs.

A review of 15 inquiries involving the analysis of SEPs found that these programs were associated with large decreases in HIV and HCV infections (Abdul-Quader et al, 2013). In particular, it was found that the establishment of New York City’s SEP resulted in an increase in annual syringe exchanges from 250,000 to 3,000,000 during the period from 1990 to 2002; this corresponded to a decrease in the prevalence of HIV in injection drug users from 50% to 17% during this period (Des Jarlais et al, 2005).

Note that there are other means for increasing the use of clean syringes, including the allowance of nonprescription sale of syringes and needles, and the introduction of statutes which eliminate syringes from being classified as drug paraphernalia.

The effects associated with SEPs are not all good of course. (Almost any changes made to a criminal justice system through new laws, programs, procedures, etc., will have both good and bad points, which is one of the essential ideas of this book.) For example, the establishment of SEPs has a high dollar cost for staffing, facilities, and supplies. Second, many people look at the establishment of an SEP as a program that can encourage drug use. Third, many government officials, especially those who are known for their conservatism, consider the political backlash associated with the implementation of SEPs as being bad for them. For example, Mike Pence, then Governor of Indiana, initially resisted the establishment of SEPs in the state over the objections of health officials, before finally relenting (Demko, 2016). These government officials will often base their objections on religious or moral grounds and will cite the possible encouragement of drug use because of the programs (Hedger, 2017).

Finally, many neighborhoods object to the establishment of SEPs in their respective localities. For example, there are usually objections to the relevant facilities being located “too close” to a school (Hinton, 2019) and people who live in the neighborhood often feel unsafe due to the inflow of addicts (Ackerman, 2019).

The decision situation with which we are concerned here involves things like whether an SEP should be implemented in an area, as well as the design and operation of the program. Design and operation imply things like where facilities are located, the capacities of the facilities, routings for mobile facilities, and rules by which the program will operate. As an example of an operating rule, a neighborhood group in Asheville, North Carolina, wanted all participants in an SEP to be required to participate in a rehabilitation program. The program operators resisted this suggestion since their experience indicated that an environment of non-coercion would be necessary to attract participants (Hinton, 2019).

There are several (often complementary) techniques available for generating a hierarchy of objectives and corresponding attributes for a decision situation. For example, Buede (1986) suggests a top-down approach for strategic decisions and a bottom-up approach for tactical or operation decisions.

As the names indicate, a top-down approach would start with one general or strategic objective at the top of the hierarchy, like:

Design the syringe exchange program so that the benefits of all stakeholders are optimized.

Note that we are referring to benefits in this top-level objective in a very general way such that, for example, a cost would be a “negative benefit”. The stakeholders in this case would be: (1) the potential clients of the SEP, (2) neighborhood residents, (3) taxpayers, (4) government officials, and (5) staff of the SEP. Note that one could further divide some of these stakeholder groups into sub-groups to define additional stakeholders; for example, neighborhood residents could be subdivided into schools, residents living in houses/apartments, business owners, etc.

A bottom-up approach would start with the different alternatives for an SEP and expand upward based upon the differences in the outcomes associated with the alternatives. For example, the different alternatives could be (1) no SEP, (2) SEP with Design 1, (3) SEP with Design 2, and so on. As noted above, a design could imply locationfs) of facilities, routes for mobile facilities, specific staffing levels (which imply specific capacities for the facilities), rules under which the program would operate, etc. One of the differences in these alternatives would be the number of clients served on an annual basis. Hence, a lower-level objective for the hierarchy could be “maximize the number of clients served on an annual basis”, with a corresponding attribute of “number of clients served on an annual basis”.

One can also use any device from a list of devices for generating objectives for a decision situation (page 57, Keeney, 1992). Some of these devices suggested by Keeney include: a wish list, problems and shortcomings, different perspectives, consequences, goals, constraints, and guidelines, among other things. As an example, one of the guidelines that might be specified by an addiction specialist would be to not have too much urging of the clients by the staff with respect to entering a rehabilitation program. Hence, an objective would be to minimize coercion of clients with respect to entering a rehabilitation program.

Another approach that can be useful in generating a hierarchy of objectives is the use of specification and means-ends. For example, in expanding downward from a higher-level objective, one could specify what is meant by the higher-level objective, or one could ask what is the means by which that higher-level objective is achieved. For example, one might ask “What is the means by which we can minimize troubles caused to the schools?”, with the answer being “maximize the distance from any school to its closest SEP facility”.

On the other hand, in moving up the hierarchy (from a lower-level objective to a higher-level objective) one might ask “To what end do we want to achieve this lower level objective?”.

Bond, Carlson, and Keeney (2010) suggest a two-stage approach for generating the hierarchy. The first stage involves having a group of diverse stakeholders for the decision situation brainstorm to generate an unstructured group of objectives. This is followed by a stage in which the “analyst” provides structure by organizing the objectives into similar groups. Then the first stage is repeated in some sense by having the stakeholders add additional objectives by thinking more intensely about the individual groups of objectives. The analyst may set a goal such as adding 50% more objectives for the second stage of the process. It may be useful for the stakeholders to employ some of the methods espoused by value-focused brainstorming (Keeney, 2012).

Additional approaches for generating objectives for decision situations include examination of the literature, categorization of stakeholders, fundamental vs. means objectives (Clemen and Reilly, 2013), and questions for moving up/down the hierarchy (Clemen and Reilly, 2013). For additional discussion on these and the other approaches discussed in the preceding paragraphs, see Evans (2017).

Using one or more of the approaches discussed above, we might formulate an initial part of the hierarchy for the decision situation involving the establishment of an SEP as shown in Figure 2.2.

Expanding further on the initial hierarchy of Figure 2.2, under the “Minimize cost” node we could have two objectives:

  • 01: Minimize initial cost and
  • 02: Minimize monthly recurring costs.

Under “Minimize troubles caused to the neighborhood” we could have the objectives of:

03: Minimize troubles caused to schools, 04: Minimize troubles caused to residents, and 05: Minimize troubles caused to businesses.

A partial hierarchy of objectives for a decision situation involving the establishment of a Syringe Exchange Program

Figure 2.2 A partial hierarchy of objectives for a decision situation involving the establishment of a Syringe Exchange Program.

Under “Optimize benefits to clients”, we could have the objectives of:

  • 06: Maximize the number of clients served on an annual basis,
  • 07: Minimize average travel distance for clients,
  • 08: Maximize the number of HIV infections averted on an annual basis,
  • 09: Maximize the number ofHBV infections averted on an annual basis,
  • 010: Maximize the number of HCV infections averted on an annual basis, and

Oil: Maximize the number of addicts rehabilitated on an annual basis.

We could consider the above objectives: O1 through Oil, as lowest-level objectives, or we could consider expanding on at least some of these objectives further. For example, for 03, minimize troubles caused to schools, we could ask the question: “What is the means by which we can minimize troubles caused to schools?”. The answer might be: “maximize the sum of the distances from each school to its closest SEP facility”; alternatively, the answer might be “maximize the average of these distances”.

Finally, for this example, we would use one of the objectives already shown in Figure 2.2 as our last “lowest- level objective”:

012: Minimize political blowback.

One could continue to expand on objective 012. For example, an objective that would be a lower-level objective, related to 012, would be “minimize probability of not being re-elected” (certainly crass in nature, but corresponding to real life). However, for this example, we will stay with the objectives stated above, Ol, 02, 03, ..., 012.

Now, for each of our lowest-level objectives we need to define an attribute to measure that objective. Attributes can either be natural (sometimes called quantitative) or constructed (sometimes called qualitative) in nature. Usually, whether one uses a natural attribute or a constructed attribute will be obvious. For the objectives defined above, 01, 02, 06, 07. 08, 09. 010, and Oil will require obvious natural attributes, as defined below:

  • 01 attribute: initial cost for SEP in dollars,
  • 02 attribute: monthly recurring costs for SEP in dollars,
  • 06 attribute: number of clients served on an annual basis,
  • 07 attribute: average travel distance for clients,
  • 08 attribute: number of HIV infections averted on an annual basis, 09 attribute: number of HBV infections averted on an annual basis, 010 attribute: number of HCV infections averted on an annual basis, and

Oil attribute: number of adults rehabilitated on an annual basis.

Note that each of these attributes can be assigned straightforward numerical values.

The attributes for objectives Ol and 02 would be relatively easy to compute values for any particular SEP design. For the attributes of the other objectives (06 through Oil), predicting values for any particular design (implied by the number and capacities of facilities, routes for mobile facilities, operating procedures, etc.) would be more difficult. At least some of these predictions might be probabilistic in nature, but the predictions could be accomplished through the use of sophisticated mathematical models, or through the use of expert opinion. Examples of the types of mathematical models that can be used to make these predictions are found in Kaplan and O’Keefe (1993).

The attributes for objectives 03, 04, 05, and 012 would be constructed attributes, or qualitative in nature. Associated with each of these attributes would be a qualitative scale, say based on the values of 0, 1, 2, 3,4, or 5, with 0 being the worst possible value, and 5 being the best possible value:

03 attribute: troubles caused to schools on a scale of 0-5, with 0 being the worst possible value and 5 being the best possible value,

  • 04 attribute: troubles caused to residents on a scale of 0-5, with 0 being the worst possible value and 5 being the best possible value, 05 attribute: troubles caused to business on a scale of 0-5, with
  • 0 being the worst possible value and 5 being the best possible value, and
  • 012 attribute: amount of political blowback on a scale of 0-5, with 0 being the worst possible value and 5 being the best possible value.

Note that the scales used do not necessarily have to range from 0 to 5, with the worst possible value in each case being 0, and the best possible value being 5. The ranges could be 0-4,0-10,1-5, or something else. However, something on the order of 0-5 typically works well. In addition, we could let the best possible value be 0 and the worst possible value be 5 if we wanted, if the succeeding activities with respect to the modeling of preferences represented this.

Associated with each of the numbers on the subjective scales should be a description of what is meant by those numbers, on an operational level. For example, a “5” on the 04 attribute scale might correspond to something like:

a very small minority, if any, of the students in the area will even realize that there is an SEP in the area.

The key thing is to make as clear as possible the meaning of the number on the subjective scale for the attribute.

A different approach than the ones described thus far, for determining a good set of objectives and attributes for a decision situation, is described by Keeney and Gregory (2005). Their process suggests that, as one is expanding downward in a hierarchy of objectives, if one natural attribute is an obvious choice for an objective, then it should be used; the expansion from that objective will then stop. If that is not the case, then one might consider several alternative natural attributes and choose the best of all of them. If neither of these actions is reasonable, then one might consider expanding the relevant objective into a set of lower-level (or component) objectives, and then repeat the process for each of these objectives.

Judging the quality of a set of objectives and corresponding attributes for a decision situation is something of a subjective process. This judging of the quality of a set of objectives, or of attributes usually involves determining whether the set has a particular group of characteristics. For example, Keeney (1992, pages 82-86) suggests that a group offundamental objectives for a decision situation has nine characteristics, namely that they be essential, controllable, complete, measurable, operational, decomposable, nonredundant, concise, and understandable.

In a similar fashion, Keeney and Gregory (2005) suggest that a set of attributes for a decision situation should have the following characteristics, as a whole: unambiguous, comprehensive, direct, operational, and understandable. The meanings of these characteristics should be clear, but as an example, consider the characteristic of being operational. This means that we can make reasonable predictions of the values of the attributes for any alternative under consideration. In particular, in the case of the SEP decision situation, the model used to predict the annual number of HIV infections averted must be accurate, and the decision makers involved with the choice of an alternative must be able to make trade-offs between this performance measure and the others; in particular, the decision makers must be able to give an answer to a question like: “How much would you be willing to pay in terms of annual operating cost in order to avert ten annual HIV infections?”.

 
Source
< Prev   CONTENTS   Source   Next >