Outcome measures: Accountability mechanisms and perceived accountability

This section explains how we tried to measure 'accountability' empirically (for a related attempt, see the 'Accountability Cube' by Brandsma and Schillemans 2012). The first objective of the chapter is to examine to what extent and under what circumstances upwards and downward accountability mechanisms are used to hold regulatory agencies accountable. Several forms of accountability mechanism were taken into account, as shown in Table 3.2. The operationalization of accountability mechanisms is strongly inspired by the kind of mechanisms used for the accountability of public organizations in general (Verschuere et al. 2006), rather than for regulatory agencies as a specific group.

One such set of mechanisms refers to individual upward accountability mechanisms, measured by the number of criteria on which the senior manager of the agency is held accountable by its political principals:

  • (a) with respect to results achieved,
  • (b) with respect to general administrative functioning, and/or
  • (c) with respect to legality of actions and decisions taken.

Each of these variables has three values, whereby 0 stands for no accountability on that specific criterion, 0.5 for being held accountable

Table 3.1 Calibration of concepts and cases

High formal independence

1 = full membership in the set .67 = more in than out of the set .33 = more out of than in the set 0 = fully out of the set

Private law agencies with legal independence Public law agencies with legal independence Departmental agencies without legal independence Departments

Large organization

1 = full membership in the set .67 = more in than out of the set .33 = more out of than in the set 0 = fully out of the set

> 150 FTE 51-150 FTE 21-50 FTE 0-20 FTE

High personnel management autonomy

1 = full membership in the set

.67 = more in than out of the set .33 = more out of than in the set 0 = fully out of the set

Organizations that scored themselves autonomous on all 3 aspects of personnel management autonomy Organizations that scored themselves on two aspects Organizations that scored themselves on one aspect Organization has no autonomy on this subject

High financial management autonomy

1 = full membership in the set

.67 = more in than out of the set .33 = more out of than in the set 0 = fully out of the set

Organizations that scored themselves autonomous on all 3 aspects of financial management autonomy Organizations that scored themselves on two aspects Organizations that scored themselves on one aspect Organization has no autonomy on this subject

High use of organizational upward accountability mechanisms

1 = full membership in the set 0 = fully out of the set

Organization that scored itself on one or two aspects of high use of organizational upward accountability

Score 0, meaning the organization uses no organizational upward accountability mechanisms

High use of individual upward accountability mechanisms

1 = full membership in the set .67 = more in than out of the set .33 = more out of than in the set 0 = fully out of the set

Scores 2.5 or 3 Scores 1.5 and 2 Scores 0.5, 1

Score 0, no individual upward accountability mechanisms

High use of downward

accountability

mechanisms

1 = full membership in the set .67 = more in than out of the set .33 = more out than in the set 0 = fully out of the set

Score 3, three downward accountability mechanisms Score 2, two downward accountability mechanisms Score 1, one downward accountability mechanism No downward accountability mechanisms

High self-perceived accountability

1 = full membership in the set .67 = more in than out of the set .33 = more out than in the set 0 = fully out of the set

  • 9-10
  • 8
  • 7

< 7

Table 3.2 The upward and downward mechanisms for accountability under review

Upwards (in relation to political principals - ministers and parliaments)

Downwards (in relation to regulatees and other stakeholders)

By means of instruments

The evaluation of organizational results by political principals or parent department

The application of sanctions or rewards related to organizational results

The use of quality standards in order to allow users to benchmark the quality of the agencies' activities to these standards

The use of regular surveys of users to give them the opportunity to voice their satisfaction with the activities or services provided by the agency

By means of top

governance

structures

The criteria on which senior management of agency is evaluated by political principals

No criteria or no evaluation

  • • With respect to legality of actions and decisions taken
  • • With respect to general administrative functioning of the organization
  • • With respect to results achieved

Through the presence of a governing board with external members representing

non-governmental actors

on that criterion to some extent, and 1 refers to 'to a large extent'. These three values were aggregated into an index that represents a value between 0 and 3. This index was calibrated to a four-value fuzzy set (see Table 3.1).

The second set of upward accountability mechanisms refers to organizational upward accountability mechanisms, which are used by the political principal to hold the organization (that is, the regulatory agency) accountable:

  • (a) the evaluation of the organizational results by, or on behalf of, minister and department, and
  • (b) whether or not organizational results are linked to rewards and sanctions.

in the case of no such evaluations and sanctions or rewards, the agency received a 0 on 'organizational upward accountability mechanisms'. 'Organizational upward accountability mechanisms' was set to 1 when only one of the characteristics was available and to 2 when both characteristics were present. We calibrated this in a 2-value (or crisp set) condition (see Table 3.1). The chosen calibration is justified when one considers that regulatory agencies are most probably less subjected to evaluation of organizational results and rewards and sanctions compared to public agencies with other tasks, because of the strongly propagated norms of 'independence'.

As an alternative set of accountability mechanisms, we included different forms of downward accountability mechanisms. 'Downward accountability mechanisms' is based on an aggregation of three variables:

  • (a) a variable indicating whether or not the agency has a governing board with external representatives of non-governmental actors,3 and
  • (b) a variable indicating whether or not the agency uses surveys of its users to give them the opportunity to voice their satisfaction with the activities or services provided by the agency, and
  • (c) a variable indicating whether or not the agency uses quality standards in order to allow users to benchmark the quality of the agencies' activities to these standards (dummy).

This results in a value between 0 and 3 indicating the use of downward accountability instruments. This index was calibrated to a four-value fuzzy set (see Table 3.1).

The second objective of the chapter is to analyse whether the self- perceived level of accountability of an agency towards society is affected by the use of downward mechanisms and (individual and organizational) upward accountability mechanisms. Hence, in the analysis for this second objective, the perception of the degree of accountability towards society of the organization was the 'outcome', whereas the accountability mechanisms were the 'conditions' (for related attempts, see Sinclair 1995; Ossege 2012). In the survey, the perception of the senior manager (that is, the CEO) as to the level of self-perceived 'accountability towards society of the agency' (on a scale from 1-10) was determined by asking: 'does the organization prove the need for and the good use of public resources to society?' Using self-evaluation to measure the level of accountability might result in dangers of overestimation. However, we assumed that the perceptions of the senior management do influence the behaviour of the public organization. Therefore, we used a very strict calibration for this outcome.

As we observed predominantly high scores for self-perceived accountability towards society, we also positioned the cut-off point for calibration high in order to avoid overestimation. Scores below 7 (out of 10) were calibrated as out of the set of high self-perceived accountability towards society (see also Table 3.1).

 
Source
< Prev   CONTENTS   Source   Next >