Improving Monitoring and Evaluation
Developing effective monitoring of labor intermediation services and evaluating both the service and the distinct programs supported by the service are an essential feature of Stage 2. In Eastern and Central Europe, for example, public employment services grew up with a social assistance mindset, a right for all beneficiaries, rather than the right to effective interventions. It is often a mindset change to a culture of accountability. The use of a range of monitoring indicators regarding the quality of job obtained, such as job retention, wages and benefits, and earnings gains has been found to make it less likely that either public or private providers will place participants in poor quality jobs.43
Continual refinement of intermediation instruments, programs and services is the only way to keep services relevant to a changing economy and workforce. Research has continued to show that impacts for active labor market policies can be quite modest to zero, but design is the key. And you can’t tell if the design fits a developing country labor market without your own monitoring and evaluation.
Impact evaluations are particularly needed to advance in Stage 2. To date, they have been used more often to assess the impact of training programs. But a meta-analysis review found that developing country positive impacts (short-term) for employment services were relatively consistent with the results for developed countries.44 In one of the few recent impact evaluations of a developing country employment service, the InterAmerican Development Bank found that the job placement and income earnings were positive for men versus a control group, but not significant for women. Only with this kind of information can the Mexican National Employment Service redouble its efforts and increase monitoring to ensure that women’s employment prospects are improved.
As services move to intermediate on a variety of fronts, impact evaluations just measuring job placement results will be insufficient. Testing whether it was one specific intervention that got the job versus a similar person who didn’t use that one intervention is tricky. First, research shows that the multiple methods for job search should be used to find work. Someone might have gotten a job through an intermediation service but it is not recorded anywhere; they might have gotten the information about a job from an intermediation service’s job fair or announcement, but they record in a survey they got their job on their own. Second, one of the important and different roles needed to be played by labor intermediation services in developing countries is to help the external market work better (see next section). That is, a national public-private service is also seeking to stimulate more self-service and combined methods, so it is playing a secondary or indirect role in more job placements. For this reason, impact evaluations combined with other monitoring and evaluation instruments will provide a bigger picture. The following summarizes the key information bases, and monitoring and evaluation instruments particularly relevant to Stage 2.
Monitoring instruments also play a greater management role in Stage 2, to help monitor the effectiveness of discrete offices and discrete interventions. South Korea founded an Employment Services Evaluation Center in 2006 as part of its set of information services. In 2009, they simplified into one composite monitoring index for offices which combines placement rates, customer satisfaction, and duration with employer. A single performance evaluation index weights job placement together with the difficulty of placement measured by the local employment rate.45 Other performance management systems in the advanced countries that use regression models on the role of local labor market conditions include Switzerland, Germany and Australia. Mexico has developed a performance monitoring system of over 60 indicators and annually pits its regional offices in an annual competition for “top state employment service” which has both motivated and rewarded higher levels of service within the system. The Swiss system, for example, measures all its state (“canton”) offices against four principal criteria, but places particular weight on the speed of getting workers into new jobs, whereas the British system, with more entrenched longterm unemployment, places more weight on the ability to retain the worker in a new job over a longer period.
Experience from advanced, middle-income and developing countries demonstrates that all systems are refining and adapting their performance measures, incentives and instruments to changing economies and markets. Performance indicators are particularly important when services strive to improve the quality of the job match for job seekers. Past research has indicated that inclusion of indicators such as those related to job retention, wages and benefits, earnings gains, all help diminish any incentive to place participants in poor quality jobs in both public and private providers.46
Performance-based contracting, including the elements of paying for employment outcomes, was first developed in the United States and subsequently extended to the UK, Australia and the Netherlands.47 Other countries have recently implemented or are testing changes in their employment services contracts with several, such as Germany, France, Denmark, Sweden, and Israel, experimenting with job outcome performance contracts and delivery through for-profit providers.
The United Kingdom originally benchmarked public against private providers and evolved to its current “pay for performance” system which pays more for longer job insertion. Latin American systems have begun making strides in management reforms and innovations via the use of “quality control” systems, these can include measuring customer satisfaction, efficiency of service, and quality of placement which are certified to international (ISO 9000) standards. San Luis Potosf, one of the state systems in the Mexican National Employment Service, has been recognized for its best practice in such systems (Kappaz and Rosa 2009).
Stage 2: Monitoring and Data Collection
- • Comprehensive beneficiary databases which record and track clients of the service, including among the following indicators: instances of service delivery (e.g. date of visits to walk-in centers), the type of service delivered and record of completion, employment (e.g. initiation date, salary). For those using case management, the individual (case) plan is recorded including the schedule of agreed services, follow-up visits to the intermediation center, and employment retention over defined periods (e.g. six-month intervals), and accomplishment of the plan.
- • Comprehensive data collected on beneficiary characteristics supporting as well impact evaluations (e.g. age, sex, employment history) either in select regions or nationally, potentially including race, disability and indigenous origin.
- • Administrative and performance input data, including spending (on personnel, programs, compliance), staff hours, staff-to-clients-served ratios.
- • Labor market indicators measured typically via a labor observatory and labor force surveys, including unemployment rates by sector, age, salaries, benefits.
Stage 2: Performance, Output and Impact Indicators
- • Impact: rate of labor market insertion, disaggregated by gender, ethnic group/indigenous peoples (if data permits).
- • Outputs: vacancy fill rate, individual plans completed, referrals (e.g. to training, counseling, social services).
- • Performance: rate and increasing rate of labor market insertion; increase in job listings and placements at higher income levels; comparative performance of regional offices.
- • Cost-efficiency: cost per worker who finds employment, unit costs for extended programs and services, cost of training versus other programs for similar outcomes.
Stage 2: Evaluation Types and Instruments
- • Impact evaluations, experimental and quasi-experimental on both specific training programs managed by the intermediation service and the intermediation service itself. More advanced systems may be able to combine impact plus cost of intervention (versus alternatives).
- • Cost-benefit analysis48 of specific services as well as of the intermediation service overall (e.g. cost per worker inserted in new employment, benefit savings with unemployment insurance, social assistance).
- • Process evaluations examining the efficiency of delivery of service.
- • Client and employer satisfaction surveys.