Monitoring the information society strategy

It is too early to evaluate the impact of the various initiatives undertaken in the Plan, but any evaluation must consider both the direct and indirect effects. The direct effects are those related to the short-term impact on the penetration of broadband services. These can be measured easily (as long as the metrics are gathered) as the number of connections provided pursuant to a particular PPP project.

The indirect (long-term) effects are much more difficult to foresee and quantify. Some connected people or households might have been able to obtain a broadband connection from another source not provided by public money, although this seems unlikely in view of the targeted areas. A principle difficulty will be the need for more detailed data collection. It will go beyond basic subscription metrics and focus on issues such as the services used and the impact of using them (either in terms of public sector cost savings or benefits to the end user).

While the Plan clearly sets out its intention of putting in place “performance indicators to measure results”, available evidence suggests that little data was collected that could be used to judge the impact and performance of investments. And where it was collected, there is no evidence that it was used to assess the impact of the programme. Different metrics could be used for measurement and impact assessment, but the challenge in many instances would be to judge the relative improvement due to the Plan, which is by definition unknown. For example:

• • Where networks were deployed using funds, would investment otherwise have been made?
• • Where networks were deployed, are they used?
• • When used, how were they used?
• • What services could users avail themselves of?
• • What services did the government deliver differently?
• • What was the benefit of such uses?

It is essential to implement effective governance mechanisms to ensure that public money is used appropriately, to check how decisions are made, and to ensure that stakeholders behave in the right way. It is also important to ensure that public money invested in broadband projects is utilised to deliver tangible benefits so that funding continues to be provided only where it is appropriate.

The Plan and, more specifically, the New Avanza Infrastructure Plan, does require operators to file data on the use of the networks that have been deployed using public funds. For example, operators that were awarded funds in 2010 are required to submit a monitoring report every six months starting in July 2012. Some required information is: date of service availability, technical characteristics of the service, the number of new subscribers in those areas where funds were granted, and the number of households connected to NGA networks at speeds higher than 50 Mbps.

While such information filing should continue, there is no evidence that it will be used to monitor the success or failure of the project in terms of economic and social benefits and/or whether any conclusions may be drawn from this information. Nor, to date, has there been any supply- and demand-side cross-checking (e.g. whether an e-government service in a particular area is being used by citizens connected to networks deployed with public funds). There is a clear need for a comprehensive monitoring plan that makes it possible to judge whether the programme has been a success.

Public bodies could usefully undertake periodic monitoring, as they are often the bodies that actually distribute public funds. Local groups and partners could also act as monitors, the advantage being they would be close to operations on the project and thus able to spot any “issues” very quickly. Similarly, if a regional or municipal public organisation were to monitor a programme, it would add financial and political clout to the work.

The disadvantage of public bodies monitoring a project, however, is that they are not directly involved in project implementation. An additional formal process would have to be put in place whereby the operating organisation reports to the regional municipality on a regular basis - as with the Dorsal project in France.¹⁶ That being said, the advantage of a central government body such as SETSI acting as a monitoring body is its awareness of the high-level objectives of any national broadband policy and its close links to the market-specific expertise of the regulator. Public bodies should use milestones and deployment controls to ensure that the roll-out goes according to plan.

To continue with the example of Dorsal, there the network operator specifies to the contracting agency metrics for operational readiness, faults, maintenance schedules, take-up rates and, finally, network performance in terms of speed and quality parameters. These also seem to be important parameters in the context of projects funded under the Plan. Ensuring compliance raises its own concerns and since SETSI does not take a stake in the projects that it finances, other forms of governance that have been effective in other regions/countries could be considered.

In the rural development programme in Sweden (Swedish Ministry of Agriculture, 2008), for instance, most investments are monitored at county level. Investment has to be used as specified in the funding application for a period of five years following the project (monitored by the Department of Agriculture which administers the funds). All projects must meet open-access requirements too, a requirement also made in the previous information society strategy. The Board of Agriculture monitors a 5% sample of projects to ensure that they meet all the relevant conditions. Monitoring requirements are set out in the contract with the network supplier, with payment of funds often conditional on meeting obligations. Such obligations may be launch dates, customers connected, and services provided (normally measured by how many subscribers have signed up with an Internet service provider - ISP).

Finally, governance can be exercised through alternative methods of influence. This approach may be necessary when no formal governance arrangement is possible between the contracting agency and the network operating organisation. However, even in these circumstances, the contracting agency is still able to monitor the project, and refer any undesirable activity to another enforcing body (e.g. the regulator).

In the North Karelia Region in Finland (Regional Council for North Karelia, 2010), the successful bidder for each project has to guarantee 30 years of service provision. One of the North Karelian Regional Council’s tasks is to check that the network performance is compliant with the terms of the contract agreement, even though it has no specific tools that it can use to perform this task. As a result, most monitoring is carried out through customer feedback. When problems are spotted, they are signalled to the Finnish regulator (FICORA) to apply penalties where appropriate. With SETSI already responsible for quality-of-service issues and overall consumer complaints on telecommunications, it would make sense that it (rather than the CMT) facilitated customer feedback - and even sanctioned poor performance - in areas where networks have been funded with government support. Such an arrangement would complement the monitoring of other specific measures.