Roadmap for implementing a systematic, data analytics-driven process

To the extent that evidence-based decision-making has been embraced by decision-makers in criminal justice, there is evidence to suggest the impacts have often been positive. For example, some evidence of the positive impact of evidence-based decision-making in criminal justice can be found in the adoption of proactive conununity-based law enforcement approaches, innovative court programs designed to divert individuals away from the criminal justice system, and innovative treatment and intervention programs for known offenders that have had demonstrable impacts, such as preventing offending or reducing the likelihood of reoffending. Key examples of success along these lines tell us that using high quality evidence and data can have demonstrable positive impacts on criminal justice policy. In the hands of properly trained experts, big data, and particularly novel analytics, also has the potential to ‘investigate the hard questions’, such as identifying whether and to what extent operational biases may exist in the provision of law enforcement services, as Skillicom et al. demonstrated in Chapter 5. This in turn produces information that can be used to correct such problems if they exist. Of course, this is much easier said than done.

Possibly more often than anyone would like to admit, research is often conducted, whether in academic criminology or in criminal justice contexts, with key answers in mind prior to the research process, and a methodology that facilitates or reinforces the (re)discovery of these answers. Examples of this have been highlighted in practical contexts where research is commissioned to demonstrate the effectiveness of a treatment program or certain policy that is ‘known’ to work. This is antithetical to the scientific process of knowledge discovery, but nonetheless remains an easy and all-too-common trap to fall into. Therefore, in Figure 8.2, we revisit some basic concepts about the scientific process of research, with an eye toward providing guidance, particularly for those conducting research in industry, and how big data and analytics can fit into this process. A roadmap on implementing a systematic data analytics-driven process is proposed.

There are no shortages of challenges faced by different facets of criminal justice systems that are tasked with preventing offending; detecting and investigating crime; promoting community safety; ensuring fairness and due process of those suspected of committing crimes; incapacitating serious and violent individuals; ensuring proportional punishment of convicted offenders; addressing rehabilitation needs; and preventing re-offending, to name a few. Internally, criminal justice organisations face many similar issues to those of any busmess, arguably to a much larger extent, such as employee turnover, employee burnout, monitoring performance, regulation and compliance, competencies and recruiting the best talent, customer service, the pace of technological change, and, most pertinent for the current volume, how to deal with increasingly vast amounts of data and ensuring their integrity.

The first two steps in the roadmap in Figure 8.2 are incredibly important and intimately linked. The first of those steps is organisational challenge. It involves

Figure 8.2 Roadmap for implementing a systematic data analytics-driven process

Big data: generic roadmaps for practice 127 identifying a specific challenge(s) that an organisation seeks to investigate and overcome through a systematic data analytics-driven process that is in line with broader organisational priorities, whether they are internal or external, as well as its mission and values. This step may most likely require asking many questions to the organisation in order to ensure that the challenge(s) are accurately identified, and the priorities correctly set up. In order to set up the right priorities, some initial exploration may be required if data already exist. The next step in the process is to formulate researchable (or ‘addressable’) questions to address the challenge and the priorities that were identified in step one, and next, determine whether and to what extent data exist to answer the questions. This is one of the most important steps in the process, particularly considering the context of big data, and one with which most organisations would struggle. Here, consistent with identifying the challenge and setting up the priorities, we expect that organisations will need to sit down and tailor the questions according to the mission and objectives of their respective organisation. In this sense, the questions at first do not need to be ‘perfect’ at all. Questions are often refined during such a process where data that were initially thought to be irrelevant may become relevant. While it is important to not use data that do not fit the questions, it is also essential to remember that it may be worth exploring the data in hand if it exists. This process may actually feedback into formulating ‘researchable’ questions to examine for the organisation, and in the end, speed up the process leading to a positive outcome for an organisation.

However, just because an organisation collects vast amounts of data, this does not necessarily mean that these data are automatically well suited to answer certain research questions or even be used all together. It is important to avoid the tendency to revise important research questions to fit the data available to analyse them; a common adage in fust year research methods courses in social sciences is: ‘garbage in = garbage out’, and this must be applied here. The quality and appropriateness of data is paramount to gaming valid answers to research questions. If the data are not suited to a particular question that carries substantial organisational importance, even if there is a lot of it, it is probably in the best interest to collect new or complementary data. One way to assess whether and to what extent the data may be suited to answering a research question is by finding out what is already known about the question, how it has been researched in the past, and what kinds of data were utilised in previous contexts. For example, corrective services are quite often interested in what programs for prisoners are effective in reducing reoffending outcomes post-release. Of course, there is robust international literature on evaluations of different progr ams delivered by corrective services from around the world that employ many different research designs and data sources. It is important to note that rarely will the data, whether already available or to be gathered, be ‘readily applicable'. In most contexts, formulating the right questions will most likely imply finding the right equilibrium between the context and objectives of the organisation and ‘good enough' data.

The next step involves identifying concepts that are measurable to answer the research question. For example, in the context of evaluating the effectiveness ofa prison program, what is meant by post-release outcome? Probably the most common post-release outcome that is employed by studies is that of reoffending, whether measured by new charges, convictions, and/or returns to custody, all of which are routinely collected administrative data and obtained through either law enforcement or collective services. However, numerous other post release outcomes are not routinely collected, especially if an individual does not return to the criminal justice system. For example, these can include employment, entering a stable relationship, joining a community organisation, volunteering, and so on. The point is that it is important to clearly identify key concepts in research that are measurable and understand what aspects of the research question they do and do not inform.

Once the research question(s) is finalised and key concepts are operationalised, the question of data extraction enters the picture. Are high quality and appropriate data already available to adequately investigate the research question posed, and elucidate the necessary concepts that have been operationalised in a specific fashion? If the answer is yes, then these data need to be extracted from larger databases held within an organisation. A common mistake made when vast amounts of data are available in this context is the tendency to believe that they all need be analysed in their entirety to produce a robust analysis. Again, if well-grounded research questions have been developed, and the variables adequately operationalised, then data extraction should involve those key variables pertinent to answering the research questions, which better fits the reality of generating outcomes rapidly enough for organisations to make the best of the data in the short-term. If these key variables are unavailable or there are other issues (e.g., large amounts of missing data), then it is necessary to develop another instrument or tool to collect/acquire the appropriate data or simply design and include the variables needed into the cunent process.

Whether the data are readily available or a new data collection procedure (e.g., an online survey) is developed, a clearly defined sampling frame may be necessary to extract the most optimal sample to analyse the research question. In an administrative data set, for example, this might include all of the relevant cases, whatever the units of analysis may be, over a certain time period, in a certain jurisdiction, and so on. In the context of big data, sampling increasingly involves ALL cases, but even so, ALL cases are typically restricted to certain time periods for which data exist or specific geographic areas, etc. In effect, this decision once again goes back to the research question and what the best data available are to answer it. Therefore, it is also necessary that there is clarity around the units of analysis. Is the research question concerned with people? Organisations? Incidents? Once the unit of analysis is selected and the sample is clearly defined, then data extraction can take place: data can be retrieved within the specified parameters described earlier, or, in the case that a new tool such as a survey is developed to collect data, pilot testing of the instrument can take place.

Much time and effort in research projects is often spent focusing on data analytic decisions and deciding on analytic/statistical techniques to analyse data, often from the outset of projects. Importantly, as Figure 8.2 shows, this is in fact one small

Big data: generic roadmaps for practice 129 element of the broader roadmap. We also argue that while it is clearly important data analysis is done properly, this element of the roadmap is the least crucial because if data analyses are incorrect, they can be redone and collected much easier than if crucial errors in any of the previous steps are evident. In effect, if there are critical errors or limitations in data collection, operationalising variables, or sampling, there is much higher risk to the overall project fidelity than if a correctable data analysis error is made. Note that hi the upper steps of the roadmap there are more bidirectional arr ows, indicating that these steps are more of an iterative process involving a cycle of activities in order to fine tune, but the roadmap becomes more streamlined from this point on, particularly once data have been collected. Furthermore, there are a multitude of different ways to analyse data, some more robust than others, and decisions around what data analysis techniques to use have as much to do with research questions as they do with the levels of measurement of variables, distributions of variables, and missing data. Therefore, while data analysis decisions are crucial, they will also be the least relevant if the prior steps of the roadmap are compromised; again, ‘garbage in = garbage out’.

The end of the roadmap for generating empirical evidence from big data in order to improve an organisation's performance involves the dissemination of the findings and what these mean for the organisation - the most critical step along with the first and second steps of the process (the entry and exit points highlighted by Davenport, 2013). This is the stage where evidence-informed (i.e., based on research) implementation processes and changes can be actioned. Again, these may be internal (e.g., new organisational policy) or external (e.g., evidence to inform programs offered to clients by the organisation) to an organisation, but without the communication of the research results any actionable change will be tmlikely to occur. What’s more, if research results are not communicated in a fashion that is appropriate to the specific target attdience, the research again runs the risk of failing to see any resultant actionable change or impact. As an example, research presented at academic conferences, typically, will look very different from research presented to policy makers, or to internal members of an organisation, or to the public. In the academic context, when it comes to the dissemination of research, much time and effort is spent on communicating important methodological details to open the research to scrutiny and feedback surrounding how robust the findings are and the validity of the conclusions put forth. While this is clearly important to any research, in the policy context, of primary relevance are the findings and implications of the research, along with the presumption that the methodology is robust and the findings valid. The reason for and implications of research are typically also very' important to internal members of an organisation for whom the research may directly affect. Finally, communicating resttits of research to the public is often typically scaled down with a focus on key findings and broader implications. Regardless of the audience, the point is that the dissemination of research needs to be tailored to the appropriate audiences; otherwise its impact will likely be diminished.

In the last step, setting ttp an implementation process for change based on the findings arising from analytics may in fact depend as much on the effectivedissemination and communication of the results as it does on the research itself. As part of the implementation of any transformation, a plan to evaluate the changes will need to be developed, as well as the metrics for success, which may involve setting up and monitoring KPIs. In short, evidence that is quantifiable will need to be collected to determine the extent of the success of the implementation. For instance, data analytics could be used to gather insights on the skills and interests of staff and link those to high performance as a priority for the organisation based on its mission, values, and objectives. Then several positions could be allocated to staff accordingly into the roles that will maximise individual and professional growth and success and, as a result, generate better outcomes for the organisation (Davenport, Hanis. & Shapiro, 2010).