Following the approval of a new drug by regulatory authorities, the evaluation of the benefits and risks continues throughout the lifecycle of the product. This typically consists of a transparent synthesis and communication of data from diverse sources relating to the drugs effectiveness, safety, tolerability, and patient preference. Since the effort involves extracting and integrating information from a large amount of heterogenous data, regulatory agencies have established guidelines and other appropriate mechanisms to ensure appropriate analysis, interpretation, and communication of the benefit-risk profiles of authorized drugs, with a view to protecting public health and advancing health outcomes (Guo et al. 2010).
There have also been parallel initiatives undertaken by the pharmaceutical industry to align with the expectations of the regulators with respect to the enhancement of the approaches for assessment of the benefit-risk of medicines. In the US, the Pharmaceutical Research and Manufacturers of America (PhRMA) initially developed the so-called Benefit-Risk Action Team (BRAT) Framework, which was eventually transferred to the Centre for Innovation in Regulatory Science (CIRS), a neutral independent UKbased subsidiary company (Levitan 2012). Efforts have also been underway to establish good-practice guidelines for conducting agencies to aid healthcare decision-making (Thokala et al. 2016).
Despite the numerous efforts and initiatives by both regulators and pharmaceutical companies, there is still a demand for a standard template to harmonize the evaluation of the benefit-risk profiles of drugs and the documentation and communication of decisions. The Universal Methodologies for Benefit-Risk Assessment (UMBRA) is an example of a framework proposed by representatives of regulators and the pharmaceutical industry, with the aim of establishing common elements of an overarching, internationally acceptable, standardized benefit-risk approach (Centre for Innovation in Regulatory Science 2012).
In this section, we highlight pertinent methodological and regulatory issues relating to the benefit-risk assessment of medicinal products and provide a summary of selected tools that are currently accepted for use by sponsors and regulatory agencies.
Methodological Considerations in Benefit-Risk Analysis
Broadly, benefit-risk assessment may be carried out either in a descriptive/qualitative or quantitative framework. Descriptive approaches typically use metrics for structuring relevant benefits and risks and involve a thorough assessment of treatment performance data on benefits, risks, and convenience of use, without applying weights. On the other hand, quantitative approaches aim at combining data on treatment effectiveness, safety, and ease of use, with stakeholder preference information, typically using a weighting scheme for various benefit and risk criteria. Preference-based approaches are generally applicable in complex situations involving several criteria and multiple treatment options. Some of the widely used quantitative approaches permit integration of data into a single measure, thereby facilitating and ensuring transparent communication of benefit-risk decisions. However, these quantitative methods may obscure the underlying data and may not be necessary if the more direct qualitative and graphical summaries are clear.
Development of a quantitative model requires determination of appropriate benefit and risk criteria, which relate to distinct and nonoverlapping clinical outcomes of interest for the treatment options under consideration. Estimates of drug performance on each criterion should then be obtained, including the associated measures of uncertainties of the estimators. When data comes just from a single RCT, the performance measures may be computed and analyzed with either a frequentist or Bayesian approach. In certain cases, data may be available from multiple trials, in which case the relevant information may be combined using methods for standard metaanalysis or network meta-analysis, depending on whether a common comparator is available or not.
A major issue associated with the analysis of data to establish the benefitrisk profile of a drug is the quantification of the uncertainty and the clinical relevance of the observed effect sizes. This in part is because the data used in benefit-risk assessment may come from diverse sources, including RCTs, epidemiology studies, literature review, or spontaneous adverse reports. Further, the uncertainty may arise in at least two other ways, namely, in the subjective choice of the criteria, or as a consequence of sampling variability. In the latter case, the handling of sampling variability requires application of suitable statistical methods. On the other hand, subjective uncertainty generally requires execution of extensive sensitivity analyses.
A commonly used quantitative approach for benefit-risk assessment is the multiple-criteria decision analysis (MCDA) method, introduced by Keeney and Raiffa (1976). It involves a decision-making process that brings together different options on multiple criteria of benefits and risks into an overall assessment, through scoring and weighting. The purpose of scoring is to quantify each criterion into a common scale, while weighting ensures comparability of the units on the criteria so that they can be combined into an overall scale. More specifically, suppose the mean of the i,h criterion is denoted by ¿q, with an associated score function and weight, s, and wit respectively. Then a measure of an overall assessment is given by:
Inference about 9 can be made by replacing ¿q with a suitable estimator. Approximate confidence intervals and test statistics may be constructed using the central limit theorem or via simulations.
An attractive feature of the MCDA approach is that it permits combining the subjective value judgments and the clinical evidence in a transparent fashion. However, its limitations include the fact that it does not handle uncertainties of outcomes and, most importantly, that it requires exact specification of the values of the preferences and weights.
To mitigate the limitations of the standard MCDA method, enhanced approaches have been proposed, assuming distributions, rather than point values, for the weights and score functions. One example is the approach proposed by Tervonen et al. (2011), dubbed the stochastic multicriteria acceptability analysis (SMAA), which is intended to account for the uncertainty in the criterion measurements as well as preferences information. More specifically, the approach assumes the weights and the criteria are random variables with joint density functions. A rank acceptability index is then computed as a multidimensional integral over the criteria distributions and the favorable rank weights.
In the above SMAA framework, estimation of the distribution of the criteria is often challenging. Waddingham et al. (2016) proposed an approach that involves a synthesis of the evidence from other studies. Saint-Hilary et al. (2017) provided a method for constructing the weight space of SMAA. More recently, Li et al. (2018) introduced a framework in which Bayesian meta-analysis and SMAA are jointly used to synthesize accumulating evidence from early stages of the clinical development to late stages in benefit-risk assessment.
Software programs are available for implementation of some of the abovementioned techniques. In R, the CRAN packages, hitandrun and SMAA can be used to implement the methods. In addition, the Aggregate Data Drug Information System (ADDIS) software can be used for both SMAA and MCDA (van Valkenhoef et al. 2013).
In addition to the structured qualitative and quantitative approaches mentioned above, there are several semi-quantitative techniques that are in routine use, depending on the scenario at hand. Examples include such procedures as number needed to treat (NNT), number needed to harm (NNH), decision trees, and Markov models. While NNT and NNH are measures that are apparently easy to interpret, they are based on the inverse of the risk difference with often wide upper-confidence bounds and thus should not be presented without the confidence limits. A detailed description of the approaches may be found, e.g., in EMA (2010a).
Lastly, as pointed out earlier, sensitivity analyses are an essential component of benefit-risk assessment to establish the robustness of the results against the treatment performance or preference estimates. This may involve either evaluating one parameter at a time or several parameters simultaneously. The latter often entails defining suitable distributions to the parameters, as is the case in the SMAA approach described above.