What Causes Reversal of Treatment Effect with Relative Risk?
Relative risk is not a symmetric metric with respect to framing binary outcomes from a positive (utility bearing) or negative (disutility bearing) perspective. That is:
In indirect comparisons, interaction between this lack of symmetry and differences in common comparator (e.g. placebo) risks across trials creates conditions for reversal of treatment effect with alternative framing of outcomes. For example, as observed with indirect comparison of natalizumab versus interferon, where relative risk was less than 1 with binary outcomes framed from both a utility and disutility bearing perspective (RR_{UAB} and RR_{DAB} < 1). Equally, reversal of treatment effect can arise where relative risk is greater than 1 with alternative framing of outcomes for indirect comparison from a utility or disutility bearing perspective (i.e. where RR_{UAB }and RR_{dab} > 1; Eckermann et al. 2009).
Reversal of the direction, but more generally differences in the extent, of treatment effect estimated with the use of relative risk in indirect comparison is inconsistent and undermines exchangeability required to support indirect comparison and creates clear scope for selection bias in framing outcomes. To overcome these problems with relative risk, their cause needs to be addressed, the lack of symmetry of the relative risks metric with alternative framing of binary outcomes. That is, a symmetric metric is required to allow consistent estimation of the direction and extent of treatment effect in indirect comparisons.
Odds ratios (ORs), as the ratio of odds, where odds are the probability of an event happening divided by the probability of it not happening:
provide a symmetric measure of treatment effect. That is for comparisons between treatments a and b for a binary outcome (which can either be framed as having an event or no event):
Eckermann, Coory and Willan (2009) show that the symmetric odds ratio ensures consistent estimation of the direction and relative extent of treatment effect in indirect comparisons. Table 3.3 illustrates this for the case of indirect comparison between natalizumab and interferon.
Considering progression first (left-hand side of Table 3.3), the odds ratio of MS progression for natalizumab versus placebo is 0.35 ( = (0.333/0.667)/(0.590/0.410)), while for interferon versus placebo is 0.42 ( = (0.687/0.313)/(0.839/0.161)). This leads to an indirect comparison odds ratio with MS prgression of 0.83 (0.35/0.42) for natalizumab versus interferon. Hence, indirect comparison of the odds ratio for progression favours natalizumab, given multiple sclerosis progression is an event
Table 3.3 Consistent estimation of direction and extent of treatment effect in indirect comparisons with OR
Natalizumab progression |
Placebo progression |
Natalizumab no progression |
Placebo no progression |
33.3% |
59.0% |
66.7% |
41.0% |
Natalizumab vs. placebo Progression OR = 0.35 |
Natalizumab vs. placebo No progression OR = 2.88 |
||
Interferon progression |
Placebo progression |
Interferon no progression |
Placebo no progression |
68.7% |
83.9% |
31.3% |
16.1% |
interferon vs. placebo Progression OR = 0.42 |
interferon vs. placebo No progression OR = 2.38 |
||
OR for progression n vs. I 0.83 (favours natalizumab) |
OR for no progression n vs. I 1.21 (favours natalizumab) |
the target population want to avoid and hence an odds ratio of progression less than 1 is favourable for natalizumab.
Now, consider indirect comparison for no progression (right side of Table 3.3). For binary events the odds, and odds ratios, of no events are in general the reciprocal of those for events. The odds ratio for no progression with natalizumab versus placebo is 1/0.35 = 2.88 = (0.667/0.333)/(0.410/0.590), while for interferon versus placebo is 1/0.42 = 2.38 ( = (0.313/0.687)/(0.161/0.839)). This leads to an indirect comparison odds ratio for no progression of 1.21 (2.88/2.38) for natalizumab versus interferon, which is equal to the reciprocal of that for progression (1.21 = 1/(2.38/2.88) = 1/0.83).
Hence, whether framed as disease progression or no disease progression, indirect comparison of odds ratios favours natalizumab with the same direction and extent of treatment effect. In general, Eckermann, Coory and Willan (2009) show that the problems of lack of consistent estimation with alternate framing with relative risk in indirect comparisons are overcome with odds ratios, which ensure consistency in the direction and extent of treatment effect with alternative framing of binary outcomes. The reciprocal nature of symmetric odds ratios ensures that for binary outcomes the indirect comparison estimates of an event (mortality, morbidity, progression, etc.) are the reciprocal of that without the event (survival, no morbidity, no progression).
The bottom line then in choosing a robust metric for simple or complex forms of indirect comparison to improve exchangeability of evidence with binary outcomes is that:
- (i) A relative treatment effect is required to inform trial-based or translated jurisdictional evidence of absolute incremental effects, where absolute treatment effect is expected to be modified by baseline risk.
- (ii) Of candidate relative treatment effects (relative risk or odds ratio), only the odds ratio provides a consistent estimate of the direction and extent of treatment effect and prevents selection bias with alternative framing of binary outcomes.
These advantages of odds ratio over relative risk with indirect comparisons also extend to network analysis (i.e. more complex forms of indirect comparison; Welton et al. 2012). Further, they also extend to translating relative treatment effects to estimation of absolute risk difference in any given jurisdiction of interest with direct or indirect comparisons, as we now show in Sect. 3.5, following Eckermann, Coory and Willan (2011).