# Expected Value and Cost of Trials with a Delayed Reimbursement Decision (DT Versus AN)

The value of trial information undertaken with delay now compared to adopting now based on current expected INB (DN vs AN) is not affected by costs of reversal, as delaying avoids reversal with what turn out with further research to be bad decisions to adopt a strategy with current evidence of expected while uncertain INB. However, there is an opportunity cost of delay in undertaking further research for cases of interest with positive expected while uncertain INB. The expected value of this opportunity costs is the prior expected INB that would otherwise have arisen with adoption now for the patients on standard therapy within and outside the trial arising during recruitment, follow-up and analysis time, that is, until a revised decision is made.

Opportunity costs of delay with DT arise for patients both outside and inside trial receiving standard care and hence contribute to both

(i) Variable opportunity costs dependent on size of the trial (n) given time of recruitment of *b _{0}(k* x

*2n/a - n)*or equivalently

n(b_{0}(2k/a - 1)) as a function of trial size; and

(ii) Fixed opportunity costs independent of trial size associated with the fixed time period of follow-up and analysis of *b _{0}rk.*

These opportunity costs with DT are additional to direct costs of trialling, which include the fixed (C_{f}) and the variable cost of trialling each patient (C_{v}) which for a standard 2 arm RCT of *n* patients per arm results in a direct cost of C_{f} + 2nC_{v}. Consequently total costs to decision-makers of trialling with DT are

Population EVSI is found by multiplying the per-patient EVSI by the size of the population to whom information from the trial is valuable, which, following Eckermann and Willan (2008b), will be reduced when accounting for the duration of trial accrual, follow-up, analysis and reporting. Time needs to be explicitly modelled allowing for study accrual rate (a), time for follow-up and analysis (t).

In determining population EVSI, the expected time for information updating is

This time reduces patient population over the time horizon *(T)* given an expected patient population incidence rate (k). Hence, the remaining patient population within jurisdiction who can benefit from trial information from a trial of size *n* per arm is

Multiplied by EVSI per patient, this represents population EVSI.

Given a time horizon T, population EVSI is shown in Fig. 5.4 to initially increase while the proportional increase in the value of additional evidence per patient is greater than the reduction in the patient population with additional recruitment time. Population EVSI reaches its highest level at the point where the proportional increase in EVSI per patient is equal to the proportional reduction in

Fig. 5.4 Population EVSI as a function of study size *(n)* with related reduction of time horizon the patient population and then reduces eventually to 0 where there is no population left, corresponding to the trial size where trial reports at the time horizon *T. *That is, population EVSI reduces to 0 where the trial has eaten up the time horizon when it reports, as there is no population left to benefit from additional evidence, where

or equivalently at the trial size where

However, note that given positive expected costs of undertaking research (direct and opportunity cost with delay and trialling) in maximising ENG, we are only interested in trial sizes from Fig. 5.4 where EVSI is still increasing with *n.* Given EVSI and total costs as a function of n, the ENG of proposed trials with delay relative to adopting now can then be found as population EVSI minus the expected cost of that trial size per arm, including the opportunity cost of delay expected to be incurred by patients who receive the old technology while the trial is performed. It is not optimal to undertake trials where ENG is negative, while it is potentially optimal to trial where ENG is positive. Hence, a positive ENG is necessary while not sufficient to establish the optimal trial design, where ENG or return on investment is maximised.

However, a negative ENG for a given trial design does not imply that there is no research design for which ENG is positive and hence current evidence sufficient to AN. For example, if EVSI is less than the expected costs of a frequentist- designed trial of given size, this does not mean that there will not be another trial or research study where EVSI is greater than expected cost and hence have positive ENG. Other sized trials or low-cost alternative research to reduce decision uncertainty can have EVSI greater than their expected cost to decision-making. Hence, consideration of EVSI relative to expected costs of frequentist-designed trials can only ever partially inform questions in relation to optimal trial design and decision-making to the extent that certain research designs are excluded where ENG is negative.

In optimising ENG of trial design and decision-making, we have noted that the expected value of sample information EVSI(*n*) per patient, and the population to which that EVSI applies to find population EVSI as their product, as well as expected direct and opportunity costs depend on the sample size (*n*). Hence, expected net gain ENG(n) = EVSI(n) - ETC(n) or return on research (ENG per dollar allocated to research funding) is also a function of the sample size, *n*.

Consequently, ENG can be optimised as a function of sample size, with sufficient evidence to adopt now for cases of interest with positive while uncertain INB if ENG is always negative. That is, if EVSI is never greater than expected cost of

Fig. 5.5 **The research option value and opportunity cost of delay (DT vs. AN) (Source: Eckermann and Willan (2007))**

alternatively sized trials. Otherwise, over some range of trial sizes EVSI will be greater than expected costs, and hence ENG with DT will be positive and optimised at a trial size as in Fig. 5.5 where marginal value from trialling (slope of the population EVSI curve) is equal to the marginal cost of trialling (slope of the expected total cost function). For trials smaller than this, at the margin, the expected value is greater than expected cost of increasing the trial size, while for trial sizes larger than this the expected cost is greater than the expected value of expanding the trial size at the margin.

Alternatively, return on research (ENG per research dollar invested) can be optimised where

is maximised and compared with other options, including AT where feasible but more generally other options for investment in research or services (Eckermann, Karnon and Willan 2010).

In interpreting Figure 5.5, dealying the decision about whether to adopt while trialling (DT) ensures feasibility of the collection of further evidence. This additional evidence has expected value in reducing the expected opportunity loss of adopting the strategy maximising expected net benefit with current evidence and represents the EVSI with DT. The expected value of sample information for DT versus AN represents a research option value of delay (Eckermann and Willan 2008a) and is not affected by costs of reversal.

The expected value of sample information per patient with delay and trialling (DT) at time t is given by

function for a normal distribution with mean b_{i} and variance *v, i* = 0, 1.

Population EVSI for a trial of size *n* with delay *(EG _{D}(n)),* following Eckermann and Willan (2007) is then given by

where *N _{0} = kT* is the expected population over the time horizon now, reduced by the incident population arising over the time until the trial reports ((t +

*2n/a)k),*to represent the remaining population to which the trial evidence has value. In considering the EVSI per patient components to which this population estimate applies in estimating population EVSI:

where Ф(.) is the cumulative distribution function for a standard normal random variable.

A closed-form solution formulation for

is provided in Willan and Pinto (2005a, b) as previously indicated.

Now, the total cost of delay and trialling as a function of trial size per arm (*n*) given opportunity cost of delay arises for all patients except the *n* in the active arm in the trial is

The expected net gain from delay and trialling with a trial of size *n* is consequently

Hence, given current estimates of uncertainty in relation to incremental net benefit (b_{0}, v_{0}), the incidence and accrual rate of patients (k, a), fixed and variable costs of trialling (C_{f} and C_{v}) and the time horizon (T) over which evidence has value, the expected net gain of trialling while delaying ENG_{D}(n) can be optimised with respect to trial size (*n*).

For positive ENG and trial size *(n),* the optimal sized trial and ENG with delay is

where (и*) is the optimal sample size for DT versus AN if ENG^ > 0.

If ENG*_{d} < 0, the optimal sample size is 0.

The decision rule for DT versus AN is seen graphically in Fig. 5.5. The TC_{D}(n) line has intercept

and slope

If the EG_{D}(n) curve lies below the TC_{D}(n) line for all n, then ENGD < 0 and the optimal sample size is 0.