In propensity score matching, treated and untreated individuals are “paired” based on their PS (Figure 1). Depending on the prevalence of the treatment, individuals can be matched on a 1:1 or 1:many basis. Nearest-neighbour matching identifies pairs with the closest PS. In “matching without replacement,” an individual can only be matched once before being removed from the matching pool. This means pairs generated later in the matching process may have larger differences in their PS (21). Matching with replacement allows control patients to be matched to more than one treated patient. An alternative to nearest-neighbour matching is optimal matching, which minimises the difference in PS between pairs across the whole population. In large populations, nearest-neighbour and optimal matching give similar results (22). Both techniques include a “caliper” to avoid the inclusion of poorly matched pairs. This specifies the maximum acceptable difference in PS for a pair to match, generally accepted as 0.2 times the standard deviation of the logit of the PS to provide the optimal balance of matching quantity and quality (23, 24). Individuals who are unmatched are excluded from further analyses. In practice, as it isn’t always clear what the “ideal” statistical method is, performing analyses using a number of these techniques can help assess how sensitive results are to method specification.

The matching technique should create two groups with an equal distribution of measured covariates (Figure 1). The balance of covariates between groups can be examined using standardised differences, calculated by dividing the difference in proportion (for binary variables) or sample mean (for continuous variables) by the pooled standard deviation. There is no definite consensus on an acceptable standardised difference; a value below 0.1–0.2 is generally accepted (25). Visual diagnostic tools can also be used to examine covariate balance, as demonstrated in our worked example (26). Once the groups are balanced, they can be compared using standard regression analyses. These analyses can be univariable or multivariable, with the multivariable technique including the variables used to generate the PS. A multivariable model compensates for imperfect covariate balance and, if specified correctly, minimises the risk of a biased estimator (27). However multivariable models lose the advantage of having only 1 covariate in the final model, so could be overfitted if the number of covariates is large relative to the number of outcome events. Further, in the event of misspecification of the PS model, this method could increase bias (28).

Inverse probability weighting (IPW, also known as propensity score weighting) creates a pseudo-population informed by all patients with a balanced distribution of measured covariates between groups (29). By doing so, IPW avoids excluding individuals from analyses and may result in better covariate balance than PS matching (30).

Each individual is assigned a “weight” depending on their measured covariates and the treatment they receive. For individuals who receive treatment, their weight is 1/PS, whilst individuals who do not receive treatment have a weight of 1/(1-PS). This means individuals receiving an “unexpected” treatment contribute larger weights to the analysis than individuals receiving their “expected” treatment (Figure 1). Each crude weight is greater than or equal to 1. If some patients have large weights, this can make results unstable. To minimise this risk, weights are frequently “stabilised” before further analysis. This is relevant if a multivariable regression model is being used; stabilisation does not affect univariable models which contain only the treatment indicator (31). Stabilisation involves multiplying the weight by the proportion of exposed patients for the treated group, and by the proportion of unexposed patients in the untreated group (32). Once stabilised, the mean weight for the population should be approximately equal to 1. A regression analysis where each individual is weighted by their inverse probability of receiving treatment can then be performed. As with PS matched analyses, this regression can be univariable or multivariable. The same caveats of the multivariable model in PS matched methodology apply to IPW analyses.

PS techniques have several advantages over conventional multivariable regression models. First, conventional multivariable Cox models require around 10 events per covariate to produce a stable estimate, and combining covariates into a single PS is useful when the population is small, event rate is low, or number of covariates is large (33, 34, 35).

Second, in conventional regression models the treated and untreated groups can systematically differ. This means estimating the effect of treatment on a patient, who would never have been considered for treatment in real life, can be unreliable as the estimation is based on model extrapolations beyond the support of the data. PS matched analyses refer to only those patients who could feasibly exist in either the “treated” or “untreated” group. Whilst PS matched analyses can therefore provide improved real-world results, identifying the population to whom the results are applicable to can be challenging, especially where there is variation in treatment practice between centres.

Third, PS models highlight the limitations within which results should be interpreted. If a large proportion of individuals are unmatched in PS matched analyses, or there are patients with large PS weights in IPW analyses, this signifies poor overlap in covariate distributions between treated and untreated groups and means the likelihood of individuals being allocated to either treatment group is low. As traditional multivariable models extrapolate results to individuals in under-represented covariate strata, this could lead to bias in effect estimates. PS methods can alert researchers to these issues and highlight the limits within which comparisons of treatment options can be made.

PS assumptions (exchangeability, positivity, and consistency) are described in Supplementary Table S1, and it may be difficult to prove these assumptions hold. If the treatment is rare, there may be insufficient data to generate the PS. Further, the PS only encompasses measured confounders. Confounders that are unknown, poorly recorded, or not measurable cannot be controlled for and may not be balanced between groups, leading to unmeasured confounding bias.

In PS matching, unmatched individuals are “lost,” reducing the study size. Individuals with the highest and lowest PS (the “always treated” and “never treated”) are less likely to be matched and are under-represented in the regression models. Whilst there is no “required” proportion of patients that must be matched, the causal effect is only applicable to matched patients, not the whole study population.

In IPW, data from all participants is retained. However, if individuals contribute large weights to analyses, results may be unstable. There is no consensus on what a “large” weight is, and weight stabilisation is often used to minimise this risk. Some advocate truncating weights to a maximum of 10 for more precise estimates, (36) but this may re-introduce some of the confounding that the method aims to remove.

For interested readers, more detailed information on propensity scores can be found at the following references (9, 37, 38, 39).

Finding a suitable IV can be challenging and large multicentre studies are often required. Ensuring assumptions of the IV are met may not be possible (55). Weak instruments may also amplify bias through violation of the exclusion restriction or independence assumption and result in more biased estimates than other analyses (9). Finally, whilst IV analyses can overcome unmeasured confounding, they are less precise as individuals are examined based on estimated not actual exposure (56).

When analysing causal inference studies, it is necessary to consider to whom the causal effect is applicable to. Terms used include the “average treatment effect” (ATE), “average treatment effect on the treated” (ATT) and “local average treatment effect” (LATE).

ATE refers to the effect of treatment on the whole population. This is typically estimated by IPW techniques, which include all study participants. ATT refers to the effect of treatment on only those individuals potentially eligible to receive it and is typically estimated by PS matched analyses. In IV analyses, the causal effect depends on whether effect homogeneity or monotonicity hold. If homogeneity is assumed, the estimate refers to the ATE. If monotonicity is assumed, the estimate refers to the LATE. This reflects the effect of treatment on the subgroup of “complier” patients who receive the expected treatment given their instrument (Supplementary Figure S1). As complier patients cannot be identified from within the study population, the LATE has limitations in informing practice/policy decisions.

As the ATE, ATT and LATE refer to different groups of patients, their effect sizes can differ. Differences can aid the interpretation of study findings by providing insights into the effect of treatment on different groups of patients, and do not necessarily signify failure of a technique.

In each of the above analyses, the final regression model that generates the causal effect can either be “marginal” or “conditional.” Models which contain only the treatment (or predicted treatment in the IV analysis) and outcome generate marginal treatment effects. Although the characteristics of treated and untreated individuals should be similar through the PS matching, IPW or IV techniques, generating truly “exchangeable” groups of treated and untreated patients remains difficult. Models which condition on (and hence adjust for) confounders in the final regression may reduce such residual imbalances and generate conditional treatment effects.

The effect sizes from marginal and conditional regression models differ and cannot be directly compared (57, 58). If the model has been correctly specified, marginal models estimate the average effect of treatment on the population (i.e., the effect of moving the population from being untreated to treated), whilst conditional effects are more individualised and apply to groups of patients within covariate levels (i.e., the effect of moving an individual person from being untreated to treated). Marginal treatment effects are frequently used for health policy decisions, whilst conditional treatment effects are helpful at an individual patient level. Further, even if conditional models from PS matching, IPW and IV techniques contain the same variables, unavoidable differences between analyses mean results are still not directly comparable. For example, PS matching is conditional on the covariates and the PS, whereas the other analyses are just conditional on the covariates.

Non-cardiac death is a competing risk for post-transplant MACE, as patients dying of non-cardiac causes cannot subsequently develop MACE. The analyses presented in the following section determine the “direct” effect of screening on MACE as patients are censored at non-cardiac death, as opposed to the “total” effect of screening on MACE which would include causal pathways involving non-cardiac death (61).

Interpreting direct treatment effects is challenging as they assume an unrealistic situation where competing events do not occur. Further, direct treatment effects have additional causal assumptions such as no unaccounted confounding of the relationship between the competing event (non-cardiac death) and outcome of interest (MACE). If there is likely to be a confounding relationship between the censoring event and the outcome of interest, techniques such as inverse probability of censoring weighting may be required to derive valid estimates of the direct treatment effect—such analyses require sufficient data availability for the probability of censoring (i.e., non cardiac death) to be modelled accurately over time (61).

As the purpose of this paper is to demonstrate the application of different causal inference techniques, for pragmatic reasons the following analyses represent the direct effect of screening on MACE. Information on competing risk analyses, which can navigate this issue by generating total treatment effects, are found at the following references (62, 63, 64).

To generate the PS, variables deemed to potentially relate to screening and MACE were determined and included in a logistic regression model. These comprised: age, sex, ethnicity, socioeconomic status, smoking status and history of ischaemic heart disease, diabetes, cerebrovascular disease, and peripheral vascular disease. Transplant centre was not included as it should not independently associate with MACE, would prevent us capitalising on variation in practice to create groups screened and unscreened patients, and could result in violation of the positivity assumption (Supplementary Table S1).

As the proportion of screened and non-screened individuals was roughly equal, PS matching was performed on a 1:1 basis without replacement using a caliper of 0.2 times the standard deviation of the logit of the propensity score. Matching was possible in 1760 individuals. The distribution of the PS before and after matching is shown in Supplementary Figure S2. The standardised mean difference after matching showed appropriate covariate balance between groups (Supplementary Table S2). The characteristics of screened and unscreened patients in PS matched and unmatched groups are shown in Figure 4. The 812 unmatched individuals were more likely to be male, of Asian ethnicity, and have a history of cardiovascular disease (Supplementary Table S3). In the PS matched population, screening did not reduce MACE at 90 days (conditional HR 0.80, 95% CI 0.31–2.05), 1 year (conditional HR 1.12, 95% CI 0.51–2.47) or 5 years (conditional HR 1.31, 95% CI 0.86–1.99) (Table 3). These results reflect the ATT: the causal effect of screening in screened patients eligible to receive either treatment (and thus “matched”), representing transplant recipients at low-medium cardiac risk.

For IPW, inverse probability of treatment weights were calculated. Weights were stabilised by multiplying them by the proportion of individuals who underwent screening in the exposed group, and proportion of individuals who did not undergo screening in the unexposed group (32). The mean of the stabilised weights was 1.00 (SD 0.47, range 0.53–8.45). Characteristics of the 57 patients with stabilised weights greater than or equal to 2 are in Supplementary Tables S4, S5. These patients were more frequently unscreened. Higher-weighted unscreened patients were older and more likely to have cardiovascular disease.

In total 2502 individuals were examined in the IPW analysis; 70 individuals were excluded due to missing data in variables used to generate the PS. Cox regression models were performed incorporating the IPW (Table 3). There was no evidence screening reduced MACE at 90 days (conditional HR 0.95, 95% CI 0.44–2.05) or 1 year (conditional HR 1.28, 95% CI 0.72–2.26). There was weak evidence that patients undergoing screening were at higher risk of MACE at 5 years (conditional HR 1.38, 95% CI 1.00–1.90), but this analysis did not meet the Cox proportionality assumption with a greater rise in MACE in screened patients over time. These results reflect the ATE: the causal effect of screening on the transplanted population. They do not provide information on the effect of screening on the total population who begin transplant workup.

It is important to note that these results represent a complete case analysis, as the 70 individuals with missing data were excluded. Complete case analyses assume data are missing completely at random, though other missing data mechanisms and their potential implications need to be considered (65).

Transplant centre is determined by geographical location so is largely randomly allocated. We determined centre had the potential to be an IV as it (at least partly) met the following assumptions (Figure 2B):

(1) Relevance assumption: the likelihood of undergoing screening is associated with transplant centre (Figure 4), even after adjustment for patient-level characteristics (18). On an individual patient level, screening is associated with older age, male sex, and a history of vascular disease (Supplementary Table S6) but when examining patients based on whether they are registered at a centre with a low, medium, or high screening use, differences in these variables is reduced (Table 4).

In the first stage, a linear regression containing potential confounders of the treatment-outcome relationship (deemed to be those used to create the PS) and transplant centre was used to predict the likelihood of an individual undergoing screening. Linear regression was selected for this analysis as opposed to logistic regression as described in IV literature (43), which also prevented individuals from centres who screened all recipients (n = 264) being dropped given instrument was a “perfect” predictor of outcome. Whilst using centre as an instrument addresses unmeasured patient-level confounding (i.e., unmeasured confounding between X and Y via U in Figure 2), centre-level confounding remains possible due to other institutional differences in practice (i.e., confounding between Z and Y in Figure 2 that may be distinct from U and/or C). (68) We considered including centre-specific variables which could influence outcome e.g. proportion of living donor or pre-emptive transplants, but these were not included in the final model due to collinearity with centre.

The first stage generated a predicted value, representing the likelihood of each individual being screened. The F statistic was 70 and the partial R-squared value was 0.33, indicating centre was a strong IV.

In the second stage, univariable and multivariable Cox regression models were performed using the predicted value from the first stage (predictor substitution method). This step can be considered as including the proportion of patients screened by centre as a patient characteristic, rather than whether each individual was screened. The multivariable model included the same confounders used to create the PS as these were deemed to potentially confound both the instrument-outcome and treatment-outcome relationship, and therefore including these confounders makes the independence assumption more likely to hold. Screening did not reduce MACE in the conditional model at 90 days (conditional HR 1.37, 95% CI 0.29–6.55), 1 year (conditional HR 1.85, 95% CI 0.65–5.29) or 5 years (conditional HR 1.21, 95% CI 0.72–2.02). These results reflect the LATE: the causal effect of screening on the ‘complier’ patients in the population.

Results from PS matched, IPW and IV analyses are shown in Table 3. In the conditional models, screening did not reduce MACE in any analysis, which each had overlapping confidence intervals, but there was variation in estimates between methods. The hazard ratios using PS methods rose over time, crossing 1 between 90 days and 1 year, whilst in the IV analysis the hazard ratio was above 1 throughout. These differences can help result interpretation by considering which patients are included in each analysis.

In the PS matched analysis, the results are only applicable to 1760 transplant recipients with low-medium baseline risk of MACE, not the overall population. The 812 individuals excluded from the analysis were more likely to be male, of Asian ethnicity, have a history of cardiovascular disease and be of a lower socioeconomic status and thus have the greatest baseline cardiovascular risk. Whilst these results suggest no benefit to screening, this cannot be directly applied to these highest risk patients.

The IPW analysis includes all patients and represents the whole transplanted population. Similar findings were observed to the PS matched analysis at 90 days and 1 year. At 5 years, there was weak evidence that individuals who had undergone screening were more likely to experience MACE in the conditional model but it should be noted that this analysis did not meet the Cox proportionality assumption.

In the IV analysis, screening did not reduce MACE on conditional analyses with a hazard ratio above 1 throughout, suggesting “complier” screened individuals had a higher risk of MACE than complier non-screened individuals, although confidence intervals were extremely wide. Given these results represent the LATE, it is not known whether the effect of screening on non-complier patients differs. Whilst the IV technique minimises unmeasured confounding, these results raise the possibility that unmeasured patient level characteristics associate with centre and outcome (i.e., clinicians screen their patients as they see their population as being inherently higher risk), or there are unmeasured differences in centre level practice, e.g., use of medical therapy that could bias results. Alternatively, it is possible that the PS matched and IPW analyses are prone to bias due to unmeasured confounding, and the IV analysis provides a result that is closer to the truth. Some studies suggest IV techniques provide less biased results than PS analyses, (69) but the challenges in identifying an appropriate instrument must be considered and results interpreted with caution until further studies examining both techniques are available (70).

The marginal hazard ratios presented in Table 3 reflect the effect of screening on the study population as opposed to an individual patient. In the PS matched and IPW analyses, screening did not reduce MACE. The results of the IV analysis differed, with screened individuals having a greater risk of MACE at 1 year (HR 4.18, 95% CI 1.79–9.76) and 5 years post-transplant (HR 3.19, 95% CI 2.09–4.87). This may reflect deviation from the independence assumption of no confounders to the instrument and outcome, the impact of which is lessened by adjusting for confounders in the conditional model.

Whilst the causal inference techniques applied to our worked example reduce confounding by indication, other forms of bias remain (Table 2). The worked example only examines patients who received a transplant. Data were not available for those who were screened and not listed due to an abnormal screening test, or listed but not transplanted due to MACE that occurred on the waitlist. Screening results are just one factor in a complex assessment of patients for transplantation, with the proportion of patients excluded due to cardiac screening abnormalities estimated at 1%–4% (71, 72, 73). In a target trial examining whether cardiac screening improves post-transplant outcomes these data would ideally be known, and neither PS or IV techniques specifically address this issue. Results therefore cannot be applied to the population who begin transplant workup nor determine the impact of screening on outcomes outwith post-transplant MACE.