A fundamental component of developing an evidentiary package that meets the requirements of regulatory endorsement for iBOX was the success of TTC’s extensive global patient-level data-sharing initiative [9–11]. Datasets from relevant clinical trials, including those used by [7] in their 2019 publication and real-world data from international clinical transplant centers, were prioritized for acquisition. A flow diagram of the dataset selection process is shown in Figure 2 with additional rationale provided in the Supplementary Material. The original iBOX [7] development included time posttransplant to account for varying iBOX assessments of individual patients and to assist in patient care and prognosis estimation. Figure 3 from [7] shows the density of iBOX risk evaluation time points after transplantation. The derivation dataset included in the EMA qualification submission represents all 4,000 subjects described in [7]. For application as a 1-year endpoint in a typical phase 3 clinical trial, we examined the number of subjects in the derivation dataset with iBOX assessments fixed at 1 year posttransplant and had outcome data of at least 5 years.

Five datasets supporting the regulatory endorsement of iBOX included data from clinical transplant centers (i.e., Loupy et al., 2019 derivation [7]. Mayo Clinic Rochester, and Helsinki University Hospital) and clinical trials (i.e., BENEFIT randomized controlled trial [RCT] [12] and BENEFIT-EXT RCT [13]) representing over 2,500 de novo kidney transplant recipients with 1-year iBOX assessments (Table 3). Participant consent was obtained from the transplant centers and clinical trials for primary uses. These datasets contained all elements necessary to assess the performance of iBOX as a pivotal trial endpoint including IST information, iBOX variables at 1 year posttransplant, and 5 years follow-up for death and graft loss of at least 5 years. Additionally, these datasets were accompanied by assay information for each component iBOX measure and laboratory certification documentation ensuring that the analytical methods were robust, reliable, and fit-for-purpose.

Clinical transplant center data are inherently heterogeneous and reflect the diversity of the kidney transplant recipient population in the United States and European Union. Datasets were curated, standardized, and aligned to conduct internal and external validation analyses to support the iBOX COU with EMA. Two clinical trial datasets have the most extensive CNI-free (belatacept [BELA]) patient-level data with the 4 core iBOX variables and sufficient follow-up period available. This represents, as stated by EMA, “extensive global effort to collect clinical trials and real-world data” [10].

Critical Path Institute explored the number of transplant recipients with full and abbreviated iBOX assessments at varying times posttransplant in the already curated and aligned validation datasets. Supplementary Figure S1 shows the distribution of assessment time points for donor-specific antibody (DSA) measurements up to 2 years postkidney transplant. DSA was selected for illustration because it is collected less frequently than eGFR and/or proteinuria and therefore acts as the key limiting factor for the availability of abbreviated iBOX measurements. The data distribution for iBOX assessments ranges from 6 months up to 2 years posttransplant. Helsinki University Hospital only assessed proteinuria and DSA data at 1 year posttransplant and therefore was excluded from the additional time points exploration. The number of transplant recipients with iBOX assessments at 6 months and 2 years posttransplant in the external validation datasets is shown in Supplementary Table S2. There were significantly more abbreviated iBOX assessments at the varying time points because biopsies were more typically “for-cause” and not taken “per protocol” at 6 months or 2 years posttransplant. Although the full iBOX measurements at 2 years were limited due to lack of biopsy information, because the abbreviated iBOX performed well at this time point, the addition of biopsy information should only further improve the performance, and therefore, the full iBOX is expected to also perform well at 2 years.

Validation analyses were performed to support the COU for iBOX with EMA for predicting death-censored graft loss. Both internal validations, evaluating iBOX on the data it was trained on (i.e., the derivation dataset), and external validation, evaluating iBOX on data it was not trained on, were performed. The abbreviated iBOX was treated as a modification of the full iBOX and not validated internally, save for checking the overall c-statistic. Both the full and abbreviated iBOX models were validated on 4 external datasets (i.e., validation datasets) (previously described above).

To avoid survivor bias, patients who did not reach their scheduled evaluation (i.e., those who lost their graft, died, or were lost to follow-up beforehand) were given an imputed worst-case iBOX score (Supplementary Tables S3, S4).

iBOX was validated by assessing its discrimination, the ability to rank individuals from a lower to a higher risk of graft loss, and its calibration, the ability to accurately predict absolute risk level [14]. Discrimination was assessed using Harrell’s c-statistic [15], which gives the probability that, for any 2 randomly selected individuals, the individual with the higher iBOX score, i.e., the higher model-predicted hazard of graft loss, has a shorter death-censored graft survival time. A c-statistic value of 0.7 or greater indicates good discriminatory ability [16]. Secondly, calibration was evaluated by checking whether observed events (graft losses) matched predicted using a Poisson calibration method (see Supplementary Material for a summary of the method) [14].

The full iBOX discrimination in the derivation dataset, when restricted to transplant recipients with an iBOX score at 1-year posttransplant and follow-up to 5 years, had a c-statistic of 0.85, demonstrating iBOX discriminates appropriately among subjects for use in a phase 3 study (Table 3). In the validation datasets, c-statistics ranged from 0.70 to 0.93 (Table 3), and the predicted versus observed graft losses were not significantly different for iBOX assessments at 1 year posttransplant (Table 4).

Given that the iBOX models are trained primarily on subjects receiving CNI-based maintenance ISTs, it was unclear if iBOX would perform similarly in kidney transplant recipients not on CNI-based therapies. Internally, the iBOX was found to discriminate appropriately between higher- and lower-risk patients receiving mTOR inhibitor-based therapies (c-statistic >0.8) (Table 5). Externally, 5 years iBOX c-statistic values for CNI-free subjects, consisting primarily of patients on BELA-based regimens, at 1 year posttransplant in the validation datasets were evaluated; full and abbreviated iBOX c-statistics were 0.75 and 0.73, respectively (Table 6). These analyses demonstrate that iBOX can discriminate between subjects at higher and lower risk of death-censored graft loss in diverse datasets, including CNI and CNI-free populations, in clinical transplant centers and RCTs. Likewise, the results also showed that iBOX has good prediction accuracy based on calibration analyses (Table 6).

The performance of the full and abbreviated iBOX were also tested in the validation datasets at 6 months and 2 years posttransplant. The 5 years posttransplant discrimination (Supplementary Table S5) and calibration analyses (Supplementary Table S6) support the inclusion of time posttransplant in the iBOX model at 6 months and 2 years posttransplant.

Based on the iBOX formulas shown in Table 7, iBOX is not just the sum of the parts (i.e., the addition of components) but includes continuous and dichotomous variables weighted differently based on the beta coefficients. The c-statistic for eGFR alone and eGFR with proteinuria in comparison with the full and abbreviated iBOX is shown in Table 8, with calibration results in Supplementary Tables S7, S8, indicating that the iBOX score is influenced most by eGFR, and the other 3 components, proteinuria, anti-human leukocyte antigen DSA, and biopsy, all increase the predictive power.

In addition to validation, an analysis of the BENEFIT and BENEFIT-EXT RCTs included imputation of the worst-case iBOX scores at 1 year posttransplant for recipients who died or lost their graft in the first year (Table 9). This sensitivity analysis was performed to replicate the clinical trial setting where avoidance of survivor bias at 1 year would be necessary, and all randomized subjects would have an iBOX score at 1 year even if there were death or graft loss before that time. In both studies, the full and abbreviated iBOX score at 1 year was significantly lower in the BELA group than in cyclosporine, indicating a lower predicted risk of long-term graft failure. This corresponded to a statistically significant improvement in 5 years graft survival in the BENEFIT study. The BENEFIT-EXT study showed directionally higher 5 years death-censored graft survival. However, the difference was not statistically significant. The larger treatment difference in iBOX score at 1 year in the BENEFIT study compared with BENEFIT-EXT also corresponded to a larger treatment difference in graft survival. The lack of statistical significance on some of the 5 years graft survival analyses is related to limitations in the power to detect differences based on sample size.

Additional analyses were performed testing the performance of the full iBOX at 1 years posttransplant on all-cause 5 years graft loss (Supplementary Tables S9, S10). The discriminatory ability of iBOX for all-cause graft loss underperforms, with the full iBOX having reduced c-statistics, many of which are below 0.7, and poor all-cause calibration. This is expected given that iBOX was originally developed using variables more likely to impact risk of graft loss. Based on this evidence, iBOX was qualified with EMA with death-censored graft loss as the outcome measure.