Skip to main content
  • Research article
  • Open access
  • Published:

Development and validation of a prognostic model to predict the prognosis of patients who underwent chemotherapy and resection of pancreatic adenocarcinoma: a large international population-based cohort study

Abstract

Background

Pancreatic cancer (PaC) remains extremely lethal worldwide even after resection. PaC resection rates are low, making prognostic studies in resected PaC difficult. This large international population-based study aimed at exploring factors associated with survival in patients with resected TNM stage I–II PaC receiving chemotherapy and at developing and internationally validating a survival-predicting model.

Methods

Data of stage I–II PaC patients resected and receiving chemotherapy in 2003–2014 were obtained from the national cancer registries of Belgium, the Netherlands, Slovenia, and Norway, and the US Surveillance, Epidemiology, and End Results (SEER)-18 Program. Multivariable Cox proportional hazards models were constructed to investigate the associations of patient and tumor characteristics with overall survival, and analysis was performed in each country respectively without pooling. Prognostic factors remaining after backward selection in SEER-18 were used to build a nomogram, which was subjected to bootstrap internal validation and external validation using the European datasets.

Results

A total of 11,837 resected PaC patients were analyzed, with median survival time of 18–23 months and 3-year survival rates of 21–31%. In the main analysis, patient age, tumor T stage, N stage, and differentiation were associated with survival across most countries, with country-specific association patterns and strengths. However, tumor location was mostly not significantly associated with survival. Resection margin, hospital type, tumor size, positive and harvested lymph node number, lymph node ratio, and comorbidity number were associated with survival in certain countries where the information was available. A median survival time- and 1-, 2-, 3-, and 5-year survival probability-predictive nomogram incorporating the backward-selected variables in the main analysis was established. It fits each European national cohort similarly well. Calibration curves showed very good agreement between nomogram-prediction and actual observation. The concordance index of the nomogram (0.60) was significantly higher than that of the T and N stage-based model (0.56) for predicting survival.

Conclusions

In these large international population-based cohorts, patients with resected PaC receiving chemotherapy have distinct characteristics independently associated with survival, with country-specific patterns and strengths. A robust benchmark population-based survival-predicting model is established and internationally validated. Like previous models predicting survival in resected PaC, our nomogram performs modestly.

Peer Review reports

Background

Worldwide, pancreatic cancer (PaC) is the seventh leading cause of cancer-related mortality [1]. Resection remains the cornerstone of curative treatment for medically fit patients with resectable locoregional PaC [2]. However, only a small proportion of patients with PaC undergo resection. In the United States (US) and Europe during 2012–2014, resection rates ranged from 13% (Estonia) to 21% (Slovenia) for all-stage PaC and from 35% (Norway) to 69% (Denmark) for stage I–II PaC [3]. The 3-year overall survival remains poor even for patients with stages I–II PaC who underwent resection (< 60 years, 23–39%; 60–69 years, 16–31%; and ≥ 70 years, 17–30%) [4]. Chemotherapy has been routinely recommended for resected PaC [5,6,7]; however, it remains challenging to get many patients to adjuvant therapy after pancreatectomy [8].

Various prognostic factors for PaC including clinical/pathological (e.g., tumor stage, size, differentiation, lymph node status, resection margin, and peri-neural and blood vessel invasion [9,10,11,12,13,14,15]), genetic (e.g., KRAS, TP53, SMAD4, and some DNA damage repair genes (e.g., BRCA1/2 and MLH1) [16,17,18]), and immunological variables (e.g., CD3, CD8, CD68, PD-L1, and HHLA2 [19,20,21]) have been reported. Further large international comparative studies on survival-associated factors at the population level could help to identify differences across countries.

Patients with resected PaC who undergo chemotherapy are a selected group of all PaC patients and have distinct characteristics [8]. Even within this patient group, survival is heterogeneous. A prognostic model for this specific patient population is important and desirable and could facilitate clinical counseling by informing both patients and doctors of predicted individualized patient survival, guide plans on follow-up and surveillance, aid to survival stratification in international studies, and offer the baseline survival estimates for further molecular or genetic investigations. Furthermore, for resected patients considering subsequent chemotherapy, the predicted results could potentially encourage a proportion of patients with specific characteristics to further receive the standard postsurgical care. Stage is the major prognostic factor for PaC. Notably, survival of patients with disease of the same TNM stage might vary greatly [14]. Other prognostic factors such as patient age and tumor differentiation could improve individualized survival-prediction. A model incorporating all these factors can be intuitively illustrated using a nomogram [22]. Apart from two institutional nomograms predicting postsurgical survival in overall patients [23, 24], population-based survival-predicting models specifically for resected PaC patients receiving chemotherapy with international validations and robustness have not been found.

To our knowledge, we herein report the first large international population-based investigation into factors associated with survival in patients with resected TNM stage I–II PaC receiving chemotherapy. We further construct a population-based survival-predicting model with international validations.

Methods

Patients

Population-based data of resected PaC patients were obtained from the national cancer registries of Belgium, the Netherlands, Slovenia, and Norway, and the US Surveillance, Epidemiology, and End Results (SEER)-18 [25] database. Data quality was previously described [3]. Institution-based data were not included due to the highly selected patients. An extensive attempt was made to contact population-based cancer registries, and the contacted registries together with reasons for exclusion are shown in Additional file 1: Table S1. The participating European national registries, located in Western, Southern, and Northern Europe were those able to provide quality data according to a standardized uniformed data-request form, which ensured the robustness of the results. All variables were uniformly (re)coded across registries. While there were other national population-based registries, they were not always able to provide eligible treatment, TNM staging, or survival data. All patient-level data were anonymous. This real-world observational study was approved by the Ethics Committee of Medical Faculty Heidelberg.

Patients with diagnosis based on death certificate only (DCO)/autopsy or with unknown/obscure follow-up time or vital status were excluded (Additional file 1: Table S2). Only patients with microscopically confirmed diagnoses of primary invasive TNM stage I–II adenocarcinomas of the exocrine pancreas who underwent surgical resection in 2003 until 2014 were selected. The time period was selected based on data availability and the fact that the fifth and prior editions of TNM staging were incompatible with the sixth/seventh versions used during 2003–2017 [7]. Since chemotherapy is standard for patients with resected PaC [5,6,7], we only included those receiving chemotherapy. Individuals with benign/premalignant tumors, non-PaC neoplasms involving the pancreas, neuroendocrine tumors/carcinoids, cystic/mucinous/serous tumors, acinar cell tumors, stromal tumors, sarcomas, germ-cell neoplasms, lymphomas, or peri-ampullar tumors were also excluded (Additional file 1: Table S3). To minimize the effect of the potential heterogeneity in surgery quality and perioperative care, we excluded cases surviving < 3 months. Patients with stage III or IV PaC were also excluded since resection is not routinely recommended for these patients [5,6,7].

Information on demographic (sex and age), clinical (year of diagnosis/surgery and treatment), and pathologic characteristics (topology, morphology, and TNM stage) was retrieved from all participating countries. Data on resection margin (the Netherlands and Slovenia), hospital type (Belgium and the Netherlands), Eastern Cooperative Oncology Group (ECOG) performance status score (Belgium), comorbidities (Eindhoven, the Netherlands), resection type (the US and the Netherlands), tumor size (the US), and positive and harvested lymph node numbers (the US and the Netherlands) were only available in certain registries.

Resection was defined as surgical removal of primary tumor, regardless of being curative or palliative and extents of excision and lymphadenectomy. Tumor topography and morphology were based on the International Classification of Diseases for Oncology (third edition). Stage was defined following the TNM staging system (sixth/seventh edition) and was a combination of pathologic and clinical stages with priority given to pathologic staging. Lymph node ratio was calculated by dividing the number of positive lymph nodes by the number of harvested lymph nodes. Vital status was based on valid national mortality registrations and official population registers.

Statistical analyses

Data in each country were analyzed separately without pooling, considering the potential heterogeneity across countries and to avoid the impact of any single large cohort. Descriptive results were reported as the smallest to the largest proportions for categorical variables or medians/means for continuous variables across countries. The cancer incidence rates by sex in each country were retrieved from the Cancer Incidence in Five Continents Volume XI (CI5 XI) by the International Agency for Research on Cancer (IARC), World Health Organization (WHO) (http://ci5.iarc.fr/CI5-XI/Default.aspx), which reports the incidence of cancers diagnosed from 2008 to 2012, standardized to the World (WHO 2000–2025) Standard Population.

The Kaplan-Meier method was applied to calculate survival time and rates. Since patients surviving < 3 months were excluded in this study, the 6- and 9-month survival was calculated as the short-term outcome. The 1-, 2-, 3-, and 5-year survival was computed as the long-term outcome. To assess the independent impact of potential prognostic factors on survival, Cox proportional hazards regression was used. Variables including year of diagnosis, age, sex, tumor location, T and N stages, and differentiation were included as covariates in the main multivariable models. For complete-case analysis, patients with missing data were excluded in multivariable analyses. In the US, results for the white patients were computed for comparison with the total patients, for whom main analyses were performed. In registries with available information, resection margin, hospital type, tumor size, positive and harvested lymph node numbers, lymph node ratio, T and N stages according to the eighth edition following Kamarajah et al. [26], ECOG score, resection type, and comorbidities were incorporated one by one into the main models to examine the survival association for each of them. The proportional hazards assumption was verified for all variables by plotting the logarithm of the negative logarithm of the survival function against the logarithm of survival time [27].

Data were centrally analyzed in the German Cancer Research Center. Results were considered statistically significant at two-sided P < 0.05. Analyses were conducted using the SAS software (version 9.4, SAS Institute Inc.).

Nomogram construction and validation

The SEER-18 dataset, the largest of the included datasets, was used as the training set for nomogram construction (models based on the other cohorts did not reveal markedly better performance). Age, sex, tumor location, T and N stages, and differentiation were entered as potentially relevant prognostic factors into the initial full multivariable Cox proportional hazards regression model, and the final model was selected through a backward step-down process using the likelihood ratio test with the Akaike information criterion as a stopping rule [28]. To permit nonlinear associations, continuous variables were modeled using restricted cubic splines where appropriate [28]. Points assigned to each variable included in the nomogram to predict the median survival time and 1-, 2-, 3-, and 5-year survival probability were proportional to the effect size of that variable in the final multivariable model. To facilitate clinical use, a corresponding online prognostic tool was created with Evidencio (https://www.evidencio.com/).

The nomogram was subjected to 1000 bootstrap resamples for internal validation of the training US cohort and was externally validated using the European datasets to assess the international generalizability of the model. The model performance and discrimination ability for predicting survival was numerically evaluated by computing Harrell’s concordance index (C-index) [28]. Comparison of C-indexes of different models followed Hanley et al. [29]. Calibration of the nomogram for 1-, 2-, 3-, and 5-year survival was done by comparing the predicted with the observed survival. Bootstrapping was used for bias correction [28].

In sensitivity analyses for the training US cohort, C-indexes were re-calculated after replacing continuous age with age group, N stage with positive lymph node number or lymph node ratio, and sixth/seventh edition of cancer stages with the eighth version, after adding harvested lymph node number and/or tumor size, after limiting patients to those diagnosed after 2009 or white patients, and after stratifying patients by tumor location. The survival and rms packages in R 3.4.1 (http://www.r-project.org) were used.

Results

Patient characteristics

In total, 168,949 PaC patients were registered in the population-based registries in 2003/2004–2013/2014 with follow-up until 2015–2016. After excluding patients diagnosed based on DCO/autopsy (n = 4403), unresected (n = 137,605), receiving no/unknown chemotherapy (n = 11,465), with microscopically unconfirmed tumors or with tumors of ineligible pathology (n = 1418), with stage 0/III/IV/unknown tumors (n = 1856), and with survival< 3 months or unknown (n = 365), 11,837 patients were eligible for analysis (Additional file 1: Table S2). The detailed counts and frequencies for discrete variables and medians and interquartile ranges for continuous variables are shown in Table 1. Age-standardized PaC incidence was higher for males than for females. Among the participating countries, incidence for males was lowest in Belgium and the Netherlands (7.4 per 100,000) and highest in Slovenia (9.3 per 100,000); for females, incidence was lowest in Belgium (5.6 per 100,000) and highest in the US, Slovenia, and Norway (6.5 per 100,000).

Table 1 Demographic and clinical characteristics of resected pancreatic cancer patients1

Of the analyzed patients, 52–76% were diagnosed in 2010 or later. The proportion of women ranged from 42% (Slovenia) to 51% (Norway) across countries. While the proportion of women was almost identical to that of men in the US, the Netherlands, and Norway, there was a smaller proportion of women in Slovenia (42%). The median age ranged from 61 (Slovenia) to 65 years (the US and Belgium) across countries. Most patients were ≥ 60 years (58% (Slovenia) to 71% (Norway)). The proportion of patients aged ≥ 70 years was greatest in the US (34%) and smallest in Slovenia (19%). Only 4% (Norway) to 9% (the Netherlands and Slovenia) of patients were < 50 years old. Tumors were most commonly located at pancreas head (82% (Belgium) to 92% (Slovenia)). Only 3% (the Netherlands) to 7% of cancers (the US and Belgium) were located at pancreas body and 3% (Slovenia) to 11% (Belgium) at pancreas tail. Only a minority of patients had T1 (0% (Slovenia) to 8% (Norway)) or T2 cancers (7% (Slovenia) to 26% (Norway)). N1 tumors comprised 55% (Norway) to 84% of all cancers (Slovenia). Most patients had either moderately differentiated (40% (Slovenia) to 63% (Norway)) or poorly differentiated/undifferentiated tumors (33% (Belgium) to 50% (Slovenia)). Only 3% (Norway) to 15% of cancers (Belgium) were well-differentiated.

Survival outcomes

The median survival time ranged from 18 (Slovenia) to 23 months (the US) across countries (Fig. 1). The short- and long-term survival outcomes are shown in Table 2. The 6-month survival rate ranged from 94% (the US) to 97% (the Netherlands), and the 9-month survival rate varied from 79% (Slovenia) to 90% (Norway). Regarding longer term outcomes, the 1-year survival rate ranged from 69% (Slovenia) to 79% (the Netherlands), and the 3-year survival rate ranged from 21% (Slovenia) to 31% (the US). The 5-year survival rate was lowest in Slovenia (10%), which was about half of that in the US (19%), the Netherlands (20%), or Norway (21%).

Fig. 1
figure 1

Kaplan-Meier overall survival curves for patients with resected stage I–II pancreatic cancer receiving chemotherapy in each country. The 95% confidence limits curves and the 95% Hall-Wellner bands are additionally shown. Median survival time (interquartile range) in months and 3-year survival rates (95% confidence interval) are calculated and provided. IQR, interquartile range; CI, confidence interval

Table 2 Short- and long-term survival for resected pancreatic cancer patients receiving chemotherapy estimated by the Kaplan-Meier method

Survival-associated factors

Results from multivariable Cox regression are shown in Table 3, and only significant results are described. Increasing age was associated with worse survival in the US (HR per year = 1.01), Belgium (HR = 1.02), and Norway (HR = 1.04). Survival was significantly worse in men only in the US (HR = 1.10) and in pancreas body compared to head tumors in Norway (HR = 2.67). Compared to T3 cancers, T1 cancers were associated with higher survival in all investigated countries (HR = 0.17–0.70), while T2 cancers were associated with better survival only in the US (HR = 0.86). Negative nodal status was associated with significantly higher survival in the US (HR = 0.65), Belgium (HR = 0.78), and the Netherlands (HR = 0.51). Better differentiation was significantly associated with higher survival in all countries except Slovenia and Norway, and the HRs for well- and intermediately versus poorly/undifferentiated tumors were 0.48–0.68 and 0.61–0.81, respectively. Association patterns and strengths were similar between white and overall US patients.

Table 3 Association of demographic and clinical variables with overall survival for resected pancreatic cancer patients estimated by adjusted multivariable Cox proportional hazards regression

Associations with further variables were explored in countries with available relevant information (Table 4). In the Netherlands, positive resection margin was associated with worse survival in (HR = 1.36), and resection in academic hospital was associated with better survival (HR = 0.79). In the US, larger tumor size was associated with inferior survival, and replacing T stage according to the sixth/seventh edition with the eighth edition revealed similar association patterns and strengths. In the US and the Netherlands, while increasing metastatic node number (HR per positive lymph node = 1.05 and 1.07) and lymph node ratio (HR = 2.60 and 3.15) were associated with inferior survival, more harvested nodes suggested better survival (both HR per harvested node = 0.99). Following the eighth version of TNM staging, N1 (HR = 1.42 and 1.68) and N2 stages (HR = 1.84 and 2.43) were associated with worse survival compared to N0 stage in the US and the Netherlands. In Eindhoven, the Netherlands, more comorbidities were associated with inferior survival (e.g., HR≥ 2 vs. 0 comorbidities = 1.86).

Table 4 Association of survival with potential prognostic factors available in at least one registry for resected pancreatic cancer estimated by adjusted Cox proportional hazard regression

Prognostic nomogram

Construction

A nomogram incorporating prognostic factors remaining after backward selection in the US (sex, age, T and N stages, and differentiation) was established (Fig. 2a). The nomogram illustrated age and differentiation to have the largest contributions to prognosis. T and N stages showed moderate impacts on survival. Each number/category of these variables is assigned a score on the Points scale. After summing up the total score and locating it on the Total Points scale, a line drawn straight down to the Median Survival or 1-/2-/3-/5-Year Survival Probability scale shows the estimated survival time or probability at each time point. The model function is provided in the Additional file 1: Supplementary Results. Score assignment for specific categories of the variables and survival for different accumulated scores are shown in Table 5. The layout of an online version of the nomogram is shown in Fig. 3.

Fig. 2
figure 2

Prognostic nomogram for patients with resected stage I–II pancreatic cancer receiving chemotherapy derived from the US cohort (a) and an example on how to use the nomogram (b). Each number/category of the prognostic variables is assigned a score on the Points scale. After summing up the total score and locating it on the Total Points scale, a line drawn straight down to the Median Survival or 1-/2-/3-/5-Year Survival scale shows the median survival time and estimated survival probability at each time point. Age is in years

Table 5 Score assignment for specific categories of the variables included in the nomogram
Fig. 3
figure 3

Layout of an online version of the developed nomogram with Evidencio (https://www.evidencio.com/models/show/1258)

An example of use

An example of how to use the nomogram is shown in Fig. 2b. A 72-year-old woman with poorly differentiated, T2N1M0 PaC who underwent resection and chemotherapy would have 28 points for her age, 0 points for her sex, 53 points for T stage, 82 points for N stage, and 97 points for differentiation, totaling 260 points. The total points correspond to the estimation of median survival time of < 20 months, a 1-year survival probability of 72%, a 2-year survival probability of 38%, a 3-year survival probability of 22%, and a 5-year survival probability of 12%, which are consistent with the results generated by the online tool (Fig. 3).

Calibration and validation

The nomogram was applied to the US and the European countries for internal and external validations, respectively. The calibration plots showed very good agreement between nomogram-predicted and actual survival in the US, Belgium, and the Netherlands (Fig. 4; plots were not shown in Slovenia or Norway where the case number was too small to generate meaningful calibration). Generally, the calibration was best for 2- and 3-year survival. In the training US cohort, the C-index for the established nomogram was significantly higher than that for the model based on both T and N stages (0.60, 95% CI = 0.59–0.61 vs. 0.56, 95% CI = 0.56–0.57). In the validation cohorts, C-indexes were also significantly higher for the nomogram than for the T and N stage-based model (Table 6).

Fig. 4
figure 4

Calibration curves for 1-, 2-, 3-, and 5-year overall survival prediction in the primary training (the US) and validation cohorts (Belgium, the Netherlands, Slovenia, and Norway). Nomogram-predicted survival is plotted on the x axis and actual survival on the y axis. The vertical bars at the top represent the frequency of the predicted probability of survival. A plot along the 45-degree line indicates a perfect calibration model where the predicted probabilities are identical to the actual proportions

Table 6 Concordance indexes for resected pancreatic cancer in training and validation cohorts and in sensitivity analyses for the training US cohort

Sensitivity analyses

Sensitivity analyses were performed for the derivative US cohort (Table 6). Using positive lymph node number or lymph node ratio instead of N stage in the nomogram did not obviously change the C-index (by 0.00 and + 0.01, respectively). Replacing the sixth/seventh version of both T and N stages with the eighth version also had minimal impact on the C-index (by + 0.01). After including examined lymph node number, tumor size, or both, the C-index only changed by 0.0, + 0.01, and + 0.01, respectively. Limiting the sample to patients diagnosed after 2009 or white people did not change the C-index. Within subgroups according to tumor location, C-index was slightly higher than the overall one in body/tail cancer (0.61).

Discussion

In our large population-based study, we identified various factors independently associated with survival after resection of PaC and for the first time established and internationally validated a population-based nomogram for predicting survival in resected PaC patients receiving chemotherapy, which is robust, accurate, reliable, and practical. However, like previous models [23, 24, 30,31,32], our model had a modest C-statistic.

There are various reports on the prognostic factors for patients who underwent resection for PaC [9,10,11,12,13,14,15]. A systematic review showed that with the exception of postsurgical blood transfusion, tumor characteristics (e.g., size, lymph node status, and differentiation) were the only features significantly associated with survival after pancreatic resection [9]. Particularly, PaC size > 2 cm was an independent factor associated with poor post-surgical prognosis [10], and this category has been incorporated in both the sixth/seventh and the eighth TNM staging systems [33, 34]. Notably, neural invasion was also determined to be an independent prognostic factor in PaC [11]. Through multivariable analyses, we demonstrated that older age, more advanced T and N stages, and poorer differentiation were independently associated with lower overall survival in resected PaC across most countries. In registries with available information, resection margin, hospital type, tumor size, metastatic and harvested lymph node numbers, lymph node ratio, and comorbidity number were also associated with prognosis. These findings are mostly consistent with previous literature [9,10,11,12,13,14,15, 35, 36] and add insights into the association strengths for resected PaC patients receiving chemotherapy at the population level and into the comparisons between countries. Some patient (e.g., age and comorbidities) and clinical characteristics (e.g., hospital type) were further identified to be prognostically significant. While previous studies have drawn differing conclusions regarding the association between resection type and survival [35, 36], our population-based investigation of chemotherapy-treated resected PaC patients did not show a significant association. Furthermore, we found mostly no significant associations between tumor location and survival.

Notably, overall, the contribution of T or N stage to postoperative survival was mostly not greater than that of differentiation. Categorization of tumor size and number of metastatic lymph nodes following the eighth TNM staging system [33, 34] discriminated survival well, supporting the implementation of the new system. Notably, harvested lymph node number was positively associated with survival. The relevance of harvested lymph node number for survival has remained controversial in PaC [37, 38]. Possible reasons supporting the positive association include that more metastasized lymph nodes may be removed with more extensive sampling, which also results in more precise staging, guiding appropriate post-surgical treatment.

Estimating mortality risk might impact treatment planning and provide information helpful for patient stratification in study design, contributing to better equivalence between study arms [39]. Post-surgical survival for patients with PaC is remarkably heterogeneous, even with the same TNM stage [14, 40, 41]. To our knowledge, the nomogram we developed is the first one derived from a large population-based database with long-term follow-up for predicting overall survival in patients with resected stage I–II PaC receiving chemotherapy, with international validations in multiple European national datasets. There is a previous institutional nomogram [23] developed by Memorial Sloan-Kettering Cancer Center (MSKCC) in 2004 for predicting post-surgical survival in Western PaC patients not accounting for chemotherapy, with three external institutional validation attempts [30,31,32]. Based on institutional patient cohorts diagnosed many years ago [23, 30,31,32], the score assignment of several variables might not be optimal currently using the MSKCC nomogram, which might also be limited in generalizability. The MSKCC nomogram did not employ a backward selection process and incorporated some detailed surgical (e.g., portal vein resection and splenectomy) and symptom parameters (back pain and weight loss). Notably, portal vein resection and splenectomy might not be routine procedures during pancreatectomy, and reporting of symptoms might show great interpersonal variations. Our population-based nomogram thus represents a more updated prognostic model compared to the MSKCC nomogram (Additional file 1: Table S5). The wide geographical distribution of patients and large sample size further enhanced the international representativeness and generalizability of our nomogram.

Resection margin, which reflects the radicality of surgery, has not received a universal standard definition in PaC [42, 43], and its relevance for survival remains highly controversial [44, 45]. While we showed a positive association of survival with negative margin in the Netherlands, the strength was not greater than that of the association with T stage, N stage, or differentiation. We did not incorporate this variable in our nomogram for better generalizability. It is encouraged to incorporate margin status into our nomogram when a standard definition comes.

Calibration plots demonstrated very good agreement between nomogram-predicted and actual survival, which assures the repeatability and reliability of our nomogram. Importantly, the model based on the US dataset also fits the multiple European national cohorts, which supports the potential for the generalization and international utilization of our nomogram, irrespective of the potential health care disparity across countries. Discrimination of the nomogram, as highlighted by the C-index, was significantly and markedly higher compared to the model based on T and N stages only. In the external validation cohorts, the discriminative potency only slightly changed. Our model performed similarly well across countries, potentially facilitating patient allocation in international studies.

In sensitivity analyses, we examined various alternative models by for instance incorporating positive lymph node number or lymph node ratio as a continuous variable in place of N stage into the nomogram, and the discrimination ability basically remained the same, supporting the robustness of our model.

Notably, the eighth edition of TNM staging system has been implemented since 2018 [33, 34]. Compared to the sixth/seventh version, in the eighth version new categories of tumor size (≤ 2, 2–4, and > 4 vs. ≤ 2 and > 2 cm) and positive node number (0, 1–3, and ≥ 4 vs. 0 and ≥ 1) are incorporated into T and N staging, respectively [26, 33, 34]. However, after integrating these factors either as continuous or corresponding categorical variables into our nomogram, the performance did not markedly change. After transforming the SEER-18 staging data according to the eighth edition following Kamarajah et al. [26], the performance also remained very similar. Moreover, it will take considerable follow-up time for the survival associated with the new staging system to be adequately assessed. Therefore, our nomogram will still be applicable without compromised accuracy in the coming years.

Strengths of our study include the international population-based design, the largest number of patients with resected PaC ever investigated, the extensive potential prognostic factors studied, the uniformly and consistently defined variables especially TNM stage across countries, and the consistency and quality control in reporting through applying rigorous registry data standards. Analyses were performed separately in each respective country without pooling, which avoids the impact of the potential heterogeneity across countries.

Our work may have important clinical impacts and provides to our knowledge the first population-based model which can predict survival for patients with stage I–II PaC who underwent resection and chemotherapy. The model is robust, accurate, well-generalizable, practical, and easy-to-use. Our model may offer personalized patient survival estimates and facilitate clinical counseling for both patients and doctors. Having an idea about the estimated survival of a specific patient could influence plans on follow-up and surveillance (e.g., frequency and examination modality) and thus possibly guide resource allocation. For some proportion of the resected patients considering further treatment, the predicted survival might encourage receipt of further chemotherapy. The international validation assures that our model could be used for survival stratification in international studies.

Patients with resected PaC do not respond equally to chemotherapy, and accordingly, the calibration plots also suggest that individual survival varied greatly despite the relatively consistent comprehensive survival across countries. Our study will help to initially stratify this patient population into subgroups with discrepant survival, and might serve as a platform for developing further endeavors to understand factors associated with chemotherapy responses and survival in resected PaC, including precise, individualized, and personalized genomic and proteomic survivorship investigations.

Like any observational registry-based investigation, our study also has some limitations. Our model predicts survival at the average population level, and when applying this model in specific centers or regions with different care patterns, there could be some inconsistencies between predicted and actual survival. Nevertheless, as revealed by the calibration plots, the real-world survival was still in good accordance with the prediction for a single individual. Residual confounding is a concern. Some significant variables (e.g., tumor size) were only registered in certain databases. Differences in survival pattern across countries might be partly associated with variation in the prescription of chemotherapy and/or the underlying ethnic/racial distribution, even though association results remained similar after limiting the US cohort to white. Notably, there were some differences in patient and tumor characteristics across registries. For instance, in Slovenia, tumors were generally more advanced and poorly differentiated, and the actual survival was the lowest. Nevertheless, these variables were adjusted for in our multivariable analyses.

Population-based registries collected limited information on variables including family and patient health history and individual-level socioeconomic status. In addition, we were unable to determine the molecular or genetic subtype of PaC [16], which probably plays a role in prognosis and explains the moderate C-index of our nomogram. Accordingly, our nomogram is limited by failure to incorporate these and other recognized prognostic parameters (e.g., neurovascular invasion and type of chemotherapy). Further efforts on collection and incorporation of more relevant variables are encouraged to improve this model.

Notably, all known models predicting PaC survival perform very modestly [23, 24, 30,31,32]. Our nomogram with selection of only chemotherapy-treated resected PaC patients does not perform better compared to previous models with selection of all patients undergoing resection [23, 24, 30,31,32], which might limit the added value of the selection for the current nomogram. The lack of detailed information on chemotherapy which has not been routinely collected in most registries is another limitation of this population-based registry-based study. Collection of such information is strongly encouraged in future registration practice. During the study period, the type of chemotherapy was mainly gemcitabine monotherapy, followed by 5-fluorouracil-based therapy. The ESPAC-3 [46] and RTOG 97-04 randomized trials [47] demonstrated similar efficacy and effectiveness regarding survival between gemcitabine and 5-fluorouracil in the adjuvant setting. The landscape of systemic treatment (e.g., agent and formula) and treatment sequence for PaC are rapidly changing, which might limit the possible use of this nomogram.

Despite the moderate C-index, the agreement between predicted and actual survival was very good. All variables included in our nomogram are easily available in clinics, compared to the not routinely measured and costly molecular markers. It is herein the first time that the contributions of these risk factors are quantified and integrated into a single model for survival prediction in resected and chemotherapy-treated PaC with international validations.

Conclusions

This large international population-based investigation revealed independent factors associated and not associated with survival in patients with resected stage I–II PaC receiving chemotherapy, with country-specific association patterns and strengths. We further established and internationally validated a novel, robust, and reliable survival-predicting model, which may provide the basis for more precise individualized survival estimation and which could be useful for clinical counseling. Our nomogram in line with all known models predicting survival in resected PaC performs modestly.

Abbreviations

HR:

Hazard ratio

PaC:

Pancreatic cancer

SEER:

Surveillance, Epidemiology, and End Results

References

  1. Torre LA, Bray F, Siegel RL, Ferlay J, Lortet-Tieulent J, Jemal A. Global cancer statistics, 2012. CA Cancer J Clin. 2015;65(2):87–108.

    Article  Google Scholar 

  2. Khorana AA, Mangu PB, Berlin J, Engebretson A, Hong TS, Maitra A, Mohile SG, Mumber M, Schulick R, Shapiro M, et al. Potentially curable pancreatic Cancer: American Society of Clinical Oncology Clinical Practice Guideline Update. J Clin Oncol. 2017;35(20):2324–8.

  3. Huang L, Jansen L, Balavarca Y, Molina-Montes E, Babaei M, van der Geest L, Lemmens V, Van Eycken L, De Schutter H, Johannesen TB, et al. Resection of pancreatic cancer in Europe and USA: an international large-scale study highlighting large variations. Gut. 2019;68(1):130–9.

    Article  Google Scholar 

  4. Huang L, Jansen L, Balavarca Y, Babaei M, van der Geest L, Lemmens V, Van Eycken L, De Schutter H, Johannesen TB, Primic-Zakelj M, et al. Stratified survival of resected and overall pancreatic cancer patients in Europe and the USA in the early twenty-first century: a large, international population-based study. BMC Med. 2018;16(1):125.

    Article  Google Scholar 

  5. Tempero MA, Malafa MP, Al-Hawary M, Asbun H, Bain A, Behrman SW, Benson AB 3rd, Binder E, Cardin DB, Cha C, et al. Pancreatic adenocarcinoma, version 2.2017, NCCN Clinical Practice Guidelines in Oncology. J Natl Compr Cancer Netw. 2017;15(8):1028–61.

    Article  Google Scholar 

  6. Khorana AA, Mangu PB, Berlin J, Engebretson A, Hong TS, Maitra A, Mohile SG, Mumber M, Schulick R, Shapiro M, et al. Potentially curable pancreatic cancer: American Society of Clinical Oncology Clinical Practice Guideline. J Clin Oncol. 2016;34(21):2541–56.

    Article  Google Scholar 

  7. Ducreux M, Cuhna AS, Caramella C, Hollebecque A, Burtin P, Goere D, Seufferlein T, Haustermans K, Van Laethem JL, Conroy T, et al. Cancer of the pancreas: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up. Ann Oncol. 2015;26(Suppl 5):v56–68.

    Article  Google Scholar 

  8. Huang L, Jansen L, Balavarca Y, van der Geest L, Lemmens V, Van Eycken L, De Schutter H, Johannesen TB, Primic-Zakelj M, Zadnik V, et al. Non-surgical therapies for resected and unresected pancreatic cancer in Europe and USA in 2003-2014: a large international population-based study. Int J Cancer. 2018;143(12):3227–39.

    Article  CAS  Google Scholar 

  9. Garcea G, Dennison AR, Pattenden CJ, Neal CP, Sutton CD, Berry DP. Survival following curative resection for pancreatic ductal adenocarcinoma. A systematic review of the literature. JOP. 2008;9(2):99–132.

    PubMed  Google Scholar 

  10. Li D, Hu B, Zhou Y, Wan T, Si X. Impact of tumor size on survival of patients with resected pancreatic ductal adenocarcinoma: a systematic review and meta-analysis. BMC Cancer. 2018;18(1):985.

    Article  Google Scholar 

  11. Schorn S, Demir IE, Haller B, Scheufele F, Reyes CM, Tieftrunk E, Sargut M, Goess R, Friess H, Ceyhan GO. The influence of neural invasion on survival and tumor recurrence in pancreatic ductal adenocarcinoma - a systematic review and meta-analysis. Surg Oncol. 2017;26(1):105–15.

    Article  Google Scholar 

  12. Pindak D, Tomas M, Dolnik J, Duchon R, Pavlendova J. Morbidity, mortality and long term survival in patients with vascular resection in pancreatic cancer - single center experience. Neoplasma. 2017;64(3):460–3.

    Article  CAS  Google Scholar 

  13. Yamamoto T, Yagi S, Kinoshita H, Sakamoto Y, Okada K, Uryuhara K, Morimoto T, Kaihara S, Hosotani R. Long-term survival after resection of pancreatic cancer: a single-center retrospective analysis. World J Gastroenterol. 2015;21(1):262–8.

    Article  Google Scholar 

  14. Jouffret L, Turrini O, Ewald J, Moutardier V, Iovanna JL, Delpero JR. Long-term survivors after pancreatectomy for cancer: the TNM classification is outdated. ANZ J Surg. 2015;85(11):860–4.

    Article  Google Scholar 

  15. Erdmann JI, Morak MJ, Duivenvoorden HJ, van Dekken H, Kazemier G, Kok NF, van Eijck CH. Long-term survival after resection for non-pancreatic periampullary cancer followed by adjuvant intra-arterial chemotherapy and concomitant radiotherapy. HPB (Oxford). 2015;17(7):573–9.

    Article  Google Scholar 

  16. Cancer Genome Atlas Research Network, Electronic address aadhe, Cancer genome atlas research N. Integrated genomic characterization of pancreatic ductal adenocarcinoma. Cancer Cell. 2017;32(2):185–203 e113.

    Article  Google Scholar 

  17. Yurgelun MB, Chittenden AB, Morales-Oyarvide V, Rubinson DA, Dunne RF, Kozak MM, Qian ZR, Welch MW, Brais LK, Da Silva A, et al. Germline cancer susceptibility gene variants, somatic second hits, and survival outcomes in patients with resected pancreatic cancer. Genet Med. 2019;21(1):213–23.

    Article  Google Scholar 

  18. Lawrence YR, Moughan J, Magliocco AM, Klimowicz AC, Regine WF, Mowat RB, DiPetrillo TA, Small W Jr, Simko JP, Golan T, et al. Expression of the DNA repair gene MLH1 correlates with survival in patients who have resected pancreatic cancer and have received adjuvant chemoradiation: NRG Oncology RTOG Study 9704. Cancer. 2018;124(3):491–8.

    Article  CAS  Google Scholar 

  19. Mahajan UM, Langhoff E, Goni E, Costello E, Greenhalf W, Halloran C, Ormanns S, Kruger S, Boeck S, Ribback S, et al. Immune cell and stromal signature associated with progression-free survival of patients with resected pancreatic ductal adenocarcinoma. Gastroenterology. 2018;155(5):1625–1639 e1622.

    Article  CAS  Google Scholar 

  20. Yan H, Qiu W, Koehne de Gonzalez AK, Wei JS, Tu M, Xi CH, Yang YR, Peng YP, Tsai WY, Remotti HE, et al. HHLA2 is a novel immune checkpoint protein in pancreatic ductal adenocarcinoma and predicts post-surgical survival. Cancer Lett. 2019;442:333–40.

    Article  CAS  Google Scholar 

  21. Sideras K, Biermann K, Yap K, Mancham S, Boor PPC, Hansen BE, Stoop HJA, Peppelenbosch MP, van Eijck CH, Sleijfer S, et al. Tumor cell expression of immune inhibitory molecules and tumor-infiltrating lymphocyte count predict cancer-specific survival in pancreatic and ampullary cancer. Int J Cancer. 2017;141(3):572–82.

    Article  CAS  Google Scholar 

  22. Balachandran VP, Gonen M, Smith JJ, DeMatteo RP. Nomograms in oncology: more than meets the eye. Lancet Oncol. 2015;16(4):e173–80.

    Article  Google Scholar 

  23. Brennan MF, Kattan MW, Klimstra D, Conlon K. Prognostic nomogram for patients undergoing resection for adenocarcinoma of the pancreas. Ann Surg. 2004;240(2):293–8.

    Article  Google Scholar 

  24. Tol JA, Brosens LA, van Dieren S, van Gulik TM, Busch OR, Besselink MG, Gouma DJ. Impact of lymph node ratio on survival in patients with pancreatic and periampullary cancer. Br J Surg. 2015;102(3):237–45.

    Article  CAS  Google Scholar 

  25. Surveillance, Epidemiology, and End Results (SEER) Program (www.seer.cancer.gov) Research Data (1973–2015), National Cancer Institute, DCCPS, Surveillance Research Program, released April 2018, based on the November 2017 submission.

  26. Kamarajah SK, Burns WR, Frankel TL, Cho CS, Nathan H. Validation of the American Joint Commission on Cancer (AJCC) 8th Edition Staging System for Patients with Pancreatic Adenocarcinoma: a Surveillance, Epidemiology and End Results (SEER) analysis. Ann Surg Oncol. 2017;24(7):2023–30.

    Article  Google Scholar 

  27. Hess KR. Graphical methods for assessing violations of the proportional hazards assumption in Cox regression. Stat Med. 1995;14(15):1707–23.

    Article  CAS  Google Scholar 

  28. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15(4):361–87.

    Article  Google Scholar 

  29. Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. 1983;148(3):839–43.

    Article  CAS  Google Scholar 

  30. de Castro SM, Biere SS, Lagarde SM, Busch OR, van Gulik TM, Gouma DJ. Validation of a nomogram for predicting survival after resection for adenocarcinoma of the pancreas. Br J Surg. 2009;96(4):417–23.

    Article  Google Scholar 

  31. Clark EJ, Taylor MA, Connor S, O'Neill R, Brennan MF, Garden OJ, Parks RW. Validation of a prognostic nomogram in patients undergoing resection for pancreatic ductal adenocarcinoma in a UK tertiary referral centre. HPB (Oxford). 2008;10(6):501–5.

    Article  CAS  Google Scholar 

  32. Ferrone CR, Kattan MW, Tomlinson JS, Thayer SP, Brennan MF, Warshaw AL. Validation of a postresection pancreatic adenocarcinoma nomogram for disease-specific survival. J Clin Oncol. 2005;23(30):7529–35.

    Article  Google Scholar 

  33. Schlitter AM, Jesinghaus M, Jager C, Konukiewitz B, Muckenhuber A, Demir IE, Bahra M, Denkert C, Friess H, Kloeppel G, et al. pT but not pN stage of the 8th TNM classification significantly improves prognostication in pancreatic ductal adenocarcinoma. Eur J Cancer. 2017;84:121–9.

    Article  Google Scholar 

  34. Allen PJ, Kuk D, Castillo CF, Basturk O, Wolfgang CL, Cameron JL, Lillemoe KD, Ferrone CR, Morales-Oyarvide V, He J, et al. Multi-institutional validation study of the American Joint Commission on Cancer (8th edition) changes for T and N staging in patients with pancreatic adenocarcinoma. Ann Surg. 2017;265(1):185–91.

    Article  Google Scholar 

  35. Schnelldorfer T, Ware AL, Sarr MG, Smyrk TC, Zhang L, Qin R, Gullerud RE, Donohue JH, Nagorney DM, Farnell MB. Long-term survival after pancreatoduodenectomy for pancreatic adenocarcinoma: is cure possible? Ann Surg. 2008;247(3):456–62.

    Article  Google Scholar 

  36. Kuhlmann KF, de Castro SM, Wesseling JG, ten Kate FJ, Offerhaus GJ, Busch OR, van Gulik TM, Obertop H, Gouma DJ. Surgical treatment of pancreatic adenocarcinoma; actual survival and prognostic factors in 343 patients. Eur J Cancer. 2004;40(4):549–58.

    Article  CAS  Google Scholar 

  37. Huebner M, Kendrick M, Reid-Lombardo KM, Que F, Therneau T, Qin R, Donohue J, Nagorney D, Farnell M, Sarr M. Number of lymph nodes evaluated: prognostic value in pancreatic adenocarcinoma. J Gastrointest Surg. 2012;16(5):920–6.

    Article  Google Scholar 

  38. Murakami Y, Uemura K, Sudo T, Hayashidani Y, Hashimoto Y, Nakashima A, Yuasa Y, Kondo N, Ohge H, Sueda T. Number of metastatic lymph nodes, but not lymph node ratio, is an independent prognostic factor after resection of pancreatic carcinoma. J Am Coll Surg. 2010;211(2):196–204.

    Article  Google Scholar 

  39. Hammel P, Huguet F, van Laethem JL, Goldstein D, Glimelius B, Artru P, Borbath I, Bouche O, Shannon J, Andre T, et al. Effect of chemoradiotherapy vs chemotherapy on survival in patients with locally advanced pancreatic cancer controlled after 4 months of gemcitabine with or without erlotinib: the LAP07 Randomized Clinical Trial. JAMA. 2016;315(17):1844–53.

    Article  CAS  Google Scholar 

  40. Luberice K, Downs D, Sadowitz B, Ross S, Rosemurgy A. Has survival improved following resection for pancreatic adenocarcinoma? Am J Surg. 2017;214(2):341–6.

    Article  Google Scholar 

  41. Benassai G, Quarto G, Perrotta S, Furino E, Benassai GL, Amato B, Bianco T, De Palma G, Forestieri P. Long-term survival after curative resection for pancreatic ductal adenocarcinoma--surgical treatment. Int J Surg. 2015;21(Suppl 1):S1–3.

    Article  Google Scholar 

  42. Konstantinidis IT, Warshaw AL, Allen JN, Blaszkowsky LS, Castillo CF, Deshpande V, Hong TS, Kwak EL, Lauwers GY, Ryan DP, et al. Pancreatic ductal adenocarcinoma: is there a survival difference for R1 resections versus locally advanced unresectable tumors? What is a "true" R0 resection. Ann Surg. 2013;257(4):731–6.

    Article  Google Scholar 

  43. Tempero MA, Malafa MP, Behrman SW, Benson AB 3rd, Casper ES, Chiorean EG, Chung V, Cohen SJ, Czito B, Engebretson A, et al. Pancreatic adenocarcinoma, version 2.2014: featured updates to the NCCN guidelines. J Natl Compr Cancer Netw. 2014;12(8):1083–93.

    Article  CAS  Google Scholar 

  44. Chandrasegaram MD, Goldstein D, Simes J, Gebski V, Kench JG, Gill AJ, Samra JS, Merrett ND, Richardson AJ, Barbour AP. Meta-analysis of radical resection rates and margin assessment in pancreatic cancer. Br J Surg. 2015;102(12):1459–72.

    Article  CAS  Google Scholar 

  45. Butturini G, Stocken DD, Wente MN, Jeekel H, Klinkenbijl JH, Bakkevold KE, Takada T, Amano H, Dervenis C, Bassi C, et al. Influence of resection margins and treatment on survival in patients with pancreatic cancer: meta-analysis of randomized controlled trials. Arch Surg. 2008;143(1):75–83. discussion 83.

    Article  Google Scholar 

  46. Neoptolemos JP, Stocken DD, Bassi C, Ghaneh P, Cunningham D, Goldstein D, Padbury R, Moore MJ, Gallinger S, Mariette C, et al. Adjuvant chemotherapy with fluorouracil plus folinic acid vs gemcitabine following pancreatic cancer resection: a randomized controlled trial. JAMA. 2010;304(10):1073–81.

    Article  CAS  Google Scholar 

  47. Regine WF, Winter KA, Abrams R, Safran H, Hoffman JP, Konski A, Benson AB, Macdonald JS, Rich TA, Willett CG. Fluorouracil-based chemoradiation with either gemcitabine or fluorouracil chemotherapy after resection of pancreatic adenocarcinoma: 5-year analysis of the U.S. Intergroup/RTOG 9704 phase III trial. Ann Surg Oncol. 2011;18(5):1319–26.

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank Dr. Dianne Pulte very much for her kind assistance with the English language. We are very grateful to the staff in Surveillance, Epidemiology, and End Results Program (SEER), Belgian Cancer Registry (BCR), Netherlands Cancer Registry (NCR), Dutch Pancreatic Cancer Group (DPCG), Cancer Registry of Slovenia (CRS), and Cancer Registry of Norway (CRN) for their kind work in data collection and delivery, and to Tom Hueting and the great Evidencio (https://www.evidencio.com/) team for the kind assistance with the development of the online tool.

Grant support

This study was supported in part by the German Cancer Aid (Deutsche Krebshilfe, #111365). The funder had no involvement in study design; in the collection, analysis, or interpretation of data; in the writing of the report; or in the decision to submit the paper for publication.

Funding

This study was supported in part by the German Cancer Aid (Deutsche Krebshilfe, #111365). The funder had no involvement in study design; in the collection, analysis, or interpretation of data; in the writing of the report; or in the decision to submit the paper for publication.

Availability of data and materials

The data that support the findings of this study are available from each participating registry but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available.

Author information

Authors and Affiliations

Authors

Contributions

Conception or design were provided by HL, JL, BY, and BH. Acquisition, analysis, or interpretation of data were supplied by HL, JL, BY, vdGL, LV, VEL, DSH, JT, ZV, P-ŽM, MM, GR, BM, S-KP, and BH. Drafting of the manuscript was done by HL. Critical revision of the manuscript for important intellectual content was imparted by JL, BY, vdGL, LV, VEL, DSH, JT, ZV, P-ŽM, MM, GR, BM, S-KP, and BH. Statistical analysis was contributed by HL and BY. Administrative, technical, or material support was provided by S-KP and BH. All authors have given the final approval of the manuscript for submission and publication.

Corresponding author

Correspondence to Lina Jansen.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the Ethics Committee of the Medical Faculty Heidelberg. Consent was waived considering the anonymous, observational, population-based, and registry-based nature. No individual patient data were reported.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:

Table S1. Selection of contacted national population-based cancer registries in Europe. Table S2. General information on participating population-based registries. Table S3. Inclusion and exclusion codes according to International Classification of Diseases for Oncology, Third Edition. Table S4. Comparison of the Memorial Sloan-Kettering Cancer Center nomogram with the nomogram established in this study for survival in resected pancreatic cancer for Western patient. (DOCX 33 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, L., Balavarca, Y., van der Geest, L. et al. Development and validation of a prognostic model to predict the prognosis of patients who underwent chemotherapy and resection of pancreatic adenocarcinoma: a large international population-based cohort study . BMC Med 17, 66 (2019). https://0-doi-org.brum.beds.ac.uk/10.1186/s12916-019-1304-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12916-019-1304-y

Keywords