Development and external validation of models to predict acute respiratory distress syndrome related to severe acute pancreatitis

Yun-Long Li, Ding-Ding Zhang, Yang-Yang Xiong, Rui-Feng Wang, Xiao-Mao Gao, Hui Gong, Shi-ChengZheng, Dong Wu

Abstract

Key Words: Acute pancreatitis; Acute respiratory distress syndrome; Nomogram; Calibration; Early identification; Predictive model

INTRODUCTION

Acute pancreatitis (AP) is a common event incurring pain, socioeconomic loss, and even death. The majority of the patients who present with mild organ injury and self-limited course are diagnosed with mild acute pancreatitis (MAP) or moderately severe acute pancreatitis (MSAP)[1 ,2 ]. However, it is estimated that approximately 20 % of patients are critically ill and develop SAP, leading to consistent organ failure and significant mortality[2 ,3 ]. Our previous studies indicated that the lung are the most commonly affected organs in SAP[4 ,5 ], and acute respiratory distress syndrome (ARDS) is recognized as an important cause of respiratory failure, with a high mortality rate[6 -8 ]. It is reported that 4 %-15 % of AP patients are complicated with ARDS[9 ], while this proportion might be as high as one third in SAP[10 ]. However, to date, the therapeutic options for SAP and ARDS are limited. Therefore, it is necessary to identify patients at risk and adopt interventions to prevent MAP or MSAP from progressing to SAP and ARDS. The protective effect of early intervention for patients with predicted SAP or patients at risk of ARDS has been confirmed by numerous clinical trials and meta-analyses, although the inclusion criteria for patients have varied according to different studies[11 -15 ].

A plethora of models have been published to predict the risk of SAP in AP patients, including acute physiology and chronic health evaluation II (APACHE-II) score, Ranson criteria, computed tomography severity index (CTSI), and bedside index for severity in acute pancreatitis (BISAP)[16 ,17 ]. Lung injury prediction score (LIPS) and other models have also been used to evaluate the risk of ARDS in patients with non-AP[18 -20 ]. However, to date the models used to predict ARDS in AP are scarce. Furthermore,the majority of the SAP predictive models are hard to use in practice due to various parameters,complicated calculation and dependence on radiological assessment. The majority of the models also lack internal or external validation, which reduces their reliability in other cohorts[16 ]. Therefore, a new concise model may be more practical in the emergency department in order to identify SAP and ARDS in the early course of AP. This model should involve limited available clinical data and should not rely on radiological examinations.

The objective of the present study was to develop and validate models to predict SAP and ARDS in patients with AP based on multicenter retrospective cohorts. The comparison of novel models with quick sequential organ failure assessment (qSOFA), systemic inflammatory response syndrome (SIRS)and BISAP is essential to display the power of different models with a low number of variables. These models usually contain three, four and five items, respectively[21 ,22 ].

MATERIALS AND METHODS

Study design and participants

This was a multicenter retrospective study. Sample size was calculated with PASS 11 .0 . The proportion of SAP was set to 20 %, and the incidence of ARDS in SAP and non-SAP patients was set to 1 /3 and 4 %,respectively. Considering that the dropout rate was low in hospitals, we set it to 5 %. In the end, with α =0 .01 and β = 0 .10 , a total of 211 participants were needed. Patients diagnosed with AP between 1 January 2017 and 31 December 2019 were recruited from different regions of China (Peking Union Medical College Hospital and The Sixth Hospital of Beijing at Northern China, The Fourth Affiliated Hospital of Harbin Medical University at Northeastern China, West China Longquan Hospital Sichuan University at Southwestern China). The patients were categorized into the derivation cohort in order to develop a clinical predictive model. The independent external validation cohort consisted of patients diagnosed with AP between 1 January 2020 and 31 May 2021 at the Peking Union Medical College Hospital. AP was diagnosed if at least two of the three following criteria were met: (1 ) abdominal pain consistent with AP; (2 ) serum lipase or amylase levels that were more than three times the upper limit of the normal range; and (3 ) characteristic radiological findings of AP on computed tomography (CT) and magnetic resonance imaging or ultrasonography[23 ]. SAP was identified by the presence of persistent organ failure for > 48 h[23 ]. ARDS was diagnosed based on Berlin definition[24 ]. Patients aged < 18 years who lacked the necessary information provided by the Atlanta Classification or relevant etiology information were excluded.

Clinical variables

The following demographic and laboratory data were collected from the electronic medical record system within 24 h of admission: age, sex, temperature, heart rate (HR), respiratory rate (RR), systolic blood pressure, Glasgow coma score (GCS), white blood counts, hematocrit, platelet, serum electrolyte concentration (K, Na and Ca), creatine, blood urea nitrogen (BUN) and glucose. Other clinical information, such as, admission date, local complication, length of hospital stay, length of intensive care unit stay, mortality and ventilator use, was also collected. Weekend admission corresponded to admission on Saturday or Sunday and local complication included acute peripancreatitc fluid collection,acute necrosis collection, pseudocyst and walled-off necrosis. The ventilator included invasive or noninvasive mechanical ventilation. qSOFA, SIRS and BISAP scores were calculated based on the aforementioned data. All data were collected and checked by two or more authors independently.Missing items were added following review of the clinical records. The data that could not be completed were removed and the complete-case dataset was finally analyzed.

Statistical analysis

Categorical variables were expressed as frequencies and percentages and compared usingχ2or Fisher’s exact tests. Normally distributed continuous variables were described as the mean ± SD and compared using a two-sided Student’sttest. Non-normally distributed continuous variables were expressed as the median with the interquartile range and compared using the Mann-WhitneyUtest. Continuous variables were analyzed in their original forms to preserve information[25 ]. The least absolute shrinkage and selection operator (LASSO) regression method was used to select predictors in the derivation cohort. The predictive models were further built using multiple logistic regression analysis. The nomogram was formulated based on multivariate logistic regression analysis. Receiver operating characteristic (ROC) curve and the area under ROC curve (AUC) were used to evaluate the discriminative power of the predictive model, which referred to the ability of the model to differentiate between the subjects that did or did not experience the outcome event[25 ]. The calibration curves were plotted to measure the predictive accuracy of the model, which reflected the agreement between predictions from the model and observed outcomes. A well-calibrated model indicated that the prediction was lying on or around the 45 ° line of the calibration plot[25 ]. Hosmer-Lemeshow (H-L) goodness of fit test was used to quantify the calibration curve. ThePvalue was determined by the H-L test.P> 0 .05 suggested an optimal consistency between model prediction and the criteria required for standard diagnosis. Decision curve analysis (DCA) was used to assess the clinical utility of the model, which indicated he relationship between a model-predicted probability threshold and the relative value of net benefit[25 ].

Statistical analysis was performed using R 4 .0 .3 [26 ] and MedCalc 15 .8 software. A two-side P < 0 .05 was considered to indicate a statistically significant difference. The nomogram and calibration curve were plotted using rms package and DCA was plotted using rmda package. ROC was plotted with MedCalc 15 .8 .

RESULTS

Baseline characteristics

Between January 12017 and May 312021 , 628 patients with AP were recruited from four hospitals and reviewed. The exclusion criteria included the following: age < 18 years (2 patients), lack of Atlanta Classification or etiology (26 patients) and incomplete data (3 patients). Following screening, 407 and 190 patients were involved in the derivation and validation cohorts, respectively. The number of participants in each cohort met the requirement of sample size. The detailed demographic and clinical information were described in Tables 1 and 2 .

Predictors and model construction

Four variables (HR, RR, Ca and BUN) were extracted as the predictors of SAP through LASSO regression. Supplementary Figure 1 and Supplementary Table 1 show this process in more detail.Multivariable logistic regression revealed that all four variables were independent predictors (Table 3 ).The probability (PA) of SAP could be calculated according to the following formula: PA = 1 /{1 + exp [-(-6 .42 + 0 .05 × HR + 0 .08 × RR - 1 .30 × Ca + 0 .14 × BUN)]}. Analysis of ARDS obtained similar results(Table 3 ). The following formula was used for ARDS: PA = 1 /{1 + exp [- (-5 .46 + 0 .05 × HR + 0 .10 × RR -1 .78 × Ca + 0 .11 × BUN)]}. Two nomogram plots were displayed using prediction models (Figure 1 ).

Model performance

The new model indicated a great power of discrimination for SAP. Following 1000 interactions of bootstrapping to minimize the risk of overfitting to the original models, the AUC in the derivation cohort was estimated to 0 .879 (95 %CI: 0 .830 -0 .928 ), which was significantly superior to that of SIRS(AUC = 0 .808 , 95 %CI: 0 .757 -0 .859 , P = 0 .002 ) and qSOFA (AUC = 0 .730 , 95 %CI: 0 .672 -0 .789 , P < 0 .001 )and not inferior to that of the BISAP score (AUC = 0 .888 , 95 %CI: 0 .847 -0 .929 , P = 0 .6629 ) (Figure 2 A,Table 4 ). In addition, the model indicated an optimal behavior in the validation cohort (AUC = 0 .898 ,95 %CI: 0 .848 -0 .949 ) (Figure 2 B). The AUC of the new model in derivation was 0 .892 (95 %CI: 0 .843 -0 .941 )for ARDS prediction, which was superior to SIRS (AUC = 0 .815 , 95 %CI: 0 .766 -0 .864 , P = 0 .001 ) and qSOFA (AUC = 0 .742 , 95 %CI: 0 .678 -0 .807 , P < 0 .001 ) and not inferior to BISAP (AUC = 0 .871 , 95 %CI:0 .827 -0 .916 , P = 0 .344 ) (Figure 2 C, Table 4 ). Despite the assessment of the model in the validation cohort,its performance was moderate (AUC = 0 .833 . 95 %CI: 0 .754 -0 .912 ) (Figure 2 D). When the cut-off value was set as PA > 25 % for SAP prediction, the novel model suggested an optimal performance in the combined dataset (sensitivity 0 .78 , specificity 0 .88 ) (Table 4 ). The best cut-off value was PA > 18 % for ARDS prediction, with a sensitivity of 0 .78 and a specificity of 0 .85 (Table 4 ).

Graphical assessment indicated a strong agreement between prediction and observation in both new models (Figure 3 ). The H-L test indicated that the difference between prediction and observation was not significant both in the derivation (χ2= 12 .675 , P = 0 .124 ) and validation cohorts (χ2 = 5 .852 , P = 0 .664 )with regard to SAP prediction. The model for ARDS prediction revealed improved performance with regard to the calibration in the derivation (χ2= 3 .753 , P = 0 .879 ) and validation cohorts (χ2 = 2 .933 ,P=0 .939 ).

DCA indicated that if the threshold PA was < 80 %, using the new model to recognize and manage SAP had a positive net benefit compared with either the treat-all or treat-none (Figure 4 A). In case the threshold probability was set to < 70 %, the prediction and intervention for ARDS also produced net benefit (Figure 4 B).

DISCUSSION

In the current study, novel prediction models were established for SAP and ARDS in patients with AP.The models were also externally validated and exhibited remarkable discriminative power and highdegree of consistency with the observation both in the derivation and external validation cohorts. These models suggested that patients with AP who manifested a higher heart rate, respiratory rate, blood urea nitrogen concentrations and lower serum calcium concentrations at admission exhibited a higher risk of developing SAP and ARDS.

Table 1 Characteristics of non-severe acute pancreatitis and severe acute pancreatitis patients in derivation and validation cohort

AP is a major cause of acute abdomen. Patients with AP usually present with multiple organ dysfunction syndrome (MODS). Although organ dysfunction is mild and transient (< 48 h), approximately 20 % of patients will proceed to consistent organ failure (> 48 h), leading to SAP and a high riskof mortality[27 ]. AP primarily affects the respiratory system and to a lesser extent the renal and cardiovascular systems[23 ]. ARDS is the critical event, which is noted during lung injury in AP[28 ].Cyclooxygenase-2 inhibitors and enteral nutrition have been shown to prevent SAP, shorten the length of hospital stay and reduce infectious complications and mortality in patients with predicted SAP[11 ,12 ,29 ]. Administration of antiplatelet therapy, withdrawal of prehospital amiodarone treatment and administration of nebulized heparin may decrease the incidence of ARDS, inhibit the progression of lung injury and accelerate the recovery of patients at risk of developing ARDS[13 -15 ,30 ]. Therefore,early identification of patients at risk of developing SAP and ARDS is clinically significant for improving the prognosis of AP.

Table 2 Characteristics of non-acute respiratory distress syndrome and acute respiratory distress syndrome patients in derivation and validation cohorts

Table 3 Multivariable logistic regression analysis for severe acute pancreatitis and acute respiratory distress syndrome prediction in derivation cohort

Table 4 Sensitivity, specificity, positive predictive value, negative predictive value for of the predictive models in combined cohort

Various models have been developed to predict SAP and organ dysfunction for AP management[16 ].Although the majority of the models were deficient due to some limitations, such as small sample size,single center studies and lack of internal or external validation, several SAP models have been widely used and validated in different cohorts. These validations were performed using APACHE-II score,Ranson criteria, CTSI and BISAP[31 -34 ]. BISAP contains only five variables and is simpler than APACHE-II score (18 items) and the Ranson criteria (11 items). However, their predictive power is equal[35 ]. In addition, ultrasound is preferred to CT as an efficient and nonradioactive examination used in the emergency department to initially evaluate potential development of AP. Therefore, pleural effusion and CT presentation could not be evaluated in this case to gain the BISAP and CTSI scores. Although LIPS is a popular model to predict ARDS for patients at risk, its calculation is considerably complicated[18 ]. In addition, its original developing cohort involved only a small part of patients with AP. LIPS had not been previously validated in patients with AP.

Therefore, a simple model with a low number of parameters and without radiology findings would be more practical. The novel predictive model reported in the current study involved only four parameters for both SAP and ARDS prediction and all these variables were routinely tested. The discriminatory power of the novel model was not inferior to that of BISAP. To the best of our knowledge, the prediction of SAP or ARDS for patients with AP has not been previously assessed by models that were as simple and accurate as this reported in the current study.

Using LASSO regression, calcium was identified as a predictor for both SAP and ARDS. Multiple logistic regression analysis indicated that serum calcium concentration was also the independent predictor. The models of the present study were easier to use than the APACHE-II, Ranson, CTSI and BISAP models and demonstrated improved efficacy than the other two simple models, suggesting their potential clinical significance. ARDS was a non-negligible manifestation of MODS in patients with AP.The majority of the models have mainly focused on the severity classification or mortality prediction of ARDS, whereas the identification of ARDS at an early stage is still challenging. Although LIPS was widely used to predict ARDS, the calculation of the LIPS score was complicated for patients admitted to the emergency department, since certain parameters may be unavailable[18 ]. The pathogenesis of ARDS involves the activation of signaling pathways, which include various cytokines and inflammatory mediators. Certain molecules, such as interleukin (IL)-6 , IL-8 , protein C, angiopoietin-2 and miRNAs and specific imaging examinations (X-ray and lung ultrasound) were also identified as predictors of ARDS in single or combined forms[8 ,36 -38 ]. The data indicated that these new predictors seemed promising. However, the molecules and the examinations identified were not part of the routine clinical practice, which limited their clinical utility. Feiet al[10 ] used an artificial neural network algorithm to predict ARDS following SAP. The model by Feiet al[10 ] indicated high accuracy. However, the variable pancreatic necrosis rate was hard to assess when CT was not used and was not evident in the early course of AP.

Figure 1 Nomograms of new predictive models. A: Nomogram of severe acute pancreatitis predictive model; B: Nomogram of acute respiratory distress syndrome prediction model. SAP: Severe acute pancreatitis; ARDS: Acute respiratory distress syndrome; HR: Heart rate; RR: Respiratory rate; Ca: Serum calcium concentration; BUN: Blood urea nitrogen.

Figure 2 Receiver operating characteristic curves of different predictive models in derivation and validation cohort. A: Receiver operating characteristic (ROC) curves of severe acute pancreatitis (SAP) predictive models in derivation cohort; B: ROC curves of SAP predictive models in validation cohort;C: ROC curves of acute respiratory distress syndrome (ARDS) predictive models in derivation cohort; D: ROC curves of ARDS predictive models in validation cohort.BISAP: Bedside index for severity in acute pancreatitis; qSOFA: Quick sequential organ failure assessment; SIRS: Systemic inflammatory response syndrome.

The pathogenesis of SAP and ARDS involves a series of acute inflammatory reactions[2 ,39 ]. SIRS is widely adopted to assess the severity of diseases associated with acute inflammation. Both HR and RR are used in the SIRS model. Therefore, it is reasonable that both HR and RR were identified as predictors of SAP and ARDS. BUN has been shown to reflect volume depletion, renal function, the quality of resuscitates and even the ischemic injury of the pancreas during AP[40 ]. It has been reported that BUN can independently predict both SAP and the mortality of SAP[40 -44 ]. Therefore, BUN was also involved in other predictive models of SAP, such as GCS, Ranson criteria and BISAP, in addition to our new model. The levels of BUN have not been used as a direct predictor of ARDS. However, this marker can be used as a predictor of pathogenesis in association with other risk factors, such as pancreatitis[39 ].Calcium concentration has been closely associated with AP[45 ]. Hypocalcemia was common in the cohort of the present study and in other AP cohorts; notably in critically ill patients[46 ]. Elevated cytosolic calcium of pancreatic acinar cells causes premature trypsinogen activation, vacuolization and acinar cell death, which play critical roles in the pathogenesis of AP[47 ]. However, during the development of certain models for the prediction of SAP, serum calcium was excluded for a variety of factors[48 ]. Calcium is also involved in the pathogenesis of ARDS as a signaling molecule, leading to paracellular hyperpermeability through endothelial junction-cytoskeleton dissociation[49 ]. The current model suggested that calcium was an independent predictor of SAP and ARDS in AP, indicating the potential of developing novel drugs for the treatment of AP[50 ]. To the best of our knowledge, the model of the present study was the simplest used to predict SAP and ARDS within 24 h of AP admission. It is also the first model that involved serum calcium concentration to predict ARDS in AP.

Figure 3 Calibration curves of new predictive models. A: Calibration plot of severe acute pancreatitis (SAP) predictive model in derivation cohort; B:Calibration plot of SAP predictive model in validation cohort; C: Calibration plot of acute respiratory distress syndrome (ARDS) predictive model in derivation cohort;D: Calibration plot of ARDS predictive model in validation cohort.

The present study had several limitations. Firstly, organ dysfunction occurred mainly in the first week of AP, whereas accurate onset time was not available in the present study. Therefore, certain patients with SAP or ARDS may have been missed. Moreover, the exact onset time of ARDS was not recorded, so the new model only predicted the risk of ARDS during the whole admission (7 -15 d) using the scores gained within 24 h of admission. It might not be appropriate, and the new model could not tell the clinicians when they should prepare for the possible onset of ARDS. Further studies are needed to verify the value of new models on a dynamic timescale. Secondly, the derivation cohort comprised tertiary and secondary hospitals from different regions of China. However, model validation was performed in a tertiary teaching hospital. Although the result of validation was also encouraging, it is hard to ignore that the incidence of SAP and ARDS in the validation cohort was considerably higher than that of the derivation cohort, which could reduce the generalizability in primary or secondary class hospitals, where MAP and MSAP exhibited high proportions. Moreover, certain laboratory examination technologies were different among four hospitals, which increased the systemic error of the data. It must be mentioned that mechanical ventilation will attenuate systemic inflammation of ARDS and the effect varies with patterns[51 ]. It is unavoidable that new scores to predict ARDS will also be affected.Unfortunately, detailed information of mechanical ventilation was not collected in this study, and further research is needed to investigate the influence of different ventilatory patterns on new models.Thirdly, selection bias was inevitable in a retrospective study. Furthermore, any missing value was deleted to obtain a complete-case dataset for analysis rather than imputating missing data with statistical methods (e.g., multi-imputation), which were not recommended. However, in the present study, only three individuals were removed due to missing data, accounting for a tiny part of the cohort. It was considered that complete data analysis would not affect the overall conclusion. Finally,other common predictive models, such as APACHE-II score, Ranson criteria, CTSI and LIPS were not evaluated due to lack of essential parameters. Therefore, direct comparison among different models was unavailable.

Figure 4 Decision curve analysis of new predictive models. A: Decision curve analysis of severe acute pancreatitis predictive model in derivation and validation cohort; B: Decision curve analysis of acute respiratory distress syndrome predictive model in derivation and validation cohort.

CONCLUSION

Novel models were developed containing only four items to predict SAP and ARDS in patients with AP,which were as accurate as BISAP but simpler. Serum calcium was identified as an important predictor,indicating a potential new strategy for management of AP. Further prospective studies are required to reveal whether early intervention based on novel prediction models could reduce the incidence of SAP and ARDS and finally improve the outcome of patients with AP.

ARTICLE HIGHLIGHTS

FOOTNOTES

Author contributions:Li YL, Zhang DD, and Xiong YY contributed equally to this work; Li YL, Zhang DD, Xiong YY and Wu D designed the research study; Li YL, Xiong YY, Wang RF, Gao XM, Gong H, Zheng SC, and Wu D performed the study and collected the data; Li YL, Zhang DD, and Xiong YY analyzed the data and wrote the manuscript; All authors have read and approved the final manuscript.

Supported bythe Chinese Natural Science Foundation, No. 32170788 .

Institutional review board statement:This study was approved by the Ethics Committee of Peking Union Medical College Hospital (Approval No. S-K1772 ).

Conflict-of-interest statement:All authors have no conflict of interest to disclose.

Data sharing statement:No additional data are available.

STROBE statement:The authors have read the STROBE statement, and the manuscript was prepared and revised according to the STROBE statement.

Open-Access:This article is an open-access article that was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution NonCommercial (CC BYNC 4 .0 ) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is noncommercial. See: https://creativecommons.org/Licenses/by-nc/4 .0 /

Country/Territory of origin:China

ORCID number:Yun-Long Li 0000 -0002 -7367 -0772 ; Ding-Ding Zhang 0000 -0002 -5234 -752 X; Yang-Yang Xiong 0000 -0002 -0266 -7592 ; Rui-Feng Wang 0000 -0002 -8614 -8481 ; Xiao-Mao Gao 0000 -0001 -5913 -2735 ; Hui Gong 0000 -0001 -6398 -384 X;Shi-Cheng Zheng 0000 -0003 -1548 -4282 ; Dong Wu 0000 -0001 -9430 -9874 .

S-Editor:Zhang H

L-Editor:Kerr C

P-Editor:Zhang H