Predicting complications of spine surgery: external validation of three models


      • SpineSage and the Risk Assessment Tool from Ratliff et al. are well calibrated for spine surgery
      • The NSQIP Risk Calculator systemically underestimates the risk of complications after spine surgery
      • Decision curve analysis suggests that using SpineSage may help clinicians and patients determine whether to undertake surgery



      Numerous prediction tools are available for estimating postoperative risk following spine surgery. External validation and comparison of these tools is critical prior to clinical use. No model for adverse events after spine surgery has undergone decision curve analysis.


      External validation, comparison, and decision curve analysis of 3 previously described models [SpineSage, Risk Assessment Tool (RAT), National Surgical Quality Improvement Program Risk Calculator (NSQIP)] for predicting 30-day postoperative complications after spine surgery


      Retrospective cohort study.


      Three hundred fifteen patients who underwent spine surgery at a tertiary academic surgical center in New Zealand between January 2019 and April 2020.


      As defined by each risk prediction tool and objectively using the Comprehensive Complication Index.


      We retrospectively reviewed risk of postoperative complication was calculated for each patient according to the 3 models. Overall model fit, calibration, discrimination, and decision curve analysis for each model were assessed in line with the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guidelines.


      100 (35%) patients experienced complications. SpineSage and RAT were well calibrated, NSQIP systematically underestimated risk. Area under the curve was greatest for SpineSage (0.75) compared with the NSQIP (0.72) and the RAT (0.69). Decision curve analysis showed SpineSage resulted in greatest net benefit across all risk thresholds.


      Of the models studied, SpineSage most accurately predicted risk and can be expected to perform better than a strategy of treating all patients if patient or surgeon deem complication risk >10% significant. NSQIP may not be suitable for the clinical use in our local population.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to The Spine Journal
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Reisener M-J
        • Pumberger M
        • Shue J
        • Girardi FP
        • Hughes AP.
        Trends in lumbar spinal fusion—a literature review.
        J Spine Surg. 2020; 6: 752-761
        • Powell JM
        • Rai A
        • Foy M
        • et al.
        The ‘three-legged stool’: a system for spinal informed consent.
        Bone Jt J. 2016; (98-B:1427–30)
      1. Iezzoni, LI. Risk adjustment for measuring health care outcomes. 2020, Chi; n.d.

        • Lee MJ
        • Cizik AM
        • Hamilton D
        • Chapman JR.
        Predicting medical complications after spine surgery: a validated model using a prospective surgical registry.
        Spine J. 2014; 14: 291-299
        • Bilimoria KY
        • Liu Y
        • Paruch JL
        • et al.
        Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons.
        J Am Coll Surg. 2013; 217 (e3): 833-842
        • Ratliff JK
        • Balise R
        • Veeravagu A
        • et al.
        Predicting occurrence of spine surgery complications using “Big Data” modeling of an administrative claims database.
        J Bone Jt Surg. 2016; 98: 824-834
        • Scheer JK
        • Smith JS
        • Schwab F
        • et al.
        Development of a preoperative predictive model for major complications following adult spinal deformity surgery.
        J Neurosurg Spine. 2017; 26: 736-743
        • Buchlak QD
        • Yanamadala V
        • Leveque J-C
        • Edwards A
        • Nold K
        • Sethi R.
        The Seattle spine score: predicting 30-day complication risk in adult spinal deformity surgery.
        J Clin Neurosci. 2017; 43: 247-255
        • Broda A
        • Sanford Z
        • Turcotte J
        • Patton C.
        Development of a risk prediction model with improved clinical utility in elective cervical and lumbar spine surgery.
        Spine. 2020; 45: E542-E551
        • Veeravagu A
        • Li A
        • Swinney C
        • et al.
        Predicting complication risk in spine surgery: a prospective analysis of a novel risk assessment tool.
        J Neurosurg Spine. 2017; 27: 81-91
        • Kasparek MF
        • Boettner F
        • Rienmueller A
        • et al.
        Predicting medical complications in spine surgery: evaluation of a novel online risk calculator.
        Eur Spine J. 2018; 27: 2449-2456
        • Wang X
        • Hu Y
        • Zhao B
        • Su Y.
        Predictive validity of the ACS-NSQIP surgical risk calculator in geriatric patients undergoing lumbar surgery.
        Medicine (Baltimore). 2017; 96: e8416
        • Groot OQ
        • Bindels BJJ
        • Ogink PT
        • et al.
        Availability and reporting quality of external validations of machine-learning prediction models with orthopedic surgical outcomes: a systematic review.
        Acta Orthop. 2021; 92: 385-393
      2. Risk Assessment Tool for Spine Surgery Procedures, 2020. App Store n.d. Available at: Accessed September 13, 2021.

      3. SpineSage n.d. 2020. Available at: Accessed June 17, 2022.

      4. ACS Risk Calculator - Patient Information n.d. Available at: 2020, Accessed September 13, 2021.

        • Han SS
        • Azad TD
        • Suarez PA
        • Ratliff JK.
        A machine learning approach for predictive models of adverse events following spine surgery.
        Spine J. 2019; 19: 1772-1781
        • McCarthy MH
        • Singh P
        • Nayak R
        • et al.
        Can the American college of surgeons risk calculator predict 30-day complications after spine surgery?.
        Spine. 2020; 45: 621-628
        • Collins GS
        • Reitsma JB
        • Altman DG
        • Moons K.
        Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement.
        BMC Med. 2015; 13: 1
        • Mirza SK
        • Deyo RA
        • Heagerty PJ
        • et al.
        Development of an index to characterize the “Invasiveness” of spine surgery: validation by comparison to blood loss and operative time.
        Spine. 2008; 33: 2651-2661
        • Collins GS
        • Ogundimu EO
        • Altman DG.
        Sample size considerations for the external validation of a multivariable prognostic model: a resampling study: sample size considerations for validating a prognostic model.
        Stat Med. 2016; 35: 214-226
        • Crowson CS
        • Atkinson EJ
        • Therneau TM.
        Assessing calibration of prognostic risk scores.
        Stat Methods Med Res. 2016; 25: 1692-1706
        • Slankamenac K
        • Graf R
        • Barkun J
        • Puhan MA
        • Clavien P-A.
        The comprehensive complication index: a novel continuous scale to measure surgical morbidity.
        Ann Surg. 2013; 258: 1-7
        • Clavien P-A
        • Vetter D
        • Staiger RD
        • et al.
        The comprehensive complication index (CCI®): added value and clinical perspectives 3 years “Down the Line.
        Ann Surg. 2017; 265: 1045-1050
        • Artiles-Armas M
        • Roque-Castellano C
        • Conde-Martel A
        • Marchena-Gómez J.
        The comprehensive complication index is related to frailty in elderly surgical patients.
        J Surg Res. 2019; 244: 218-224
        • Kim T-H
        • Suh Y-S
        • Huh Y-J
        • et al.
        The comprehensive complication index (CCI) is a more sensitive complication index than the conventional Clavien–Dindo classification in radical gastric cancer surgery.
        Gastric Cancer. 2018; 21: 171-181
        • de la Plaza Llamas R
        • Ramia Ángel JM
        • Bellón JM
        • et al.
        Clinical validation of the comprehensive complication index as a measure of postoperative morbidity at a surgical department: a prospective study.
        Ann Surg. 2018; 268: 838-844
        • Steyerberg EW
        • Vergouwe Y.
        Towards better clinical prediction models: seven steps for development and an ABCD for validation.
        Eur Heart J. 2014; 35: 1925-1931
        • Rufibach K.
        Use of Brier score to assess binary predictions.
        J Clin Epidemiol. 2010; 63: 938-939
        • Austin PC
        • Steyerberg EW.
        Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers.
        Stat Med. 2014; 33: 517-535
        • Ozenne B
        • Subtil F
        • Maucort-Boulch D.
        The precision–recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases.
        J Clin Epidemiol. 2015; 68: 855-859
        • DeLong ER
        • DeLong DM
        • Clarke-Pearson DL.
        Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.
        Biometrics. 1988; 44: 837-845
        • Vickers AJ
        • Elkin EB.
        Decision curve analysis: a novel method for evaluating prediction models.
        Med Decis Making. 2006; 26: 565-574×06295361
        • Vickers AJ
        • van Calster B
        • Steyerberg EW.
        A simple, step-by-step guide to interpreting decision curve analysis.
        Diagn Progn Res. 2019; 3: 18
      5. Predicting Adverse Events after Spinal Surgery. 2020, <img height="65" width="250" src="st.jpg"/></div>n.d. Available at: Accessed September 13, 2021.

        • Shahsavari S
        • Naderi M
        • Abbasi M.
        Letter to the Editor concerning “Predicting medical complications in spine surgery: evaluation of a novel online risk calculator”.
        Eur Spine J. 2018; 27 (by M. F. Kasparek et al. (Eur Spine J)): 2885-2886
        • Bennette C
        • Vickers A.
        Against quantiles: categorization of continuous variables in epidemiologic research, and its discontents.
        BMC Med Res Methodol. 2012; 12: 21
        • Moons KGM
        • Altman DG
        • Reitsma JB
        • et al.
        Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration.
        Ann Intern Med. 2015; 162: W1
        • Fisher ES
        • Whaley FS
        • Krushat WM
        • et al.
        The accuracy of Medicare's hospital claims data: progress has been made, but problems remain.
        Am J Public Health. 1992; 82: 243-248
        • Guimarães PO
        • Krishnamoorthy A
        • Kaltenbach LA
        • et al.
        Accuracy of medical claims for identifying cardiovascular and bleeding events after myocardial infarction.
        JAMA Cardiol. 2017; 2: 750-757
        • Rudrapatna VA
        • Glicksberg BS
        • Avila P
        • Harding-Theobald E
        • Wang C
        • Butte AJ.
        Accuracy of medical billing data against the electronic health record in the measurement of colorectal cancer screening rates.
        BMJ Open Qual. 2020; 9e000856
        • Moons KGM
        • Wolff RF
        • Riley RD
        • Whiting P.F.
        • Westwood M.
        • Collins G.S.
        • et al.
        PROBAST: a tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration.
        Ann Intern Med. 2019; 170: W1
        • Vickers A.J.
        • Holland F.
        Decision curve analysis to evaluate the clinical benefit of prediction models.
        Spine J. 2021; (S1529943021001121)