Clinical Study| Volume 20, ISSUE 1, P14-21, January 2020

Download started.


External validation of the SORG 90-day and 1-year machine learning algorithms for survival in spinal metastatic disease

Published:September 07, 2019DOI:



      Preoperative survival estimation in spinal metastatic disease helps determine the appropriateness of invasive management. The SORG ML 90-day and 1-year machine learning algorithms for survival in spinal metastatic disease were previously developed in a single institutional sample but remain to be externally validated.


      The purpose of this study was to externally validate these algorithms in an independent population from another institution.


      Retrospective study at a large, tertiary care center.


      Patients 18 years or older who underwent surgery between 2003 and 2016.


      Ninety-day and 1-year mortality.


      Baseline characteristics of the validation cohort were compared to the developmental cohort for the SORG ML algorithms. Discrimination (c-statistic and receiver operating curve), calibration (calibration slope, intercept, calibration plot, and observed proportions by predicted risk groups), overall performance (Brier score), and decision curve analysis were used to assess the performance of the SORG ML algorithms in the validation cohort.


      Overall, 176 patients underwent surgery for spinal metastatic disease, of which 44 (22.7%) experienced 90-day mortality and 99 (56.2%) experienced 1-year mortality. The validation cohort differed significantly from the developmental cohort on primary tumor histology, metastatic tumor burden, previous systemic therapy, overall comorbidity burden, and preoperative laboratory characteristics. Despite these differences, the SORG ML algorithms generalized well to the validation cohort on discrimination (c-statistic 0.75–0.81 for 90-day mortality and 0.77–0.78 for 1-year mortality), calibration, Brier score, and decision curve analysis.


      Initial results from external validation of the SORG ML 90-day and 1-year algorithms for survival prediction in spinal metastatic disease suggest potential utility of these digital decision aids in clinical practice. Further studies are needed to validate or refute these algorithms in large patient samples from prospective, international, multi-institutional trials.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to The Spine Journal
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Lawton AJ
        • Lee KA
        • Cheville AL
        • Ferrone ML
        • Rades D
        • Balboni TA
        • et al.
        Assessment and management of patients with metastatic spinal cord compression: a multidisciplinary review.
        J Clin Oncol. 2018; 37: 61-71
        • Loblaw DA
        • Laperriere NJ
        • Mackillop WJ
        A population-based study of malignant spinal cord compression in Ontario.
        Clin Oncol. 2003; 15: 211-217
        • Klimo Jr., P
        • Schmidt MH.
        Surgical management of spinal metastases.
        Oncologist. 2004; 9: 188-196
        • Verlaan JJ
        • Choi D
        • Versteeg A
        • Albert T
        • Arts M
        • Balabaud L
        • et al.
        Characteristics of patients who survived < 3 months or >2 years after surgery for spinal metastases: can we avoid inappropriate patient selection?.
        J Clin Oncol. 2016; 34: 3054-3061
        • Prasad D
        • Schiff D
        Malignant spinal-cord compression.
        Lancet Oncol. 2005; 6: 15-24
        • Patchell RA
        • Tibbs PA
        • Regine WF
        • Payne R
        • Saris S
        • Kryscio RJ
        • et al.
        Direct decompressive surgical resection in the treatment of spinal cord compression caused by metastatic cancer: a randomised trial.
        Lancet. 2005; 366: 643-648
        • Kelly ML
        • Kshettry VR
        • Rosenbaum BP
        • Seicean A
        • Weil RJ
        Effect of a randomized controlled trial on the surgical treatment of spinal metastasis, 2000 through 2010: a population-based cohort study.
        Cancer. 2014; 120: 901-908
        • Nater A
        • Tetreault LA
        • Kopjar B
        • Arnold PM
        • Dekutoski MB
        • Finkelstein JA
        • et al.
        Predictive factors of survival in a surgical series of metastatic epidural spinal cord compression and complete external validation of 8 multivariate models of survival in a prospective North American multicenter study.
        Cancer. 2018; 124: 3536-3550
        • Karhade AV
        • Thio QCBS
        • Ogink PT
        • Bono CM
        • Ferrone ML
        • Oh KS
        • et al.
        Predicting 90-day and 1-year mortality in spinal metastatic disease: development and internal validation.
        Neurosurgery. 2019; 85: E671-E681
        • Collins GS
        • Reitsma JB
        • Altman DG
        • Moons KG
        Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement.
        BMC Med. 2015; 13: 1
        • Katagiri H
        • Okada R
        • Takagi T
        • Takahashi M
        • Murata H
        • Harada H
        • et al.
        New prognostic factors and scoring system for patients with skeletal metastasis.
        Cancer Med. 2014; 3: 1359-1367
        • Quan H
        • Li B
        • Couris CM
        • Fushimi K
        • Graham P
        • Hider P
        • et al.
        Updating and validating the Charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries.
        Am J Epidemiol. 2011; 173: 676-682
        • Stekhoven DJ
        • Bühlmann P
        MissForest—non-parametric missing value imputation for mixed-type data.
        Bioinformatics. 2011; 28: 112-118
        • Steyerberg EW
        • Vergouwe Y
        Towards better clinical prediction models: seven steps for development and an ABCD for validation.
        Eur Heart J. 2014; 35: 1925-1931
        • Brier GW
        Verification of forecasts expressed in terms of probability.
        Mon Weather Rev. 1950; 78: 1-3
        • Vickers AJ
        • Elkin EB
        Decision curve analysis: a novel method for evaluating prediction models.
        Med Decis Making. 2006; 26: 565-574
        • Harel R
        • Angelov L
        Spine metastases: current treatments and future directions.
        Eur J Cancer. 2010; 46: 2696-2707
        • Barzilai O
        • Laufer I
        • Yamada Y
        • Higginson DS
        • Schmitt AM
        • Lis E
        • et al.
        Integrating evidence-based medicine for treatment of spinal metastases into a decision framework: neurologic, oncologic, mechanicals stability, and systemic disease.
        J Clin Oncol. 2017; 35: 2419-2427
        • Barzilai O
        • Fisher CG
        • Bilsky MH
        State of the art treatment of spinal metastatic disease.
        Neurosurgery. 2018; 82: 757-769
        • Ahmed AK
        • Goodwin CR
        • Heravi A
        • Kim R
        • Abu-Bonsrah N
        • Sankey E
        • et al.
        Predicting survival for metastatic spine disease: a comparison of nine scoring systems.
        Spine J. 2018; 18: 1804-1814
        • Tokuhashi Y
        • Matsuzaki H
        • Toriyama S
        • Kawano H
        • Ohsaka S
        Scoring system for the preoperative evaluation of metastatic spine tumor prognosis.
        Spine. 1990; 15: 1110-1113
        • Tokuhashi Y
        • Matsuzaki H
        • Oda H
        • Oshima M
        • Ryu J
        A revised scoring system for preoperative evaluation of metastatic spine tumor prognosis.
        Spine. 2005; 30: 2186-2191
        • Tomita K
        • Kawahara N
        • Kobayashi T
        • Yoshida A
        • Murakami H
        • Akamaru T
        Surgical strategy for spinal metastases.
        Spine. 2001; 26: 298-306
        • Bauer HC
        • Wedin R
        Survival after surgery for spinal and extremity metastases: prognostication in 241 patients.
        Acta Orthop Scand. 1995; 66: 143-146
        • Katagiri H
        • Takahashi M
        • Wakai K
        • Sugiura H
        • Kataoka T
        • Nakanishi K
        Prognostic factors and a scoring system for patients with skeletal metastasis.
        J Bone Joint Surg Br Vol. 2005; 87: 698-703
        • van der Linden YM
        • Dijkstra SP
        • Vonk EJ
        • Marijnen CA
        • Leer JW
        Prediction of survival in patients with metastases in the spinal column: results based on a randomized trial of radiotherapy.
        Cancer. 2005; 103: 320-328
        • Paulino Pereira NR
        • Janssen SJ
        • van Dijk E
        • Harris MB
        • Hornicek FJ
        • Ferrone ML
        • et al.
        Development of a prognostic survival algorithm for patients with metastatic spine disease.
        J Bone Joint Surg Am Vol. 2016; 98: 1767-1776
        • Steyerberg EW
        • Vickers AJ
        • Cook NR
        • Gerds T
        • Gonen M
        • Obuchowski N
        • et al.
        Assessing the performance of prediction models: a framework for traditional and novel measures.
        Epidemiology. 2010; 21: 128-138
        • Debray TP
        • Vergouwe Y
        • Koffijberg H
        • Nieboer D
        • Steyerberg EW
        • Moons KG
        A new framework to enhance the interpretation of external validation studies of clinical prediction models.
        J Clin Epidemiol. 2015; 68: 279-289
        • Steyerberg EW
        • Harrell Jr., FE
        Prediction models need appropriate internal, internal-external, and external validation.
        J Clin Epidemiol. 2016; 69: 245-247
        • Steyerberg EW
        • Uno H
        • Ioannidis JPA
        • van Calster B
        Poor performance of clinical prediction models: the harm of commonly applied methods.
        J Clin Epidemiol. 2018; 98: 133-143
        • Van Calster B
        • Nieboer D
        • Vergouwe Y
        • De Cock B
        • Pencina MJ
        • Steyerberg EW
        A calibration hierarchy for risk models was defined: from utopia to empirical data.
        J Clin Epidemiol. 2016; 74: 167-176