Advertisement

Fostering reproducibility and generalizability in machine learning for clinical prediction modeling in spine surgery

  • Author Footnotes
    † Hung-Kuan Yen and Jiun-Jen Yang have equal contribution as co-first authors.
    Hung-Kuan Yen
    Footnotes
    † Hung-Kuan Yen and Jiun-Jen Yang have equal contribution as co-first authors.
    Affiliations
    Department of Orthopedic Surgery, National Taiwan University Hospital, Taipei City, Taiwan

    Department of Medical Education, National Taiwan University Hospital, Hsin-Chu Branch, Hsin-Chu City, Taiwan
    Search for articles by this author
  • Author Footnotes
    † Hung-Kuan Yen and Jiun-Jen Yang have equal contribution as co-first authors.
    Jiun-Jen Yang
    Footnotes
    † Hung-Kuan Yen and Jiun-Jen Yang have equal contribution as co-first authors.
    Affiliations
    Department of Orthopedic Surgery, National Taiwan University Hospital, Taipei City, Taiwan
    Search for articles by this author
  • Olivier Q. Groot
    Affiliations
    Department of Orthopedics Surgery, Massachusetts General Hospital

    Department of Orthopedic, University Medical Center Utrecht, Utrecht, The Netherlands
    Search for articles by this author
  • Mao-Hsu Yen
    Affiliations
    Department of Computer Science and Engineering, National Taiwan Ocean University, Keelung, Taiwan
    Search for articles by this author
  • Ming-Hsiao Hu
    Correspondence
    Corresponding author. Department of Orthopedics, College of Medicine, National Taiwan University, No1, Jen-Ai Rd, Taipei, Taiwan.
    Affiliations
    Department of Orthopedic Surgery, National Taiwan University Hospital, Taipei City, Taiwan

    Department of Orthopedics, College of Medicine, National Taiwan University , No1, Jen-Ai Rd, Taipei, Taiwan
    Search for articles by this author
  • Author Footnotes
    † Hung-Kuan Yen and Jiun-Jen Yang have equal contribution as co-first authors.
Published:November 03, 2022DOI:https://doi.org/10.1016/j.spinee.2022.10.011
      Current survival-prediction models (SPMs) were designed using historical data to predict future events. Over a dozen of SPMs exists in spine surgery, such as predicting survival in patients with spinal metastases or chordoma [
      • Karhade AV
      • Thio Q
      • Ogink PT
      • Bono CM
      • Ferrone ML
      • Oh KS
      • et al.
      Predicting 90-day and 1-year mortality in spinal metastatic disease: development and internal validation.
      ,
      • Ahmed AK
      • Goodwin CR
      • Heravi A
      • Kim R
      • Abu-Bonsrah N
      • Sankey E
      • et al.
      Predicting survival for metastatic spine disease: a comparison of nine scoring systems.
      ,
      • Karhade AV
      • Thio Q
      • Ogink P
      • Kim J
      • Lozano-Calderon S
      • Raskin K
      • et al.
      Development of machine learning algorithms for prediction of 5-year spinal chordoma survival.
      ,
      • Karhade AV
      • Fenn B
      • Groot OQ
      • Shah AA
      • Yen HK
      • Bilsky MH
      • et al.
      Development and external validation of predictive algorithms for 6-week mortality in spinal metastasis using 4304 patients from 5 institutions.
      ,
      • Yang JJ
      • Chen CW
      • Fourman MS
      • Bongers MER
      • Karhade AV
      • Groot OQ
      • et al.
      International external validation of the SORG machine learning algorithms for predicting 90-day and 1-year survival of patients with spine metastases using a Taiwanese cohort.
      ]. It cannot be overemphasized the importance of a robust methodology to construct a reliable SPM. However, random splitting the training and validation datasets, which was widely adopted as mentioned by Dr Azad et al., might not always be an ideal strategy in all clinical settings [
      • Azad TD
      • Ehresman J
      • Ahmed AK
      • Staartjes VE
      • Lubelski D
      • Stienen MN
      • et al.
      Fostering reproducibility and generalizability in machine learning for clinical prediction modeling in spine surgery.
      ], especially in settings where medical treatment is evolving in a fast pace. For example, with the rapid progress of antineoplastic agents, patients with spinal metastasis had a longer life-expectancy nowadays [
      • Cao B
      • Bray F
      • Beltran-Sanchez H
      • Ginsburg O
      • Soneji S
      • Soerjomataram I.
      Benchmarking life expectancy and cancer mortality: global comparison with cardiovascular disease 1981-2010.
      ]. This improvement poses a challenge for developing a SPM for such patients since it could be outdated in the near future. Therefore, we argue that a temporal splitting should also be considered in clinical situations with significant medical advances as it can help the SPM “learn” the progressing trend [
      • Ahmed NK
      • Atiya AF
      • Gayar NE
      • El-Shishiny H
      An empirical comparison of machine learning models for time series forecasting.
      ]. To validate this hypothesis, we tested how SPMs perform using both splitting methods in a database of 2,578 patients with spinal metastases.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to The Spine Journal
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Karhade AV
        • Thio Q
        • Ogink PT
        • Bono CM
        • Ferrone ML
        • Oh KS
        • et al.
        Predicting 90-day and 1-year mortality in spinal metastatic disease: development and internal validation.
        Neurosurgery. 2019; 85: E671-EE81
        • Ahmed AK
        • Goodwin CR
        • Heravi A
        • Kim R
        • Abu-Bonsrah N
        • Sankey E
        • et al.
        Predicting survival for metastatic spine disease: a comparison of nine scoring systems.
        Spine J. 2018; 18: 1804-1814
        • Karhade AV
        • Thio Q
        • Ogink P
        • Kim J
        • Lozano-Calderon S
        • Raskin K
        • et al.
        Development of machine learning algorithms for prediction of 5-year spinal chordoma survival.
        World Neurosurg. 2018; 119: e842-e8e7
        • Karhade AV
        • Fenn B
        • Groot OQ
        • Shah AA
        • Yen HK
        • Bilsky MH
        • et al.
        Development and external validation of predictive algorithms for 6-week mortality in spinal metastasis using 4304 patients from 5 institutions.
        Spine J. 2022; https://doi.org/10.1016/j.spinee.2022.07.089
        • Yang JJ
        • Chen CW
        • Fourman MS
        • Bongers MER
        • Karhade AV
        • Groot OQ
        • et al.
        International external validation of the SORG machine learning algorithms for predicting 90-day and 1-year survival of patients with spine metastases using a Taiwanese cohort.
        Spine J. 2021; 21: 1670-1678
        • Azad TD
        • Ehresman J
        • Ahmed AK
        • Staartjes VE
        • Lubelski D
        • Stienen MN
        • et al.
        Fostering reproducibility and generalizability in machine learning for clinical prediction modeling in spine surgery.
        Spine J. 2021; 21: 1610-1616
        • Cao B
        • Bray F
        • Beltran-Sanchez H
        • Ginsburg O
        • Soneji S
        • Soerjomataram I.
        Benchmarking life expectancy and cancer mortality: global comparison with cardiovascular disease 1981-2010.
        BMJ. 2017; 357: j2765
        • Ahmed NK
        • Atiya AF
        • Gayar NE
        • El-Shishiny H
        An empirical comparison of machine learning models for time series forecasting.
        Econometric Rev. 2010; 29: 594-621
        • Yen H-K
        • Hu M-H
        • Zijlstra H
        • Groot OQ
        • Hsieh H-C
        • Yang J-J
        • et al.
        Prognostic significance of lab data and performance comparison by validating survival prediction models for patients with spinal metastases after radiotherapy.
        Radiat Oncol. 2022; 175: 159-166https://doi.org/10.1016/j.radonc.2022.08.029
        • Katagiri H
        • Okada R
        • Takagi T
        • Takahashi M
        • Murata H
        • Harada H
        • et al.
        New prognostic factors and scoring system for patients with skeletal metastasis.
        Cancer Med. 2014; 3: 1359-1367
        • Riley RD
        • Ensor J
        • Snell KI
        • Debray TP
        • Altman DG
        • Moons KG
        • et al.
        External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges.
        BMJ. 2016; 22: 353https://doi.org/10.1136/bmj.i3140
        • Orenday-Barraza JM
        • Cavagnaro MJ
        • Avila MJ
        • Strouse IM
        • Dowell A
        • Kisana H
        • et al.
        10-year trends in the surgical management of patients with spinal metastases: a scoping review.
        World Neurosurg. 2022; 157: 170-186https://doi.org/10.1016/j.wneu.2021.10.086
        • McNabb-Baltar J
        • Chang MS
        • Suleiman SL
        • Banks PA
        • de Silva DPS.
        A time trend analysis of CT and MRI scan imaging in acute pancreatitis patients presenting to US emergency departments.
        Am J Emerg Med. 2018; 36: 1709-1710
        • Ohana O
        • Soffer S
        • Zimlichman E
        • Klang E.
        Overuse of CT and MRI in paediatric emergency departments.
        Br J Radiol. 2018; 9120170434