Calculators of revascularization risk: peering into the crystal ball

Corresponding Author:

Jaap N Hamburger
Vancouver General Hospital
Vancouver BC, Canada
Tel: +1 604 875 5523
Fax: +1 604 875 5544
E-mail: jaap.hamburger@gmail.com

Abstract

Keywords

30-day mortality, coronary artery bypass graft surgery, percutaneous coronary intervention, revascularization risk models

The clinical importance of an accurate coronary revascularization risk assessment cannot be overemphasized and is manifold. Foremost, it guides the treating physician to the optimal treatment strategy for the individual patient and allows patients to make informed decisions regarding their healthcare. The recent emergence of percutaneous coronary intervention (PCI) risk calculators that provide 30-day predicted mortality data have been instrumental in facilitating optimal treatment selection, enabling the direct comparison of PCI and surgical risk. In addition, accurate risk prediction models can ‘level the playing field’ of performance metrics to accurately audit clinical results and are of particular use in the current climate of increased medical scrutiny. They can also be used to facilitate the interpretation of observational research data.

Risk model performance metrics

The utility of a risk model is assessed by two features: calibration and discrimination. Calibration is the accuracy of the model for predicting risk in a group of patients and is assessed by comparing average observed and predicted values within ten equal-sized subgroups arranged in increasing order of patient risk, and then applying the Hosmer–Lemeshow statistical test to assess ‘goodness of fit’. Discrimination refers to the model’s ability to distinguish between low- and high-risk patients, and is measured using the c-index, which represents the ‘area under the receiver-operating characteristic curve’. If the area under the receiver-operating characteristic curve is 0.5, the model does not discriminate at all; good discrimination begins at 0.7 and if the area is 1.0, the model is no longer a risk-prediction model but acts as a crystal ball, always correctly predicting future events.

Surgical risk models

Registries and statistical models have been available for many years to predict adverse cardiothoracic surgical outcomes, such as mortality [1–5]. These scoring systems often assess 30-day mortality, rather than in-hospital mortality, to meaningfully estimate overall surgical mortality related to the revascularization procedure. The first surgical risk-prediction tool was the Parsonnet score [1], but this was subjective and often overestimated surgical risk [6]. The most widely used risk calculators available today to assess surgical revascularization risk are the European System for Cardiac Operative Risk Evaluation (EuroSCORE and logistic EuroSCORE) and the North American Society of Thoracic Surgeons (STS) score, although a shortcoming may be their longevity.

▪ European perspective: EuroSCORE & logistic EuroSCORE

The EuroSCORE was developed to improve the prediction of early mortality following cardiothoracic surgery. In a developmental subset of 13,302 patients from the EuroSCORE database (circa 1995), several objective, credible, obtainable and difficult to falsify variables linked with surgical outcome were weighted according to multiple regression analysis in order to construct an additive score. The overall 30-day mortality rate for the cohort was 4.7%, and the EuroSCORE achieved a 30-day mortality discrimination c-index of 0.79 for the developmental subset and 0.76 for the subsequent 1479-patient validation group [3]. Although the simple, additive EuroSCORE enabled risk calculation at the bedside with excellent predictive ability across different populations [7,8], it was noted that it underestimated risk in high-risk patients and, therefore, the full, logistic regression version of the score was published to improve its predictive accuracy (available at [101]) [9,10]. Recent evidence from national databases has called into question the accuracy of the logistic EuroSCORE [11]. Technological advances and improvements in surgical technique have resulted in a 50% decrease in actual surgical mortality, despite gradual worsening of the risk profile of patients, resulting in an overestimation of the predicted risk by the logistic EuroSCORE [12]. Therefore, the logistic EuroSCORE requires recalibration from a new, contemporaneous patient database in order to improve the accuracy of the model. In the meantime, the predicted risk should be corrected for local surgical performance.

▪ North American perspective: Society of Thoracic Surgeons score

Similarly, on the other side of the Atlantic, data entered from 503,478 coronary artery bypass graft (CABG) procedures performed between 1997 and 1999 in the STS National Adult Cardiac Surgical Database were used to develop a risk-adjusted mortality and morbidity model (available at [102]) [13]. The overall 30-day mortality in this database was 3.05% and the STS score had a discrimination c-index of 0.78 and was accurate.

PCI risk models

In recent years, the acceptance of PCI as a viable alternative to established surgical revascularization has required supportive data, particularly in the setting of complex anatomical and clinical cases. In addition, similar to cardiothoracic surgery, procedural risk is required to meaningfully audit individual center and operator results for clinical governance. The development of PCI risk calculators, now available online, focus on either anatomical parameters alone (SYNTAX – SX score [14]) or anatomical and clinical parameters together (British Columbia [BC] PCI score [15] and National Cardiovascular Data Registry [NCDR] score [16]) to predict risk. The limited SX score only predicts repeat revascularization rates, whereas the more comprehensive BC PCI and NCDR scores provide a more robust prediction of 30-day mortality. The latter enables the direct comparison with surgical 30-day mortality, enabling physicians to easily select the optimal revascularization strategy for their patients.

▪ Coronary anatomy assessment: SX score

This widely publicized SX scoring system was developed as part of the SYNTAX trial, which compared surgical treatment and PCI for multivessel and left main stem disease [17]. The SX score (available at [103]) was designed to predict PCI outcomes from coronary stenosis and anatomy characteristics derived from the diagnostic coronary angiogram alone [14]. The trial demonstrated that in the 903 patients randomized to the PCI arm of the study, the raw SX score was predictive of major adverse cardiac and cerebrovascular events at 1 year, although this was driven primarily by repeat revascularization. In only the subgroup with the highest SX score (>33) was there a significant increase in 1-year major adverse cardiac and cerebrovascular events compared with surgical revascularization (23.4 vs 10.9%; p < 0.0001). There was only a modestly increased trend in the incidence of the more robust combined end point of death, stroke and myocardial infarction in this high SX score group compared with CABG (11.9 vs 7.6%; p = 0.08). In those with a low or intermediate SX score, both surgical and PCI revascularization had comparable outcomes and, therefore, the SX score did not inform regarding revascularization choice in these patient groups. The intra- and inter-observer variability of the SX score caluculated by both clinicians and core laboratory is modest at best, which further limits its predictive value [18,19].

To adequately compare CABG and PCI results, more robust PCI risk-assessment models that include both anatomical and clinical data, validated in large, diverse populations, predicting clinically meaningful end points such as mortality, are required to provide data for direct comparison with those of surgical risk.

▪ Clinical & anatomical assessment: BC PCI & NCDR scores

A major limitation of several early clinical PCI risk scores was that they did not account for mortality beyond hospital discharge and, therefore, ignored important late clinical events such as subacute stent thrombosis [20–22]. Comparison with the extensively validated risk scores for surgical revascularization that predict 30-day postoperative mortality was also not possible. Furthermore, the earlier risk scores predate the routine use of stents and adjuvant antithrombotic therapy, and do not necessarily reflect contemporary clinical practice.

The BC PCI score (available at [104]) was the first contemporary PCI risk calculator to predict 30-day mortality [15]. It was developed from the BC Cardiac Registry database, which recorded the outcome and clinical details of 32,899 patients undergoing PCI in British Columbia (Canada). The cohort included 26,350 PCI procedures between 2000 and 2004 (the training cohort), and after multivariate logistic regression analysis to select variables associated with mortality, a statistical model (risk calculator) was designed and validated in 6549 PCI procedures performed in 2005. A number of univariate anatomical and clinical predictors of mortality were discovered, but only parameters that were available to the physician before PCI were included in the risk calculator (Figure 1). The actual 30-day mortality for the overall population was 1.5%. The BC PCI score was highly accurate and discriminative, predicting 30-day mortality with a c-index of 0.90 in the training dataset and 0.91 in the validation dataset. Subsequent external validation by the Massachusetts Data Analysis Center (Mass-DAC) confirmed the discriminative power of the BC PCI score (c-index: 0.87) in predicting 30-day mortality in 35,937 patients undergoing emergent and elective PCI in Massachusetts (USA) [23].

Figure 1. British Columbia percutaneous coronary intervention calculator.
Available at [104].

The NCDR similarly developed and prospectively validated an updated risk score in 588,398 contemporaneously PCI-treated patients observed between 2004 and 2007 [16]. They confirmed the importance of clinical variables in calculating PCI risk by identifying eight preprocedural clinical factors that were associated with in-hospital mortality: age, shock, prior congestive heart failure, peripheral vascular disease, chronic lung disease, renal impairment, New York Heart Association functional class 4 and PCI status. Angiographic variables provided only modest incremental information. The overall NCDR model and simplified NCDR risk score (based on the eight clinical factors alone) demonstrated excellent discrimination of in-hospital mortality (c-index: 0.93 and 0.91, respectively). The robustness of the NCDR model to predict 30-day mortality was confirmed in a large Medicare population with highly respectable c-index values of 0.86 and 0.83, respectively.

The clinical utility and superiority of comparing the predicted 30-day mortalities of PCI (BC PCI score) and surgical revascularization (logistic EuroSCORE) in the same patient to guide optimal therapy with a strategy guided by anatomical assessment (SX score) alone have recently been highlighted [24]. Comparing the BC PCI score and logistic EuroSCORE provided the most expedient assessment of revascularization risk, largely owing to simple, less subjective binary data entry into online calculators. The SX score incorrectly or incompletely assigned the optimal revascularization strategy in a significant proportion of patients (27%), again confirming the inferiority of revascularization guided by anatomical considerations alone.

Future perspective

We are now entering an era in which well-validated revascularization risk scores that calculate 30-day mortality can be easily compared to determine the optimal coronary revascularization strategy. This obviates the need for complex, subjective, labor-intense anatomical risk-assessment tools. Regular revalidation and recalibration of the currently available risk score calculators will be needed in the future (as is being performed for the logistic EuroSCORE) to ensure that they accurately reflect the risk of contemporaneous revascularization practices applied to future and almost certainly older and higher risk patient populations. This will ensure the optimal treatment selection of future patients with coronary artery disease.

Financial & competing interests disclosure

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

No writing assistance was utilized in the production of this manuscript.

Executive summary

Importance of predicting revascularization risk

▪ Predicting revascularization risk enables the correct treatment decisions to be made, allows valid informed consent and accurate patient counseling, as well as improving performance metric analysis and the auditing of results.

Risk performance metrics

▪ The key measure of risk calculation performance is calibration (accuracy) – comparing observed versus predicted 30-day mortality over a range of values and discrimination – in identifying high- and low-risk groups.

Surgical risk models

▪ The logistic EuroSCORE and Society of Thoracic Surgeons scores are well-established and extensively validated risk-prediction tools that predict 30-day surgical mortality.

Percutaneous coronary intervention risk models

▪ Initially, percutaneous coronary intervention (PCI) risk was limited to short-term event prediction that ignored important late events and prevented direct comparison with surgical risk-prediction tools.

▪ The British Columbia (BC) PCI score was the first developed PCI-risk calculator to accurately predict 30-day mortality risk, and it was later followed by the large National Cardiovascular Data Registry risk calculator.

▪ Both the BC PCI and National Cardiovascular Data Registry risk models confirmed that both anatomical and clinical data are required to provide accurate and discriminatory 30-day mortality risk prediction.

Future

▪ Validated PCI-risk calculators, such as the BC PCI score, are now available that allow the comparison of predicted 30-day mortality with coronary artery bypass graft surgical data, to guide optimal revascularization.

References

Papers of special note have been highlighted as:

▪ of interest

▪▪ of considerable interest

1. Parsonnet V, Dean D, Bernstein AD: A method of uniform stratification of risk for evaluating the results of surgery in acquired adult heart disease. Circulation 79(6 Pt 2), I3–I12 (1989).
2. Roques F, Nashef SA, Michel P et al.: Risk factors and outcome in European cardiac surgery: analysis of the EuroSCORE multinational database of 19030 patients. Eur. J. Cardiothorac. Surg. 15(6), 816–822 (1999).
3. Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R: European system for cardiac operative risk evaluation (EuroSCORE). Eur. J. Cardiothorac. Surg. 16(1), 9–13 (1999).

▪▪ Landmark paper on cardiothoracic surgical 30-day mortality risk prediction.

4. Hattler BG, Madia C, Johnson C et al.: Risk stratification using the Society of Thoracic Surgeons Program. Ann. Thorac. Surg. 58(5), 1348–1352 (1994).
5. Grover FL, Shroyer AL, Hammermeister K et al.: A decade’s experience with quality improvement in cardiac surgery using the Veterans Affairs and Society of Thoracic Surgeons national databases. Ann. Surg. 234(4), 464–472 (2001).
6. Wynne-Jones K, Jackson M, Grotte G, Bridgewater B: Limitations of the Parsonnet score for measuring risk stratified mortality in the north west of England. The North West Regional Cardiac Surgery Audit Steering Group. Heart 84(1), 71–78 (2000).
7. Nashef SA, Roques F, Hammill BG et al.: Validation of European System for Cardiac Operative Risk Evaluation (EuroSCORE) in North American cardiac surgery. Eur. J. Cardiothorac. Surg. 22(1), 101–105 (2002).

▪ External validation of the European System for Cardiac Operative Risk Evaluation (EuroSCORE) in a North American cohort.

8. Roques F, Nashef SA, Michel P et al.: Does EuroSCORE work in individual European countries? Eur. J. Cardiothorac. Surg. 18(1), 27–30 (2000).
9. Michel P, Roques F, Nashef SA: Logistic or additive EuroSCORE for high-risk patients? Eur. J. Cardiothorac. Surg. 23(5), 684–687 (2003).
10. Roques F, Michel P, Goldstone AR, Nashef SA: The logistic EuroSCORE. Eur. Heart J. 24(9), 881–882 (2003).
11. Gummert JF, Funkat A, Osswald B et al.: EuroSCORE overestimates the risk of cardiac surgery: results from the national registry of the German Society of Thoracic and Cardiovascular Surgery. Clin. Res. Cardiol. 98(6), 363–369 (2009).
12. Nashef SA: The new EuroSCORE project. Nowa skala EuroSCORE. Kardiol. Pol. 68(1), 128–129 (2010.
13. Shroyer AL, Coombs LP, Peterson ED et al.: The Society of Thoracic Surgeons: 30-day operative mortality and morbidity risk models. Ann. Thorac. Surg. 75(6), 1856–1864 (2003).

▪ Validation of the Society of Thoracic Surgeons score to predict 30-day cardiothoracic surgical mortality.

14. Sianos G, Morel MA, Kappetein AP et al.: The SYNTAX score: an angiographic tool grading the complexity of coronary artery disease. EuroIntervention 1(2), 219–227 (2005).
15. Hamburger JN, Walsh SJ, Khurana R et al.: Percutaneous coronary intervention and 30-day mortality: the British Columbia PCI risk score. Catheter Cardiovasc. Interv. 74(3), 377–385 (2009).

▪▪ First percutaneous coronary intervention risk calculator predicting 30‑day mortality.

16. Peterson ED, Dai D, DeLong ER et al.: Contemporary mortality risk prediction for percutaneous coronary intervention: results from 588,398 procedures in the National Cardiovascular Data Registry. J. Am. Coll. Cardiol. 55(18), 1923–1932 (2010).

▪ North American percutaneous coronary intervention risk-prediction tool validated in the National Cardiovascular Data Registry.

17. Serruys PW, Morice M-C, Kappetein AP et al.: Percutaneous coronary intervention versus coronary-artery bypass grafting for severe coronary artery disease. N. Engl. J. Med. 360(10), 961–972 (2009).
18. Garg S, Girasis C, Sarno G et al.: The SYNTAX score revisited: a reassessment of the SYNTAX score reproducibility. Catheter Cardiovasc. Interv. 75(6), 946–952 (2010).
19. Serruys PW, Onuma Y, Garg S et al.: Assessment of the SYNTAX score in the Syntax study. EuroIntervention 5(1), 50–56 (2009).
20. Wu C, Hannan EL, Walford G et al.: A risk score to predict in-hospital mortality for percutaneous coronary interventions. J. Am. Coll. Cardiol. 47(3), 654–660 (2006).
21. Resnic FS, Ohno-Machado L, Selwyn A, Simon DI, Popma JJ: Simplified risk score models accurately predict the risk of major in-hospital complications following percutaneous coronary intervention. Am. J. Cardiol. 88(1), 5–9 (2001).
22. Moscucci M, Kline-Rogers E, Share D et al.: Simple bedside additive tool for prediction of in-hospital mortality after percutaneous coronary interventions. Circulation 104(3), 263–268 (2001).
23. Khurana R, Normand ST, Silbaugh T et al.: Independent US Validation of the British Columbia PCI risk score. J. Am. Coll. Cardiol. 55(10A), 134 (2010).
24. Hoole SP, Hamburger J: Comparing procedural risks to select the optimal revascularization strategy – certainty in an uncertain anatomical world. Catheter Cardiovasc. Interv. 77(2), 313–314 (2011).

▪ Websites

101. EuroSCORE www.euroscore.org

102. The Society of Thoracic Surgeons: risk calculator www.sts.org/sections/stsnationaldatabase/riskcalculator

103. SYNTAX score www.syntaxscore.com

104. Vancouver Coastal Health: welcome to the PCI calculator www.canadascore.org