Original Article
External validation of the ProACS score for risk stratification of patients with acute coronary syndromes
Validação externa do score de risco ProACS para estratificação de risco de doentes com síndrome coronária aguda
Ana Teresa Timóteoa,, , Sílvia Aguiar Rosaa, Marta Afonso Nogueiraa, Adriana Belob, Rui Cruz Ferreiraa
a Serviço de Cardiologia, Hospital Santa Marta, Centro Hospitalar de Lisboa Central, Lisboa, Portugal
b Centro Nacional de Coleção de Dados em Cardiologia (CNCDC), Sociedade Portuguesa de Cardiologia, Coimbra, Portugal
Received 27 August 2015, Accepted 23 November 2015
Abstract
Introduction

The ProACS risk score is an early and simple risk stratification score developed for all-cause in-hospital mortality in acute coronary syndromes (ACS) from a Portuguese nationwide ACS registry. Our center only recently participated in the registry and was not included in the cohort used for developing the score. Our objective was to perform an external validation of this risk score for short- and long-term follow-up.

Methods

Consecutive patients admitted to our center with ACS were included. Demographic and admission characteristics, as well as treatment and outcome data were collected. The ProACS risk score variables are age (≥72 years), systolic blood pressure (≤116 mmHg), Killip class (2/3 or 4) and ST-segment elevation. We calculated ProACS, Global Registry of Acute Coronary Events (GRACE) and Canada Acute Coronary Syndrome risk score (C-ACS) risk scores for each patient.

Results

A total of 3170 patients were included, with a mean age of 64±13 years, 62% with ST-segment elevation myocardial infarction. All-cause in-hospital mortality was 5.7% and 10.3% at one-year follow-up. The ProACS risk score showed good discriminative ability for all considered outcomes (area under the receiver operating characteristic curve >0.75) and a good fit, similar to C-ACS, but lower than the GRACE risk score and slightly lower than in the original development cohort. The ProACS risk score provided good differentiation between patients at low, intermediate and high mortality risk in both short- and long-term follow-up (p<0.001 for all comparisons).

Conclusions

The ProACS score is valid in external cohorts for risk stratification for ACS. It can be applied very early, at the first medical contact, but should subsequently be complemented by the GRACE risk score.

Resumo
Introdução

O score ProACS é um score simples e precoce desenvolvido para estratificação de risco de mortalidade hospitalar nas síndromes coronárias agudas (SCA), a partir de um registo nacional de SCA. O nosso centro só participou mais recentemente, pelo que os nossos doentes não foram incluídos na coorte de desenvolvimento do score. O nosso objetivo é o de validar externamente o score ProACS para mortalidade a curto e longo prazo.

Métodos

Foram incluídos doentes consecutivos admitidos no nosso centro por SCA. Obtiveram-se as características demográficas e da admissão, bem como o tratamento e seguimento. O score ProACS inclui as seguintes variáveis: idade ≥72 anos, pressão arterial sistólica ≤116mmHg, classe Killip na admissão e elevação do segmento ST. Para cada doente foi calculado o score ProACS, Global Registry of Acute Coronary Events (GRACE) e o C-ACS.

Resultados

Incluíram-se 3170 doentes, idade média de 64±13 anos, 62% com enfarte com elevação de ST. A mortalidade total hospitalar foi de 5,7 e 10,3% a um ano de seguimento. O score ProACS mostrou uma boa capacidade discriminativa (AUC>0,75) e boa calibração, semelhante ao C-ACS, mas inferior quando comparado com o score GRACE e ligeiramente inferior quando comparado com a coorte de desenvolvimento original. Permite uma boa diferenciação entre doentes com risco baixo, intermédio e alto quer para mortalidade a curto quer a longo prazo (p<0,001 para todas as comparações).

Conclusões

O score ProACS é um score válido em coortes externas. Pode ser aplicado muito precocemente no primeiro contacto médico, mas posteriormente deverá ser complementados pelo score GRACE.

Keywords
ProACS, Risk stratification, Acute coronary syndrome, Prognosis
Palavras-chave
ProACS, Estratificação de risco, Síndrome coronária aguda, Prognóstico
Introduction

The approach to acute coronary syndrome (ACS) has undergone many changes in recent years, particularly regarding invasive treatments, not only for ST-segment elevation myocardial infarction (STEMI) but also other ACS and in patients at high risk for cardiovascular events.1–4 Various scores have been developed over the last 20 years to stratify this risk. The first were developed on the basis of large multicenter clinical trials, such as the TIMI score; the disadvantage was they were not always sufficiently representative of real-world populations.5,6 More recently, the most widely used score has been the Global Registry of Acute Coronary Events (GRACE) score, which was based on a multinational ACS registry and hence is more representative. It also has better discriminative ability for both STEMI and other ACS.7,8 However, such scores are not always used in clinical practice, partly due to their complexity.9,10

Our group has previously demonstrated that risk scores can be simplified, albeit with a slight reduction in discriminative ability.11 The ProACS score was accordingly developed on the basis of the Portuguese Registry on Acute Coronary Syndromes (ProACS).12 The aim of the present study was to validate this score in a Portuguese population and to determine its short- and long-term predictive value.

Methods

The ProACS score was developed from the population of the ProACS registry, and included all patients enrolled between January 1, 2002 and October 31, 2014. ProACS is a continuous, prospective observational registry with 33 participating cardiology departments in Portugal. The inclusion criteria and variables have been published previously.13 The ProACS score was based on 37460 records of all-cause in-hospital mortality. Patients enrolled between January 1, 2002 and June 30, 2011 were randomly separated into the development (60%) and internal validation (40%) cohorts, while the 8586 patients enrolled from after July 1, 2011 made up the external validation cohort. Since our center only recently began to participate in the ProACS registry, our records were not included in the development cohort, and we therefore decided to perform an external validation of the ProACS score in a Portuguese population.

The score uses the following variables: age ≥72 years (2 points), systolic blood pressure ≤116 mmHg, Killip class at admission 2 or 3 and ST-segment elevation (1 point each), and Killip class 4 (3 points). Individuals who score 0 are at low risk of in-hospital mortality, 1 or 2 represents intermediate risk, and ≥3 points indicates high risk. The inclusion criteria and variables of our center's registry are similar to those of ProACS and also include demographic characteristics, risk factors, previous heart disease, characteristics at admission and treatment. We assessed all-cause in-hospital and one-year mortality.

All patients gave written informed consent for inclusion in the registry. The study is in accordance with the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis) recommendations for the reporting of studies validating prediction scores.14

Statistical analysis

Continuous variables were expressed as means and standard deviation and were compared using the Student's t test. Normal distribution was tested with the Kolmogorov-Smirnov test. Categorical variables were expressed as percentages and differences between groups were analyzed by the chi-square test or Fisher's test, as appropriate.

The ProACS score was calculated for each patient. For purposes of comparison the GRACE score and the Canada Acute Coronary Syndrome Risk Score (C-ACS) were also calculated.8,15 The latter was developed on the basis of the same assumptions as the ProACS score but in a Canadian population.

The predictive value of the ProACS score was analyzed by means of the area under the receiver operating characteristic (ROC) curve (AUC), and the model's goodness of fit was assessed by the Hosmer-Lemeshow test, in which a model has a good fit if its p value is not significant. AUCs for dependent and independent samples were compared using Henley and MacNeil's method.16,17

The statistical analysis was performed using IBM SPSS Statistics (version 19.0.0.2). A p value <0.05 was considered statistically significant.

Results

The external validation cohort of our center included 3170 patients with a mean age of 64±13 years, 71% male, and 62% with STEMI. Table 1 compares this cohort with the development cohort of the ProACS score. In general, patients in the validation cohort were younger and had more cardiovascular risk factors (except diabetes), fewer had a history of myocardial infarction and stroke, but more had undergone coronary angioplasty. STEMI was the most frequent form of ACS in the validation cohort, more often treated invasively and with a better profile at admission. In-hospital mortality was similar to the development cohort. All-cause mortality in the validation cohort was 7.2% at 30 days and 10.3% at one year.

Table 1.

Clinical characteristics of the development cohort (from the ProACS registry) and the validation cohort.

  Development cohort (n=17380)  External validation cohort (n=3170) 
Age (years)  66±13  64±13  <0.001 
Male (%)  69.9  70.6  0.438 
BMI (kg/m2)  27±4  27±4  1.000 
Previous history (%)
Hypertension  62.2  65.9  <0.001 
Smoking  24.0  37.1  <0.001 
Diabetes  27.0  26.1  0.300 
Dyslipidemia  45.2  50.7  <0.001 
MI  18.4  15.1  <0.001 
PCI  8.0  10.7  <0.001 
CABG  4.1  4.0  0.830 
Stroke/TIA  7.0  5.8  0.015 
PAD  3.1  3.5  0.255 
Admission
STEMI (%)  43.6  62.2  <0.001 
HR (bpm)  79±20  78±20  <0.001 
SBP (mmHg)  141±30  136±29  <0.001 
Killip class ≥2 (%)  18.9  12.6  <0.001 
CA (%)  68.9  87.5  <0.001 
PCI (%)  43.8  63.5  <0.001 
CABG (%)  1.4  1.6  0.425 
In-hospital mortality (%)  5.4  5.7  0.517 

BMI: body mass index; CA: coronary angiography; CABG: coronary artery bypass grafting; HR: heart rate; MI: myocardial infarction; PAD: peripheral arterial disease; PCI: percutaneous coronary intervention; SBP: systolic blood pressure; STEMI: ST-elevation myocardial infarction; TIA: transient ischemic attack.

In our population, the ProACS score showed good discriminative ability for short-, medium- and long-term mortality, with a good fit (Table 2). Its discriminative ability was similar to that of the C-ACS score but lower than the GRACE score. Applying the risk thresholds described also correctly identified individuals at low, intermediate and high risk for in-hospital mortality (Figure 1). The ProACS score had a similar predictive ability for STEMI and other ACS, in all cases comparable to that of the C-ACS score but lower than the GRACE score (Table 3).

Table 2.

Predictive ability and goodness of fit of the ProACS, GRACE and C-ACS risk scores for short-, medium- and long-term mortality.

Score  AUC (95% CI)  pa  p (HL) 
In-hospital mortality
ProACS  0.769 (0.732-0.805)  –  0.280 
GRACE  0.857 (0.829-0.886)  <0.001  0.263 
C-ACS  0.769 (0.731-0.806)  1.000  0.056 
30-day mortality
ProACS  0.755 (0.720-0.789)  –  0.389 
GRACE  0.829 (0.802-0.857)  <0.001  0.098 
C-ACS  0.748 (0.713-0.784)  0.670  0.136 
One-year mortality
ProACS  0.748 (0.719-0.777)  –  0.623 
GRACE  0.804 (0.779-0.829)  <0.001  0.633 
C-ACS  0.752 (0.722-0.782)  0.781  0.018 

AUC: area under the curve; C-ACS: Canada Acute Coronary Syndrome Risk Score; CI: confidence interval; GRACE: Global Registry of Acute Coronary Events; HL: Hosmer-Lemeshow test.

a

Comparison with the ProACS score.

Figure 1.
(0.1MB).

Observed in-hospital, 30-day and one-year mortality in groups stratified according to the ProACS risk score.

Table 3.

Comparison of the ProACS, GRACE and C-ACS risk scores for ST-elevation myocardial infarction and other acute coronary syndromes in terms of short-term (in-hospital) and long-term (one year) mortality.

    ProACS  GRACE  pa  C-ACS  pa 
In-hospital
STEMIAUC  0.77 (0.73-0.82)  0.87 (0.84-0.90)  0.002  0.77 (0.72-0.81)  0.816 
HL  0.247  0.540    0.800   
NSTE-ACSAUC  0.74 (0.67-0.80)  0.82 (0.75-0.88)  0.149  0.79 (0.73-0.84)  0.391 
HL  0.113  0.785    <0.001   
pb    0.435  0.208    0.669   
One year
STEMIAUC  0.76 (0.72-0.79)  0.82 (0.79-0.85)  0.009  0.74 (0.71-0.78)  0.694 
HL  0.231  0.289    0.885   
NSTE-ACSAUC  0.74 (0.69-0.79)  0.79 (0.74-0.83)  0.221  0.77 (0.72-0.81)  0.504 
HL  0.200  0.486    <0.001   
pb    0.661  0.244    0.513   

AUC: area under the curve with 95% confidence interval; C-ACS: Canada Acute Coronary Syndrome Risk Score; GRACE: Global Registry of Acute Coronary Events; HL: Hosmer-Lemeshow test; NSTE-ACS: non-ST-elevation acute coronary syndrome; STEMI: ST-elevation myocardial infarction.

a

Comparison with the ProACS score.

b

Comparison between types of acute coronary syndrome.

Discussion

In recent years, demonstration of the advantages of early invasive treatment of ACS has made effective risk stratification essential in order to identify individuals at high or intermediate risk who will benefit most from an invasive approach.1–4 While this stratification is less important in cases of STEMI, since primary revascularization is the first-line treatment for the majority of these patients, in non-ST-elevation ACS – a much more heterogeneous group in terms of risk – timely risk stratification is vital.1–4 Even in STEMI patients, transfer of those at higher risk to specialized centers is also desirable.

The TIMI score was one of the first to be developed and rapidly gained popularity due to its clear benefits.5,6 However, the early scores were based on multicenter randomized clinical trials with highly specific inclusion criteria and therefore did not truly represent real-world ACS populations; for example, very elderly patients and those with severe renal failure are usually excluded from such trials and were thus not considered in the development of the risk scores. The GRACE score,7,8 developed some years later, was based on an ACS registry and included new variables with a strong impact on prognosis, including renal function; it was shown to have better discriminative ability, and for this reason it came to replace the TIMI score. However, the improved discriminative ability of more recent scores came at the cost of a significant increase in complexity, which partly explains physicians’ reluctance to use them.9,10 Computer software has been developed to facilitate calculation of the scores, but even so, the fact that they rely on laboratory test results that may only be available some time after admission can delay their application. This is important, since the results of these scores can be decisive in the management of urgent patients.

Our group has previously demonstrated that risk scores can be simplified, albeit with a slight reduction in discriminative ability.11 However, in view of the limitations of our sample, particularly its small size, we did not propose a new score. A Canadian group recently developed a new score, the C-ACS, based on large Canadian registries,15 in which they identified age, Killip class, systolic blood pressure and heart rate at admission as the main variables. Although the predictive ability of this score was lower than the GRACE score, it was considered acceptable and close to that of the TIMI score.5,6 Given the differences between North American and European (particularly southern European) populations, we decided to develop a score based on a representative sample of Portuguese ACS patients, and the ProACS registry was an obvious source of data for this purpose. The result was the ProACS score, which includes very simple variables (age, Killip class on admission, systolic blood pressure and ST elevation).12 The continuous variables were dichotomized to simplify calculation. The score enables rapid stratification of ACS patients, even by health care professionals without advanced medical training, in pre-hospital settings or emergency departments. The ProACS score showed similar discriminative ability to the C-ACS score, but lower than the GRACE score, and with an AUC of ≥0.75.

In our external validation cohort, the ProACS score showed reasonably good predictive ability, not only in the short term as demonstrated by the original validation, but also in the medium and long term. The fact that its predictive ability is lower than that of the GRACE score does not hinder its application, since an AUC of ≥0.75 means it is valid for use at the first medical contact, when it may be important to decide whether to refer the patient to a center able to perform interventions 24/7 or to a less specialized unit. It would also be valuable in the context of STEMI, enabling some patients to be transferred directly to specialist tertiary centers. However, when full clinical and laboratory data are available the GRACE score should be applied, since it provides a complementary stratification that may be useful for the management of these patients, particularly in terms of medication or the possibility of early discharge.

Limitations

Our external validation cohort differs significantly from the population used to develop the score and also from the populations of other hospitals, particularly in the high proportion of patients with STEMI (>60%). For this reason our results may not apply to other populations, and further validation is required.

Conclusions

The ProACS score, developed on the basis of a representative Portuguese population, is a simple and easily applied risk score for stratification of ACS patients, and has been validated in external cohorts. It can be applied very early, at the first medical contact, but should subsequently be complemented by more effective scores, particularly the GRACE risk score.

Ethical disclosuresProtection of human and animal subjects

The authors declare that no experiments were performed on humans or animals for this study.

Confidentiality of data

The authors declare that they have followed the protocols of their work center on the publication of patient data.

Right to privacy and informed consent

The authors declare that no patient data appear in this article.

Conflicts of interest

The authors have no conflicts of interest to declare.

References
1
G. Steg,S.K. James,D. Atar
ESC Guidelines for the management of acute myocardial infarction in patients with ST-segment elevation
Eur Heart J, 33 (2012), pp. 2569-2619 http://dx.doi.org/10.1093/eurheartj/ehs215
2
C.W. Hamm,J.P. Bassand,S. Agewall
ESC Guidelines for the management of acute coronary syndromes in patients without persistent ST-segment elevation
Eur Heart J, 32 (2011), pp. 2999-3054 http://dx.doi.org/10.1093/eurheartj/ehr236
3
E.A. Amsterdam,N.K. Wenger,R.G. Brindis
2014 AHA/ACC Guidelines for the management of patients with non-ST-elevation acute coronary syndromes
Circulation, 130 (2014), pp. e344-e426 http://dx.doi.org/10.1161/CIR.0000000000000134
4
P.T. O’Gara,F.G. Kushner,D.D. Ascheim
2013 ACCF/AHA Guidelines for the management of ST-elevation myocardial infarction
Circulation, 127 (2013), pp. e362-e425 http://dx.doi.org/10.1161/CIR.0b013e3182742cf6
5
E.M. Antman,M. Cohen,P.J. Bernink
The TIMI risk score for unstable angina/non-ST-elevation MI: a method for prognostication and therapeutic decision making
JAMA, 284 (2000), pp. 835-842
6
D.A. Morrow,E.M. Antman,A. Charlesworth
TIMI risk score for ST-elevation myocardial infarction: a convenient, bedside, clinical score for risk assessment at presentation
Circulation, 102 (2000), pp. 2031-2037
7
C.B. Granger,R.J. Goldberg,O. Dabbous
Predictors of hospital mortality in the Global Registry of Acute Coronary Events
Arch Intern Med, 163 (2003), pp. 2345-2353 http://dx.doi.org/10.1001/archinte.163.19.2345
8
K.S. Pieper,J.M. Gore,G. Fitzgerald
Validity of a risk-prediction tool for hospital mortality: the Global Registry of Acute Coronary Events
Am Heart J, 157 (2009), pp. 1097-1105 http://dx.doi.org/10.1016/j.ahj.2009.04.004
9
A.T. Yan,R.T. Yan,T. Huynh
Understanding physicians’ risk stratification of acute coronary syndromes: insights from the Canadian ACS II Registry
Arch Intern Med, 169 (2009), pp. 372-378 http://dx.doi.org/10.1001/archinternmed.2008.563
10
A.J. Bagnall,S.G. Goodman,K.A. Fox
Optimal medical therapy for non-ST-segment-elevation acute coronary syndromes: exploring why physicians do not prescribe evidence-based treatment and why patients discontinue medications after discharge
Cir Cardiovasc Qual Outcomes, 3 (2010), pp. 530-537
11
A.T. Timóteo,A.L. Papoila,J.P. Lopes
Is it possible to simplify risk stratification scores for patients with ST-segment elevation myocardial infarction undergoing primary angioplasty?
Rev Port Cardiol, 32 (2013), pp. 967-973 http://dx.doi.org/10.1016/j.repc.2013.06.006
12
A.T. Timóteo,S.A. Rosa,M.A. Nogueira
Score ProACS – um score simples e precoce para estratificação de risco em doentes com síndromes coronárias agudas [abstract]
Rev Port Cardiol, 34 (2015), pp. 49
13
J. Ferreira Santos,C. Aguiar,C. Gavina
Registo Nacional de Síndromes Coronárias Agudas: sete anos de actividade em Portugal
Rev Port Cardiol, 28 (2009), pp. 1465-1500
14
G.S. Collins,J.B. Reitsma,D.G. Altman
Transparent Reporting of a multivariable prediction Model for Individual Prognosis or Diagnosis (TRIPOD). The TRIPOD Statement
15
T. Huynh,S. Krouz,A. Yan
Canada Acute Coronary Syndrome Risk Score: a new risk score for early prognostication in acute coronary syndromes
Am Heart J, 166 (2013), pp. 58-63 http://dx.doi.org/10.1016/j.ahj.2013.03.023
16
J.A. Hanley,B.J. McNeil
A method of comparing the areas under receiver operating characteristic curves derived from the same cases
17
J.A. Hanley,B.J. McNeil
The meaning and use of the area under a receiver operating characteristic (ROC) curve

Please cite this article as: Timóteo A, Aguiar Rosa S, Nogueira MA, et al. Validação externa do score de risco ProACS para estratificação de risco de doentes com síndrome coronária aguda. Rev Port Cardiol. 2016;35:323–328.

WARNING Invalid argument supplied for foreach() (includes_ws/librerias/html/item.php[1203])
Copyright © 2016. Sociedade Portuguesa de Cardiologia

Metrics

  • Impact Factor: 0.827(2017)
  • 5-years Impact Factor: 0,775
  • SCImago Journal Rank (SJR):0,267
  • Source Normalized Impact per Paper (SNIP):0,419

Open Access

Creative Commons License
Portuguese Journal of Cardiology is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

+ info