OUTCOME PREDICTION FOR CRITICAL PATIENTS UNDER INTENSIVE CARE, USING BACKPROPAGATION NEURAL NETWORKS P. Felipe Jr.(1), R.M.E. Sabbatini(1), P.M. Carvalho-J£nior(1), R.E. Beseggio(2), R.G.G. Terzi(2) (1) N£cleo de Inform tica Biom‚dica; (2) Unidade de Terapia Intensiva do Hospital das Cl”nicas e Faculdade de Ciˆncias M‚dicas da Universidade Estadual de Campinas, Campinas, Brasil ABSTRACT Several scores have been designed to estimate death probability for patients admitted to Intensive Care Units, such as the APACHE and MPM systems, which are based on regression analysis. In the present work, we have studied the potential of a model of artificial neural network, the three-layer perceptron with backpropagation learning rule, to perform this task. Training and testing data were derived from a Brazilian database which was previously used for calculating APACHE scores. All networks were able to reach convergence with a small global prediction error. Maximum percentages of 75% correct predictions in the test dataset and 99.6 % in the training dataset, were achieved. Maximum sensitivity and specificity were 60% and 80%, respectively. We conclude that the neural network approach has worked well for outcome prognosis in a highly "noisy" dataset, with a similar, if slightly lower performance than APACHE II, but with the advantage of deriving its parameters from a regional dataset instead from an universal model. INTRODUCTION In the last decades, hospital-based Medicine has been dramatically changed. According to Knaus [4], the initial emphasis on diagnosis has been replaced by complex multidisciplinary therapeutic approaches, particularly in advanced stages of disease, such as during intensive care. The increases in health care costs prompted the development of risk-assessment procedures, with the purpose of obtaining more precise evidence of its indications and benefits. Thus, several scores for assessing quantitatively the severity of disease and to perform outcome prediction have been developed, such as the APACHE (Acute Physiologic and Chronic Health Evaluation) [4], and MPM systems [5], which were designed to estimate death probability for patients admitted to Intensive Care Units (ICU). These systems are usually based on weighing parameters derived from logistical regression analyses, performed on extremely large databases, collected by international multicentrical studies, with the aim of achieving universal validation. An extensive evaluation of outcome prediction and observed mortality could be used as quality control tools in the ICU, because quality of medical and nursing care are considered as the single most important factor interfering with outcome [11]. However, despite the wide acceptance and good performance of such universal prognostic indices, regional differences in patient populations and in specific groups of pathologies might significantly influence its effectiveness and applicability. Thus, many studies have aimed at recalculating regression parameters for data taken from local databases, specific groups of patients, etc [11], increasing the difficulties associated to its day-to-day application. An alternative approach to outcome prediction might be using connectionist, or neural network (NN) systems, which are being increasingly used to represent and to processs information by means of networks of interconnected processing elements, similar to neurons. Several emerging global properties of connectionist systems, such as associative memory, distributed parallel processing, learning, etc.; have favored its applications in a large variety of tasks involving pattern classification and recognition [6]. Connectionist systems constitute a new and interesting paradigm for the area of Artificial Intelligence in Medicine, and have found applications in processing and interpretation of biological signals and images, decision support systems, etc., substituting advantageously many conventional techniques based on statistical classification and logical systems [8]. In particular, its quick deployment in situations were fuzzy pattern recognition based on small regional databases is required, make it attractive for the purpose of predicting fatal outcome in ICUs. In the present work, we have studied the potential of a particular model of artificial neural network, the three-layer perceptron with backpropagation learning rule, in this context. Training and testing data were derived from a Brazilian database which was previously used for the purpose of calculating APACHE scores. Although neural networks have been used for prognostic purposes in other medical areas [10], to our knowledge this is the first application in prognosis of patient outcome in critical medicine. MATERIAL AND METHODS Training and testing datasets The data records used for training and testing the neural networks were derived from the clinical database of the Intensive Care Unit, University Clinics Hospital of the State University of Campinas, encompassing about 3,000 patients interned from 1988 to 1992. From these, a systematic audit of the variables for completeness and reliability yielded a final sample of 382 patients, including all kinds of pathologies except for heart infarction and coronary disease (which are not used in the APACHE II system, either). In order to produce a valid set of examples for training, we performed an initial selection of variables (Table 1). Using the EPI- INFO statistical system, we calculated a contingency table and chi-square statistics for each variable against outcome (deceased/not deceased), which was also recorded into the original dBASE III database. In this case, the death of a patient was recorded as such only when it occurred while in the ICU, due to the difficulty of ascertaining late deaths. All variables associated with outcome with a probability less or equal than 0.05 were selected, as well a few other variables which, although not significantly associated to outcome, per se, might be a clinical factor (such as the patient's sex, mean arterial pressure, etc.). Once the variable set was selected, the variables were divided into binary (yes/no) and analog types. Binary variables were coded as 0 (zero) for yes/positive/true and 1 (one) for no/negative/false. Analog variables were recalculated according to the physiopathological criteria defined by Gustafson [1] and used by Knaus [4] in the APACHE II system, i.e., a monotonically ascending severity index was derived for each physiological variable whose extreme values were considered detrimental or predictive of increased severity of disease. Finally, all analog variables were scaled to a (0,1) real interval, in order to avoid unequal network saturation. Hardware and software Neural networks were built and simulated using the NEURONET shell connectionist system developed in our laboratory, running under MS- DOS in a 32-bit microcomputer (PC 30486 with 33 MHz clock, 8 Mbytes RAM, SuperVGA color video display and 200 Mbytes of hard disk storage). This program is described in detail elsewhere [9]. The neural network architecture was a non-linear, three-layer, feedforward perceptron, with one input layer, accepting the values of variables described in Table 1; one hidden layer and one output layer, containing just one processing element (outcome, expressed as a binary number in training and real number between zero and one in classification). In order to experiment with the effect o hidden layer size on classification performance, we trained and tested identical datasets with 5, 10, 15 and 20 elements. Weight initialization was random, between -0.3 and 0.3, and the activation function was a sigmoid. The learning rule was unmodified off-line delta backpropagation [7], with a fixed learning rate of 0.99. After preparing all datasets recorded onto dBASE III-compatible format, randomization was performed by a program, which divided the dataset into two: one containing the training dataset (282 cases), and another containing the test dataset (100 cases). Both were then converted to NEURONET-compatible input files, using the program DBF2NET. Following this, the shell program was used to create the network and to train it. At least 45.000 training cycles were used for each configuration. After training (network convergence), the net was used first to classify the examples used for it, and then the test dataset. Performance was evaluated by comparing desired versus obtained output in each case, and indices of sensitivity, specificity and predictivity were calculated on the basis of 2 x 2 contingency tables, with the EPI-INFO program. RESULTS All networks trained to convergence within 45.000 training cycles. Global error was below 3 in all cases. The distribution of network output for all cases gave evidence that the solutions converged to the near proximity of 0 and 1 (Fig.1). All networks were able to predict outcome, with rates varying between 97.7 % (net with 5 hidden elements) and 99.6 % (nets with 10 and 15 hidden elements), for the training dataset; and rates varying between 73% (nets with 10 and 15 hidden elements) and 75 % (nets with 5 and 20 hidden elements). Figure 2 shows sensitivity, specificity and predictivity indices for each network: - Sensitivity is the ability to detect deaths correctly. It was highest (60 %) for the 20-neurode net and lowest (39 %) for the 5-neurode net. - Specificity is the ability to detect non-fatal outcomes correctly. It was the opposite of sensitivity, but quite good: highest (88 %) for the 5-neurode net and lowest (80 %) for the 25- neurode net - Predictivity is the ability to predict fatal outcomes among false and true fatal predictions. It was uniformly low, varying from 57 % (for the 5-neurode net) to 51 % (10- and 15- neurode nets. The learning curves (Fig. 3) were slow in decay and irregular for all networks. The net with 15 hidden layers performed better in this aspect (final global error of 1.16 after 45.000 cycles). The distribution of network output for the test datasets showed, in contrast to the training datasets, that from 4 to 7 % of the outcome predictions fell into intermediate ranges (from 0.3 to 0.8), showing a 'fuzzy' prognosis. DISCUSSION AND CONCLUSIONS It is difficult to assess which network performed better, because, as it happens with medical test evaluators, there is a conflict between sensivitity and specificity, i.e., when you are able to detect better negative outcomes, positive outcome prediction ability suffers, and vice-versa. Since, in an ICU context, is preferrable to have false positives than false negatives, i.e., exaggerating on the side of death prevention is considered ethically more acceptable), the best net is that with 20 neurodes, which has 75 % hits in the test set, 60 % sensitivity and 80 % specificity. This means that the net predicted incorrectly only 3 out of 282 cases in the training set, and 25 cases out of 100 cases in the test set. This performance could be considered as very good, because the number of variables is relatively large, variability is large, association with outcome is not large for most of the variables, deaths outside ICU are not recorded, and there is a wide range of etiologies in the dataset. However, having too many false negatives will increase the costs of intensive care, so a compromise could be the 10-neurode network, which has 73% hits in test cases, 57 % specificity and 79 % sensitivity. The results show clearly some puzzling aspects of using neural networks. First, the networks with the largest number of hidden elements is not necessarily the better. Second, learning curves show sudden jumps and falls and other irregular or chaotic features which are not easy to explain. Another point of interest is that we are forcing the network to learn to output a binary number in response to a highly graded input. This is not the case of APACHE, which is also a graded output. Surprisingly, the network learns this task quite well, as judging from the distribution of outputs (Fig. 1), but, as consequence, it performes not so well when the dataset is different. We are allowed to conclude that this first neural network approach has worked well for outcome prognosis in a highly 'noisy' dataset, as it has been noted before [2,3]. In relation to the APACHE system, it has a similar, if slightly lower performance, but has the advantage of deriving its parameters from a regional dataset instead from an universal model. It is slow to train (at least with the hardware we had available) but, once trained, may be incorporated into an useful application for routine usage. Anytime a larger dataset is achieved, retraining is possible, improving the performance. In fact, using on- line learning paradigms, this could happen continuously. REFERENCES 1. Gustafson, D.H.; Fryback, D.; Rose, J. et al - Theoretical Methodology for Severity Index Development. Madison, WI: Center for Health Systems Research and Analysis. University of Winsconsin, 1981. 2. Hripcsak, G. - Using connectionist modules for decision support. Meth. Inform. Medicine, 29: 167-181, 1990. 3. Huang, W.Y. & Lippman, R.P. - Comparisons between neural nets and conventional classifiers. Proc. 1st. International Neural Networks Conference. IEEE, New York, p. 485, 1987. 4. Knaus, W.A.; Draper, E.A.; Wagner, D.; Zimmerman, J.E. - APACHE II: A severity of disease classification system. Critical Care Medicine, 13 (10): 818-829, 1985. 5. Lemeshow, S.; Teres, D.; Pastides, H.; Avrunin, J.S.; Steingrub, J.S. - A method for predicting survival and mortality of ICU patients using objectively designed weights. Critical Care Medicine, 13: 519, 1985. 6. Reggia, J.A.; Sutton III, G.G. - Self-processing networks and their biomedical implications. Proceed. IEEE, 76(6): 580-92, 1988. 7. Rumelhart, D.E.; Hinton, G.E.; Williams, R.J. - Learning internal representation by back-propagating errors. Nature, 323: 533-6, 1986. 8. Sabbatini, R.M.E. - Applications of connectionist systems in Biomedicine. Proceed. 7th World Congress on Medical Informatics. North Holland, Amsterdam, 1992. 9. Sabbatini, R.M.E.; Carvalho-J£nior, P.M.; Felipe Jr., P.M.; Freitas, N.C. - NEURONET: um sistema shell conexionista para classificacao automatica de padroes em Biomedicina. Anais I Forum Nacional de Ciencia e Tecnologia em Saude, Caxambu, MG, 1992. 10. Smith, J.W.; Everhart, J.E.; Dickson, W.C.; Knowler, W.C.; Johannes, R.S. - Using the ADAP learning algorithm to forecast the onset of diabetes mellitus. Proc. 12th. Ann. Symp. Comput. Appl. Med. Care. New York: IEEE Press, p. 261-265, 1988. 11. Terzi, R.G.G.; Joffily, M.; Scandiucci, M.; Meira, S.; Beseggio, R.E.; Gomez, M.I.; Araujo, S.; Mhlen, S.S. - Comparison of three predictive models of hospital death in ICU admissions for head trauma (submitted for publication), 1992. MAILING ADDRESS Renato M.E. Sabbatini N£cleo de Inform tica Biom‚dica da UNICAMP Caixa Postal 6005 13081-970 Campinas, SP - Brasil Fone: (0192) 39-7130, Fax: (0192) 39-4717 E-mail: SABBATINI@CCVAX.UNICAMP.BR TABLE 1 Variables Used in the Study ---------------------------------------------------- VARIABLE TY ASS Chi-square p ---------------------------------------------------- Mean arterial pressure C 1.67 0.4328 Renal failure L P 11.16 0.0008 Purulent peritoneum L P 4.99 0.0254 Sinusal cardiac rhythm L 2.18 0.1401 Atrial fibrilation L P 4.30 0.0380 Imunodepression L P 3.95 0.0469 Non-surgical case L N 10.38 0.0012 Elective surgical case L N 25.72 0.0013 Emergency surgical case L P 3.90 0.0484 Age C 5.06 0.1671 Sex L 1.94 0.1631 Coma on admittance L P 11.75 0.0006 Immediate post-surgical L P 8.84 0.0029 Hemodynamic instability L P 10.17 0.0014 Diagnostic coefficient C P 79.56 0.0000 Subaxilar temperature C* P 17.40 0.0006 Heart rate C* P 22.45 0.0000 Respiratory rate C* 4.76 0.1904 FIO2 C* P 9.21 0.0265 Blood pH C* P 16.75 0.0007 Serum Na+ C* P 8.70 0.0335 Serum K+ C* P 11.87 0.0007 Serum creatinine C* 5.21 0.0740 Hematocrit C* 1.69 0.6403 White blood cell count C* P 14.61 0.0021 Glasgow index C P 42.27 0.0000 Coma L P 29.15 0.0000 Neoplastic disease L 1.09 0.2957 Infection L P 11.75 0.0006 Number of failed organs C P 26.42 0.0000 ----------------------------------------------------- Chi-square: of variable against death outcome ASS: P - positively associated N - negatively associated TY: L - logical variable C - continuous scaled variable * - worst measurement taken during ICU stay p - probability of random association For detailed explanation aboutvariables, see [4]. (c) 1992 Copyright Paulo Felipe Jr., Renato Marcos Endrizzi Sabbatini, Paulo Marcondes Carvalho Jr., Roseli Eliana Beseggio, Renato Giuseppe Giovanni Terzi Published in: Anais I Forum Nacional de Ciencia e Tecnologia em Saude, Caxambu, MG, Brasil, Novembro 1992, pp. 344-47. ADRESS FOR CORRESPONDENCE: Renato M.E. Sabbatini, PhD Center for Biomedical Informatics State University of Campinas P.O. Box 6005 Campinas, Sao Paulo, 13081-970, Brazil Tel. +55 192 39-7130 Fax. +55 192 39-4717 Tlx. +55 19 1150 uec br