The Accuracy of Neural Networks in Clinical Outcome Prognosis: a Comparison with Statistical and Heuristic Approaches in Cardiology Renato M.E. Sabbatini (1), Juarez Ortiz (2) and Cláudia G.M. Ghefter (2) (1) Center for Biomedical Informatics, State University of Campinas, P.O. Box 6005, Campinas, SP 13081-970, Brazil; (2) Centro de Cardiologia Não Invasiva, R. Cubatao 726, Sao Paulo, SP, Brazil Dilated myocardial disease (DMD) is a kind of congestive heart failure which occurs when the left ventricle fails to dilate properly in the diastolic phase. Several variables seem to be significantly altered in patients with an unfavorable outcome in this disease, such as ejection fraction, fractional shortening, cardiac index, left ventricular diameters and volumes, myocardial thickness, pattern of left ventricular inflow, etc. Aims: To evaluate the usefulness and accuracy of artificial neural networks (ANN) in the prediction of one-year mortality of patients with DMD, and to compare it with a multivariate statistical method (LDA - linear discriminant analysis); a logics-based machine learning method and human expert clinical judgement.. Methods: Clinical and echocardiographic data gathered from 95 consecutive patients presenting this diagnosis, with known outcome, were randomly divided into training and test datasets of approximately the same size. A heteroassociative, backpropagation 3-layer feedforward perceptron was then constructed and simulated by software in a conventional computer. Its input layer consisted of a number of nodes, representing the standardized values of the clinical and echocardiographic variables measured in the patients. Its output layer consisted of a single node giving a graded estimate of death outcome. Several experiments were done using different number of input variables, number of hidden nodes (2, 5 and 10); and its results were compared in terms of the calculated performance indexes. Two sets of variables were investigated: one (complete set) with 29 neurodes for 18 variables and another (reduced) with the 11 most significant variables, as previously indicated by a statistical univariate; giving 13 input neurodes. Two additional methods were used to compare prediction accuracy with the neural networks: a numerical LDA algorithm; and the computer program KnowledgeSeeker (First Mark Technology Ltd., USA). The models thus obtained were tested with the same separate test dataset. In addition, we tested the performance of a simple decision rule proposed by an expert in echocardiography on the basis of his clinical experience. Performance indices were computed after the test (specificity, sensitivity, predictivity and the ROC curve) for each model.Results: All neural networks trained to convergence within 3,000 training cycles. Global mean squared error (RMS) was below 0.01 in all cases, reaching up to 0.004 for the best cases. The solutions converged to the near proximity of 0 and 1 in all cases. All networks were able to predict outcome, with accuracy ranging from 93.33 % (net 13-10-1) to 100 % , for the training dataset; and ranging from 72 % (net 29-2-1) to 90 % (net 13-5-1). In general, the networks with the reduced set of variables fared better than those with the full set; and within each of these two groups of ANNs, those with the intermediate number of hidden neurodes had the best performance. Differences were not large, however, except for the best network (13-5-1), with 11 variables and 5 hidden neurode, which yielded 90 % accuracy, 71,4 % sensibility and 93 % specificity. When applied to the full dataset, the rule derived from simple clinical judgment yielded an accuracy of 82 %, a sensitivity of 46.7 % and an specificity of 88 %. The method of automatic induction used by KnowledgeSEEKER, on the other hand, derived an entirely different production rule, which achieved an accuracy of 91.5 %, a sensitivity of 46.7 % and an specificity of 100.0 %, which is better than human judgment. The ANN was, thus, the best overall performance of all four decision methods.Conclusions: The ANNs performance in the prognosis of outcome for patients with DMD can be rated as very good, considering that: 1) the time of measurements was not fixed in relation to the onset of disease; 2) a large number of input variables was associated with outcome; 3) input variables presented a large variability; 4) the number of training examples was too low; and 5) subsequent therapeutic intervention was not controlled or accounted for. While both heuristic solutions have good to excellent accuracies and specificities, which are similar to those obtained by neural networks, sensitivities are unacceptable low for good clinical decision support. All methods systematically predicted more deaths than actually occurred. This has been observed also for the LDA, which, in consequence, had a very low positive predictivity; as well as for the simple heuristic rules derived by clinical expertise or automatic induction. Generally, we can conclude that the ANN approach, due to its nonlinear characteristics, was the most accurate prognostic model for the present problem. ----------- INFORMATION SHEET Authors: Renato Marcos Endrizzi Sabbatini, PhD Juarez Ortiz, MD Claudia M. Ghefter, MD Address: Center for Biomedical Informatics State University of Campinas P.O. Box 6005 Campinas, SP 13081-970 BRAZIL Phone: +55 192 39-7130 Fax: +55 192 39-4717 Email: sabbatini@ccvax.unicamp.br ----------------