Results

Model Comparison: Accuracy Results

To be able to test the validity of the presented models on a holdout sample, only a subset of the cases is used to build the models (i.e. 'training set'). The decline in goodness-of-fit between this 'training' set and the validation set is taken as an indicator of the degree of overfitting. The purpose of the validation test is also to evaluate the predictive ability of the three techniques for a new set of cases. For each decision step, we used a random sample of 75% of the cases to build and optimize the models. The other subset of 25% of the cases was presented as 'unseen' data to the models; this part of the data was used as the validation set. The accuracy percentages that indicate the predictive performance of the three models on the training and test sets are presented in Table 7.2.

It can be seen from this table that the accuracy percentages of the BN and BNT classifier outperform the accuracy results of CHAID decision trees. A paired /-test comparing the accuracies of the test data of the CHAID models with these of the

Table 7.2: Comparison of accuracy percentages for CHAID decision trees, Bayesian networks and integrated BNT classifier.

BN and BNT models yields respective P-values of 0.006 (CHAID versus BN) and 0.0052 (CHAID vs. BNT). Therefore, the null hypothesis stating that there is no difference between the accuracies of the test sets of both models could not be accepted at a 5% significance level. On the other hand, the accuracy of the BNT classifier is not significant compared to Bayesian networks, as indicated by a P-value of 0.1643. This should not be surprising of course, because Bayesian networks were used as the underlying structure of the decision trees. More important is the observation that the newly proposed BNT classifier outperformed the CHAID decision trees for all nine decision agents of the Albatross model. This means that by using Bayesian networks as the underlying structure for building decision trees, better results can be obtained than using a traditional CHAID-based decision tree approach. In terms of validity of the three models, we can conclude that the degree of over-fitting (i.e. the difference between the training and the validation set) is low for all decision agents. Therefore, we conclude that the transferability of the models to a new set of cases is satisfactory.