Evaluation

For evaluation, we simply compared the associations produced by the test persons with the expected results as taken from the EAT (see Table 4.2). In a first round of evaluation, we only considered exact matches as correct, with the only flexibility that word capitalization was not taken into account. However, in a second round, we also counted inflectional variants and derived forms of the expected word as correct. For example, for the given words “car round cart spoke bicycle”, a test person responded with wheels, whereas the expected solution was wheel. This was now also considered to be a match.

Table 4.6. Results by language proficiency for the two modes of evaluation

The overall results are shown in Table 4.6. For each proficiency group, the percentage of correctly predicted words is specified for the two modes of evaluation. All accuracies are in a range between 2 and 8%. As can be seen, the students who rated their language proficiency to be very good did considerably better than the ones who rated themselves to be only good; but surprisingly, the students with an only satisfactory self-rating did even better. Note, however, that this is a small group of only four subjects who cannot be taken as representative (and whose self-ratings might simply reflect modesty).