Results and evaluation

To compute the associations for a given stimulus word, we look at its association vector as computed in the way described above, and rank the words in the vocabulary according to association strength. Table 4.2 (two columns on the right) exemplifies the results for the stimulus word colct. For comparison, the two columns on the left list the responses from the EAT, and words occurring in both lists are printed in bold. It can be seen that especially the test persons’ most frequent responses are predicted rather well in the simulation: among the top eight experimental responses, six can be found among the computed responses.

Surprisingly, although the system solely relies on word co-occurrences, it predicts not only syntagmatic but also paradigmatic associations (e.g. not only cold ^ ice but also cold ^ hot; see [DE 96, RAP 02]).

We conducted a straightforward evaluation of the results. It is based on lemmatized versions of both the British National Corpus and, as this is the quasi-standard for evaluation in related works, the Kent and Rosanoff [KEN 10] subset of the Edinburgh Associative Thesaurus which comprises 100 words.

For 17% of the stimulus words, the system produced the primary associative response, which is the most frequent response as produced by the human subjects[1] [2]. In comparison, the average participant in the Edinburgh Associative Thesaurus [KIS 73] produced 23.7% primary responses to these stimulus words. This means that the system performs reasonably but not quite as well as the test persons.

Observed

responses

# of subjects

Computed

responses

# of subjects

hot

34

water

5

ice

10

hot

34

warm

7

weather

0

water

5

wet

3

freeze

3

blooded

0

wet

3

ice

10

feet

2

air

0

freezing

2

winter

2

nose

2

freezing

2

room

2

bitterly

0

sneeze

2

damp

0

sore

2

wind

0

winter

2

warm

7

arctic

1

felt

0

bad

1

war

1

beef

1

night

0

blanket

1

icy

0

blow

1

heat

1

cool

1

shivering

0

dark

1

cistern

0

drink

1

feel

0

flu

1

windy

0

flue

1

stone

0

frozen

1

morning

0

hay fever

1

shivered

0

head

1

eyes

0

heat

1

clammy

0

hell

1

sweat

0

ill

1

blood

0

north

1

shower

0

often

1

rain

0

shock

1

winds

0

shoulder

1

tap

0

snow

1

dry

0

store

1

dark

1

uncomfy

1

grey

0

war

1

hungry

0

Table 4.2. Comparison between observed and computed associative responses to the stimulus word cold (matching words in bold; no lemmatization; capitalized words transferred to lower case)

  • [1] In order not to lose information, in contrast to all other results presented in this chapter, thistable is based on an unlemmatized corpus and an unlemmatized association norm.
  • [2] Wettler et al. [WET 05] report somewhat better results by additionally taking advantage of the observation that test persons typically answer with words from the mid-frequency range. As it is not clear how this affects the results when computing associations for several givenwords, we did not do so in the current paper.
 
Source
< Prev   CONTENTS   Source   Next >