Word alignment

In order to compare sentences in terms of similarity, we not only need a method for doing so, but also need the data to be in a comparable form. Hence, we need the input to be given in a standardized form or we need to carry out some normalization. The latter can be accomplished to some extent via a dependency parser that reveals the roles played by different words. We can now align the words of the various sentences and compare those playing the same semantic role.

Word alignment consists not simply in finding identical words in different sentences, but rather in finding and aligning words playing the same role in these sentences. This means in our case that we have to compare, say, the subject of one sentence with the subject of another, and do the same for the other syntactic categories or semantic roles (verbs, deep- objects, etc.). To allow for this, we rely again on the dependency information produced by the parser (section 7.6.1). Note that our example showed only surface relations (subject, object, etc.), while ideally we need information in terms of deep-case roles: agent, beneficiary, etc. [FIL 68]. Applied to our examples, “Foxes eat fruit” and “Foxes eat eggs”, it is clear that “fruit” can be aligned with “eggs”, since both nouns play the same role.

Note that we also need to be able to detect synonyms or semantic equivalences: “instrument = is used for”; “resemble = be alike”, “for example = somehow = like”, etc. These words are very useful and could be used as topic-signatures [LIN 00], hence seed words. Note that such information is obtained indirectly in our approach via the vector space model which is briefly described in the next section.