Determination of the similarity values of the aligned words

While there are various ways to detect links between sentences or words (e.g. shared features or associations), two obvious ones are coreference and class-membership. See our example in Figure 7.4(a), where the two sentences - (“Foxes eat eggs”; “Foxes eat fruits”) - have an identical referent, the actor “fox”, and two different instances of the same class, the generic element “food”.

As mentioned already, in order to compare sentences in terms of their meaning, the compared structures must have a common format. Similarity of meaning supposes, of course, that we are able to extract somehow the meaning of the analyzed objects (sentences, words). Yet, word meanings depend on the context in which a word occurs. Words occurring in similar contexts tend to have similar meanings. This idea known as the “distributional hypothesis” has been proposed by several scholars, e.g. [FIR 57, HAR 54, WIT 22]. For surveys, see [SAH 08, DAG 99], or (http://en.wikipedia.org/wiki/ Distributional_semantics).

Since we try to capture meaning via word similarity, the question of how to operationalize this notion arises. One way of doing so is to create a vector space composed of the target word and its neighbors [LUN 96]. The meaning of a word is represented as a vector based on the co-occurring words. In the following two sentences - “Foxes eat eggs” and “Foxes eat fruits” - we have four distinct tokens or words: foxes, eat, fruits and eggs. Hence, we constructed a vector for each one of them by considering their cooccurrences in COHA (Corpus of Historical American English), a part of speech tagged n-gram corpus of 400 million words [MAR 11]. This allowed us to apply the vector space model in order to compute the degree of similarity between a set of words. To this end, we computed the distance (cosine) of the respective vectors. Let us suppose that there are only four words co-occurring with “fruit” and “egg” (“juice, vitamin, price, eat” and “chicken, protein, eat and oval”), then the vector for fruit would be “juice, vitamin, price, eat” and the vector for egg would be chicken, protein, eat, oval.

Note that we will also count the frequency of the co-occurrence. To compute the similarity between two vectors, we computed the cosine of their angle. Hence, we constructed such vectors for all major words of our sentences. For the example here above, we have four vectors, one for each of the words occurring in both sentences: fox, eat, fruit and eggs. Note that the words in the vectors are replaced by their weight, i.e. a numerical value representing the meaning of the respective concepts. For instance, for the fruit vector, all of the following words, “juice, vitamin, price, eat” are replaced by a numerical value (weight). For details, see step 1 here below.

Consider the following example that measures the proximity between fruit and eggs by using the co-occurrence information gleaned from COHA, yielding the matrix shown here below.

Table 7.3. Sample word space matrix

The vectors are built by carrying out the following four steps:

• 1) Extract words co-occurring with fruit in a predefined window. In our experiments, we considered the window to be six content-bearing words of the sentence containing the target, hence, three terms preceding and following the term fruit that would yield in this case the following list of words: banana, food, gene and store.
• 2) Do the same for eggs, that would yield: bird, hummus, food, incubator and gene.
• 3) Build the corresponding vectors for each word (fruit, egg) based on its co-occurrences (see the table here above). In its simplest form, the vectors for each word are built on the basis of their co-occurrences. A term fruit, eggs) receives frequency values depending on the number of times it cooccurs with another term within the defined window, which would yield the following vectors for fruit [0,0,1,0,0,1,1] and vehicle

that the words here are referred to via their index, e.g. their position in the word space. Hence, the first position refers to the first term, the second to the second, and the last to the last one in the matrix. Unlike in the example given here above, we use in our experiments the weighted frequency of the words’ co-occurrences rather than binary values.

4) Measure the distance between the vectors. In order to quantify the similarity between two words, we have to compute the cosine similarity of their respective vector representations. This is done in the following way:

Take a pair of words, say fruit (T₁) and eggs (T₂), with their respective vectors and weights, and carry out the steps described here below. To determine the similarity between “fruit” and “eggs” we compute their cosine similarity, i.e. the angle between their vectors. The cosine similarity value is computed in the following way:

• 1) sum the product of the weight of the terms in the respective vectors (^and T2);
• 2) sum the square of the weights of each vector term of T₁ and T₂;
• 3) take the square root of the product of the results of step 2;
• 4) divide the result of step 1 by the result of step 3.

We have used this method already for another task, the automatic extraction of part-whole relations [ZOC 15a]. Since then, we have extended it to allow for the computation of similarity between words. The method consists basically of two operations: creating a vector for all words and identifying the similarity of the aligned words.

Step 1: Creation of a vector for all words based on co-occurrence information

The co-occurrence information is gleaned from COHA. The vector is built on the basis of a word’s co-occurrence within a defined window (phrase, sentence and paragraph). Since different words make different contributions, we assign weights to reflect their relative contribution in terms of relevance allowing determination of the meaning of a sentence or word. Hence, meanings are expressed via a weight, which may depend on the context of a given term, a factor that needs to be taken into account. The formula used to assign the weight is very similar to the TF-IDF. Since our objective is determining the similarity between words, the document-term matrix in the conventional TF-IDF is adopted to term-term matrix in our case. In order to determine the weight (W), we use the percentage of the term’s co-occurrence frequency (TCF) with respect to a given concept out of the total number of co-occurrences (TOTNBC) of the term with any other term in the corpus. For example, in order to determine the weight of the term egg (one of the words in the “chicken vector”) for the vector of “chicken”, we count the co-occurrence frequency of chicken with egg and then divide this result by the total number of co-occurrences of the term egg with any other term in the corpus. We did the same for all relevant terms. The abovedescribed operations can be captured via the following formula, which is used to determine the weight (W) of a given word for determining the meaning of another co-occurring term:

W = TCF-yz/TOTNBC-xy, where

TCF-y is the frequency, i.e. the number of times y co-occurs with z;

TF-y is the total frequency y in the corpus.

To build a vector for a given concept, we use the weighted value of all co-occurring words. Hence, we calculated the relative weight of each word of the vectors in defining the meaning of the term for which the vector was built.

Step 2: Identification of the cosine similarity between vectors of the aligned words

The similarities between words are computed on the basis of the cosine of the words’ vectors. Note that the similarity value is calculated only for the aligned words.