Menu
Home
Log in / Register
 
Home arrow Language & Literature arrow COGNITIVE APPROACH TO NATURAL LANGUAGE PROCESSING
Source

Evaluation

In this section, we present the experimental setting for the evaluation and comparison of our system with state-of-the-art algorithms.

Experimental setting

We evaluated our algorithm with three fine-grained datasets: Senseval-2 English all-words1 (S2) [PAL 01], Senseval-3 English all-words[1] [2] (S3) [SNY 04], SemEval-2007 all-words[3] (S7) [PRA 07] and one coarse-grained dataset, SemEval-2007 English all-words[4] (S7CG) [NAV 07b], using WordNet as a knowledge base. The descriptions of the datasets are presented in Table 6.1.

The results of the evaluation are presented as F1, which is calculated as:

This measure determines the weighted harmonic mean of precision and recall. Precision is defined as the number of correct answers divided by the number of provided answers and recall is defined as the number of correct answers divided by the total number of answers to be provided. In our evaluation, we excluded labeled points in this calculation. Experimentally we noticed that precision is always equal to recall, since the system is always able to provide an answer.

We evaluated two different versions of the system, one using a uniform probability distribution to initialize the strategy space of the games and the other using information from sense labeled corpora (see section 6.4.2). Furthermore, to make the evaluation unbiased, we present the mean and standard deviation results of our system over 25 trials with different sizes of randomly selected labeled points.

Dataset

Text

N

C

Tot. N

S2

1

670

2195

2387

S2

2

997

1836

S2

3

720

1916

S3

1

783

2472

2007

S3

2

633

1426

S3

3

591

1881

S7

1

111

593

455

S7

2

150

798

S7

3

194

1035

S7CG

1

368

1287

2268

S7CG

2

379

1473

S7CG

3

499

1926

S7CG

4

677

1666

S7CG

5

345

1410

Table 6.1. Number of target words and senses for each text of the datasets

  • [1] www.hipposmond.com/senseval2
  • [2] http://www. senseval.org/senseval3
  • [3] http://nlp.cs. swarthmore.edu/semeval/tasks/index.php
  • [4] http://lcl.uniroma1.it/coarse-grained-aw
 
Source
Found a mistake? Please highlight the word and press Shift + Enter  
< Prev   CONTENTS   Next >
 
Subjects
Accounting
Business & Finance
Communication
Computer Science
Economics
Education
Engineering
Environment
Geography
Health
History
Language & Literature
Law
Management
Marketing
Mathematics
Political science
Philosophy
Psychology
Religion
Sociology
Travel