LODReﬁne, a powerful tool for cleansing and automatically reconciling data with external databases, includes all core features of OpenReﬁne and extends them with LOD-speciﬁc ones. Core features include:
• Importing data from various formats.
• Cleansing data: ﬁnding duplicates, removing them, ﬁnding similar values.
• Filtering data using faceted browsing.
• Filtering data with regular expressions.
• Google Reﬁne Expression language (GREL): a powerful language for transforming data.
• Reconciling with Freebase: the ability of linking your data to Freebase.
• Extending data from Freebase: the ability of adding data from Freebase to your reconciled data.
Figure 3 features faceted browsing, using regular expressions and GREL.
LOD-enabling features added support for:
• Reconciliation and extending data with DBpedia.
• Named-entity recognition: recognizing and extracting named entities from text using diﬀerent services.
• Using crowdsourcing: creating crowdsourcing jobs and uploading data to crowdsourcing platforms.
Fig. 3. LODReﬁne: faceted browsing, support for regular expressions and using GREL
The quality of reconciliation may vary with data and manual evaluation of those results is needed. In case we do not have available human resources crowdsourcing might be considered as a viable solution.
In the following we describe three use cases of using crowdsourcing from and within LODReﬁne. For further details readers are referred to the corresponding project deliverable .
Evaluating reconciliation results. Quality of linking (reconciliation) in the context of Linked Data can be evaluated by using rather sophisticated algorithms or manually with human evaluators. In the last case crowdsourcing can signiﬁcantly speed up the process, especially when LODReﬁne is used to create a job from reconciliation evaluation template.
In this use case crowdsourcing was used to evaluate the quality of reconciled dataset of National Football League players. Data contains names of players and links to their oﬃcial proﬁles on NFL webpage. Freebase was used for reconciliation, and manual evaluation was done ﬁrst by a group of in-house trained evaluators and then by workers at CrowdFlower. Because we already had veriﬁed evaluation results, we were able to asses the quality of results obtained by crowdsourcing.
Validation using crowdsourcing was split in two batches. For the ﬁrst batch we collected three judgments per unit and for the second batch we lowered the overall costs by collecting only two judgments per unit. Although the quality of judgments dropped slightly for the second batch, the ratio costs versus quality of results was satisﬁable.
Validating named entities extracted from blogs. Integrating new entities into recommendation system is crucial for suggesting relevant contents to bloggers. New entities can be discovered by extracting links from blog posts. New links are considered as potential new named entities and links' anchors as entity aliases. In this use case we use crowdsourcing to verify extracted links from blog posts and mark them as entities if appropriate. If link was considered an entity, contributors also provided the type of entity.
We published this job on all available channels, which was reﬂected in the high number of overall units per hour, while in other two use cases we only used 2–4 labor channels (Amazon MTurk was always selected) and thus obtained much lower overall units per hour.
Data enrichment – finding missing information about festivals. Finding missing bits of information and enriching data is a frequently appearing type of assignment on crowdsourcing platforms.
In this use case we used crowdsourcing to enrich a dataset about festivals, which was extracted from blog posts mentioning festival and conference-like events either with their short names or using the full titles. In some cases blog posts mentioned words “festival” or “fest”, but in a diﬀerent context, and were wrongly extracted as a festival. We wanted to identify such cases and enrich data about actual festivals.
Data enrichment took much longer than other two use cases. Searching for data about festivals was more time consuming and questions were slightly more diﬃcult. The price set was also relatively low, which was the other factor impacting time needed to collect responses.
Quality Evaluation of Crowdsourcing Results
All results obtained by crowdsourcing have been evaluated by comparing them to results provided by in-house trained evaluators. A lot depends on how instructions and questions are formulated, how much quality control is involved and which labor channels tasks are published on. In our case we got the best results in the ﬁrst use case, in which contributors had to choose one of the provided suggestions or ﬁnd a link in Freebase and thus there was not much room for subjectivity. Second best use case was data enrichment, where contributors had to check, whether the link contains information about a certain type of event – a festival – and provide a full name, a short name and a homepage for it. Again, the instructions and questions did not allow for too much subjectivity. The least good results were obtained in the second use case involving the evaluation of named entities. There are many possible causes for this: it might be not easy to grasp the notion of a named entity for an average contributor, contributors might not read instructions carefully enough, or instructions might have been too complicated.
Crowdsourcing is a relatively new business model and still requires some time before it will be fully mature and properly supported by legislation, but it can be regarded as a useful and feasible approach at least for some types of LOD-related problems and tasks that were described and demonstrated above. There are several considerations that need to be taken into account while using crowdsourcing: quality assurance measures and constraints have to be applied and ethical issues related to fair price and reward for all contributors have to be considered. Although we would not use it for sensitive data or for gigabytes of data at once, crowdsourcing can serve as a starting point for developing automated solutions. It can provide means to collect enough data to train algorithms and to evaluate results obtained by those algorithms.
-  Oﬃcial NFL webpage: nﬂ.com/