We present GERBIL, an evaluation framework for semantic entity annotation. The rationale behind our framework is to provide developers, end users and researchers with easy-to-use interfaces that allow for the agile, fine-grained and uniform evaluation of annotation tools on multiple datasets. By these means, we aim to ensure that both tool developers and end users can derive meaningful insights pertaining to the extension, integration and use of annotation applications. In particular, GERBIL provides comparable results to tool developers so as to allow them to easily discover the strengths and weaknesses of their implementations with respect to the state of the art. With the permanent experiment URIs provided by our framework, we ensure the reproducibility and archiving of evaluation results. Moreover, the framework generates data in machine-processable format, allowing for the efficient querying and post-processing of evaluation results. Finally, the tool diagnostics provided by GERBIL allows deriving insights pertaining to the areas in which tools should be further refined, thus allowing developers to create an informed agenda for extensions and end users to detect the right tools for their purposes. GERBIL aims to become a focal point for the state of the art, driving the research agenda of the community by presenting comparable objective evaluation results.

Purpose – Linking Open Data (LOD) provides a vast amount of well structured semantic information, but many inconsistencies may occur, especially if the data are generated with the help of automated methods. Data cleansing approaches enable detection of inconsistencies and overhauling of affected data sets, but they are difficult to apply automatically. The purpose of this paper is to present WhoKnows?, an online quiz that generates different kinds of questionnaires from DBpedia data sets. Design/methodology/approach – Besides its playfulness, WhoKnows? has been developed for the evaluation of property relevance ranking heuristics on DBpedia data, with the convenient side effect of detecting inconsistencies and doubtful facts. Findings – The original purpose for developing WhoKnows? was to evaluate heuristics to rank LOD properties and thus, obtain a semantic relatedness between entities according to the properties by which they are linked. The presented approach is an efficient method to detect popular properties within a limited amount of triples. Ongoing work continues in the development of sound property ranking heuristics for the purpose of detecting the most relevant characteristics of entities. Originality/value – WhoKnows? uses the approach of “Games with a Purpose” to detect inconsistencies in Linked Data and score properties to rank them for sophisticated semantic search scenarios.