Knowledge Graph (KG) Construction is the prerequisite for all other KG research and applications. Researchers and engineers have proposed various approaches to build KGs for their use cases. However, how can we know whether our constructed KG is good or bad? Is it correct and complete? Is it consistent and robust? In this paper, we propose a method called LP-Measure to assess the quality of a KG via a link prediction tasks, without using a gold standard or other human labour. Though theoretically, the LP-Measure can only assess consistency and redundancy, instead of the more desirable correctness and completeness, empirical evidence shows that this measurement method can quantitatively distinguish the good KGs from the bad ones, even in terms of incorrectness and incompleteness. Compared with the most commonly used manual assessment, our LP-Measure is an automated evaluation, which saves time and human labour.
When the estimated probabilities do not match the relative frequencies, we say these estimated probabilities are uncalibrated [39], which may cause incorrect decision making, and is particularly undesired in high-stakes tasks [45]. Knowledge Graph embedding models are reported to produce uncalibrated probabilities [36], e.g., for all the triples predicted with probability 0.9, the percentage of them being truly correct triples is not . In this article, we take a closer look at this problem. First, we confirmed the issue that typical KG Embedding models are uncalibrated. Then, we show how off-the-shelf calibration techniques can be used to mitigate this issue, among which binning-based calibration produces more calibrated probabilities. We also investigated the possible reasons for the uncalibrated probabilities and found that the expit transform, the way used to convert embedding scores into probabilities, is ineffective in most cases.