ML Anthology

Comparing Apples and Oranges? on the Evaluation of Methods for Temporal Knowledge Graph Forecasting

Julia Gastinger, Timo Sztyler, Lokesh Sharma, Anett Schuelke, Heiner Stuckenschmidt
ECML-PKDD 2023 pp. 533-549
doi:10.1007/978-3-031-43418-1_32 /ecmlpkdd/2023/gastinger2023ecmlpkdd-comparing/

Abstract

Due to its ability to incorporate and leverage time information in relational data, Temporal Knowledge Graph (TKG) learning has become an increasingly studied research field. To predict the future based on TKG, researchers have presented innovative methods for Temporal Knowledge Graph Forecasting. However, the experimental procedures employed in this research area exhibit inconsistencies that significantly impact empirical results, leading to distorted comparisons among models. This paper focuses on the evaluation of TKG Forecasting models: We examine the evaluation settings commonly used in this research area and highlight the issues that arise. To make different approaches to TKG Forecasting more comparable, we propose a unified evaluation protocol and apply it to re-evaluate state-of-the-art models on the most commonly used datasets. Ultimately, we demonstrate the significant difference in results caused by different evaluation settings. We believe this work provides a solid foundation for future evaluations of TKG Forecasting models, thereby contributing to advancing this growing research area.

PDF ECML-PKDD Semantic Scholar

Cite

Text

Gastinger et al. "Comparing Apples and Oranges? on the Evaluation of Methods for Temporal Knowledge Graph Forecasting." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2023. doi:10.1007/978-3-031-43418-1_32

Markdown

[Gastinger et al. "Comparing Apples and Oranges? on the Evaluation of Methods for Temporal Knowledge Graph Forecasting." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2023.](https://mlanthology.org/ecmlpkdd/2023/gastinger2023ecmlpkdd-comparing/) doi:10.1007/978-3-031-43418-1_32

BibTeX

@inproceedings{gastinger2023ecmlpkdd-comparing,
  title     = {{Comparing Apples and Oranges? on the Evaluation of Methods for Temporal Knowledge Graph Forecasting}},
  author    = {Gastinger, Julia and Sztyler, Timo and Sharma, Lokesh and Schuelke, Anett and Stuckenschmidt, Heiner},
  booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
  year      = {2023},
  pages     = {533-549},
  doi       = {10.1007/978-3-031-43418-1_32},
  url       = {https://mlanthology.org/ecmlpkdd/2023/gastinger2023ecmlpkdd-comparing/}
}