Methods for Automatic Machine-Learning Workflow Analysis

Abstract

Developing real-world Machine Learning-based Systems goes beyond algorithm development. ML algorithms are usually embedded in complex pre-processing steps and consider different stages like development, testing or deployment. Managing workflows poses several challenges, such as workflow versioning, sharing pipeline elements or optimizing individual workflow elements - tasks which are usually conducted manually by data scientists. A dataset containing 16 035 real-world Machine Learning and Data Science Workflows extracted from the ONE DATA platform ( https://onelogic.de/en/one-data/ ) is explored and made available. Based on our analysis, we develop a representation learning algorithm using a graph-level Graph Convolutional Network with explicit residuals which exploits workflow versioning history. Moreover, this method can easily be adapted to supervised tasks and outperforms state-of-the-art approaches in NAS-bench-101 performance prediction. Another interesting application is the suggestion of component types, for which a classification baseline is presented. A slightly adapted GCN using both graph- and node-level information further improves upon this baseline. The used codebase as well as all experimental setups with results are available at https://github.com/wendli01/workflow_analysis .