The performance indicator, Overall Equipment Effectiveness (OEE), is one of the most important ones for production control, as it merges information of equipment usage, process yield, and product quality. The determination of the OEE is oftentimes not transparent in companies, due to the heterogeneous data sources and manual interference. Furthermore, there is a difference in present guidelines to calculate the OEE. Due to a big amount of sensor data in Cyber Physical Production Systems, Machine Learning methods can be used in order to detect several elements of the OEE by a trained model. Changeover time is one crucial aspect influencing the OEE, as it adds no value to the product. Furthermore, changeover processes are fulfilled manually and vary from worker to worker. They always have their own procedure to conduct a changeover of a machine for a new product or production lot. Hence, the changeover time as well as the process itself vary. Thus, a new Machine Learning based concept for identification and characterization of machine set-up actions is presented. Here, the issue to be dealt with is the necessity of human and machine interaction to fulfill the entire machine set-up process. Because of this, the paper shows the use case in a real production scenario of a small to medium size company (SME), the derived data set, promising Machine Learning algorithms, as well as the results of the implemented Machine Learning model to classify machine set-up actions.