There is increasing interest in automating train operations of mainline services, e.g. to increase network capacity. Automatic train operation (ATO) is already achieved by several pilot projects, but not implemented on a large scale. Before the general introduction of new or adapted technologies can have a transformative effect on the operation of such a complex system as train operation on mainlines, they have to pass functional, interoperability and performance tests. A virtual preliminary analysis is one way to ensure a smooth as well as safe introduction and implementation. This paper aims to present an approach that applies to the performance testing of ATO systems. Therefore, methods and test standards for technologies enabling automatic operation in other transport sectors are reviewed. The main findings have been adapted, transformed and combined to be used as a general strategy for virtual performance testing in the railway sector. Specifically, universal performance indicators, namely punctuality, accuracy, energy consumption, safety and comfort, are presented. A layer model for scenario description is adapted from the automotive sector, as well as the definition of different scenario types. Lastly, factors that can influence the performance of an ATO algorithm are identified. To demonstrate the developed approach, a straightforward investigation of a case study is conducted using a microscopic train simulator in combination with an ATO algorithm.