Resistive crossbars using non-volatile memory devices have become promising components for implementing deep learning models in hardware. However, crossbar-based computations encounter a notable challenge due to device and circuit-level non-idealities. In this study, we explore three recent crossbar simulation tools, namely, AIHWKIT, CrossSim, and MemTorch, and evaluate four pre-trained convolutional neural networks (CNNs) for image classification on CIFAR-100, MNIST, and SVHN datasets using these tools. To the best of our knowledge, this is the first study using all three simulation tools of analog accelerators, representing three distinctive hardware settings, to evaluate CNN-based model robustness under analog noise and nonlinearities. We first test the robustness of four models (VGG19, InceptionV3, ResNet50, & MobilenetV2) using different levels of white Gaussian noise as baselines. Then we evaluate their inference performance on the three tools to determine their resilience to analog noise and nonlinearities in the hardware environment. Results show that while all CNNs' suffer performance degradation as expected, insights are obtained such as ResNet50 outperforms others in two out of three simulators despite real-world hardware imperfections due to its deep structure. Furthermore, we explore the impact of three key factors: cell bits, ADC resolution, and tile size on CNN performance across various hardware configurations, providing recommendations for the efficient deployment of CNN on hardware. Our results suggest that increasing cell bits and ADC resolution improves CNNs' performance. It is also observed that smaller tiles are suitable for lightweight models, while for complex networks, we should choose larger shape of tiles.