With their ability to analyse and learn from complex data, neural networks have emerged as an ubiquitous machine learning tool with applications in many industries. Despite the success of neural networks in various tasks, obtaining the optimal structure for a specific problem is both time-consuming and difficult because of the large number of hyperparameters that need to be properly selected. To address this issue, this work proposes an approach for automating the design of neural networks through the use of genetic algorithms (GAs). More specifically, the proposed GA-based technique aims to find the optimal Convolutional Neural Network (CNN) architecture and learning-related hyperparameters for the problem of image classification. The approach is evaluated on the MNIST dataset, a widely used benchmark for image classification tasks, on which our top performing architecture obtains an accuracy of 99.35%. These results demonstrate the effectiveness of the proposed approach in terms of finding architectures that outperform those designed by human experts and are competitive with state-of-the-art models. The evaluation results are accompanied by a comprehensive analysis of the evolution of hyperparameters within generations and the changes in network architecture over time, providing insights into how the GA explores the search space. The proposed approach is a step forward in the direction of automating the process of neural network design and its results indicate that it can be a valuable tool for researchers.