The artificial neural networks (ANNs)-based model has been used to predict the compressive strength of concrete, assisting in creating recycled aggregate concrete mixtures and reducing the environmental impact of the construction industry. Thus, the present study examines the effects of the training algorithm, topology, and activation function on the predictive accuracy of ANN when determining the compressive strength of recycled aggregate concrete. An experimental database of compressive strength with 721 samples was defined considering the literature. The database was used to train, validate, and test the ANN-based models. Altogether, 240 ANNs were trained, defined by combining three training algorithms, two activation functions, and topologies with a hidden layer containing 1–40 neurons. The ANN with a single hidden layer including 28 neurons, trained with the Levenberg–Marquardt algorithm and the hyperbolic tangent function, achieved the best level of accuracy, with a coefficient of determination equal to 0.909 and a mean absolute percentage error equal to 6.81%. Furthermore, the results show that it is crucial to avoid the use of overly complex models. Excessive neurons can lead to exceptional performance during training but poor predictive ability during testing.