Plate–fin heat exchangers (PFHEs) are widely used in petrochemical, energy and electric power, aerospace, and other industries with large heat transfer requirements. The development of performance prediction and optimization methods for PFHEs has become increasingly important in the design and operation of such heat exchangers (HEs). This paper establishes a database of flow and heat transfer characteristics for four types of PFHEs with different structural parameters. Based on this database, the back-propagation artificial neural network (BP-ANN) model was optimized using the particle swarm optimization (PSO) algorithm to form the PSO-BP-ANN model for the performance prediction of these four types of PFHEs. This combination has been found to improve the prediction accuracy and generalization ability of the BP-ANN model. Additionally, the non-dominated sorting genetic algorithm II (NSGA-II) method was used to characterize the relationship between four structural parameters to be optimized (the length, height, spacing, and thickness of the HE fin) and the two objective functions (j and f) of the serrated PFHE in laminar flow. This enables the Pareto optimal solution to be obtained. The results show that, under laminar flow conditions (Re = 800), the serrated fin HE achieves the best heat transfer performance when the fin height, spacing, thickness, and length are 9.29, 1.22, 0.16, and 3.06, respectively.