Purpose: The purpose of this paper is to investigate the optimal geometry parameters in a dimple/protrusion-pin finned channel with high thermal performance.Design/methodology/approach: The BSL turbulence model is used to calculate the flow structure and heat transfer in a dimple/protrusion-pin finned channel. The optimization algorithm is set as Non-dominated Sorting Genetic Algorithm II (NSGA-II). The high Nusselt number and low friction factor are chosen as the optimization objectives. The pin fin diameter, dimple/protrusion diameter, dimple/protrusion location and dimple/protrusion depth are applied as the optimization variables. An in-house code is used to generate the geometry model and mesh. The commercial software Isight is used to perform the optimization process.Findings: The results show that the Nusselt number and friction factor are sensitive to the geometry parameters. In a pin finned channel with a dimple, the Nusselt number is high at the rear part of the dimple, while it is low at the upstream of the dimple. A high dissipative function is found near the pin fin. In the protrusion channel, the Nusselt number is high at the leading edge of the protrusion. In addition, the protrusion induces a high pressure drop compared to the dimpled channel.Originality/value: The originality of this paper is to optimize the geometry parameters in a pin finned channel with dimple/protrusion. This is good application for the heat transfer enhancement at the trailing side for the gas turbine.