The application of directional overcurrent relays (DOCRs) plays an important role in protecting power systems and ensuring their safe, reliable, and efficient operation. However, coordinating DOCRs involves solving a highly constrained and nonlinear optimization problem. The primary objective of optimization is to minimize the total operating time of DOCRs by determining the optimal values for decision variables such as the time multiplier setting (TMS) and plug setting (PS). This article presents an efficient hybrid optimization algorithm that combines the modified firefly algorithm and genetic algorithm to achieve improved solutions. First, this study modifies the firefly algorithm to obtain a global solution by updating the firefly’s brightness and to prevent the distance between the individual fireflies from being too far. Additionally, the randomized movements are controlled to produce a high convergence rate. Second, the optimization problem is solved using the genetic algorithm. Finally, the solution obtained from the modified firefly algorithm is used as the initial population for the genetic algorithm. The proposed algorithms have been tested on the IEEE 3-bus, 8-bus, 9-bus and 15-bus networks. The results indicate the effectiveness and superiority of the proposed algorithms in minimizing the total operating time of DOCRs compared with other optimization methods presented in the literature.