In this article, a new hybrid metaheuristic optimization algorithm is proposed to solve the coordination problem of directional overcurrent relays (DOCRs). The proposed algorithm is constructed using hybrid whale optimization algorithm and gray wolf optimizer (HWGO) that enhance the performance and reliability of the traditional whale optimization algorithm (WOA). The proposed method enhances the exploitative phase of the WOA using a leadership hierarchy of the gray wolf optimizer (GWO) to find the best optimum solution. The coordination problem of DOCRs is subject to numerous constraints. The goal function for optimal coordination of DOCRs aims to minimize total operation time for all primary relay without violation in constraints to maintain reliability and security of the electric power system. The effectiveness of the proposed algorithm has been investigated on four different interconnected networks. The results using HWGO algorithm are compared with the original WOA, GWO, and earlier reported results of other optimization techniques. The results prove the viability of the proposed algorithm to solve the DOCR coordination problem and the ability of the proposed algorithm to overcomes the drawbacks and cover the weakness of the conventional WOA.