The generation of test vectors is a key technique that affects the efficiency and fault detection rate of the boundary scan test. Aiming at the local optimal solution problem of the current common test vectors generation algorithm, this paper proposes a test vectors generation algorithm based on improved GA-AO* model, through which the test vectors are generated by using the idea of heuristic search and backtracking correction. In order to speed up the heuristic search, this paper designed a heuristic function with both prior and posterior parameters to describe the influence of typical faults on the failure probability index of the test vectors. At the same time, this paper used a genetic algorithm (GA) to determine the specific values of the posterior parameters iteratively. Finally, through theoretical analysis and physical verification, compared with the test vector generated by the traditional method, the test vector generated by this method is optimized on the prior failure probability index and performs better in the physical experiment.