We propose a polarized image defogging algorithm according to the sky segmentation results and transmission map optimization. Firstly, we propose a joint sky segmentation method based on scene polarization information, gradient information and light intensity information. This method can effectively segment the sky region and accurately estimate the global parameters such as atmospheric polarization degree and atmospheric light intensity at infinite distance. Then, the Gaussian filter is used to solve the light intensity map of the target, and the information of the polarization degree of the target is solved. Finally, based on the segmented sky region, a three-step transmission optimization method is proposed, which can effectively suppress the halo effect in the reconstructed image of large area sky region. Experimental results shows that defogging has a big improvement in the average gradient of the image and the grayscale standard deviation. Therefore, the proposed algorithm provides strong defogging and can improve the optical imaging quality in foggy scenes by restoring fog-free images.