The theory of polarization and the image fusion based on standard HSV are studied. According to the characteristic of strong contrast and weak intensity in infrared polarization image,the improved HSV fusion algorithm is proposed. By using average gray scale and extreme value of original polarization image,the image intensity was corrected;the fusion weight was calculated; the polarization image was fused;the contrast and intensity of infrared polarization image was enhanced. The experiments indoor and outdoor has been done,the results show the HSV fusion method can enhance the average gray scale by 216% at most,and the contrast improved 1384% at most compared to origin IR images;It also has an improvement of 1363% average gray scale and 292% contrast compared to standard HSV. The algorithm is helpful to enhance the average gray-scale and contrast of the polarization images.