Stray light interference is one of the main effect factors for image process of star sensor. To eliminate the effect of stray light,features of targets and background in star image were analyzed. According to 2D-Gaussian distribution,a weight reference function was constructed,and maximum background estimation model was improved. A Euclid four-quadrant rotational symmetric mask was designed,combining with image segmentation and denoising,an image processing algorithm of star sensor based on maximum background estimation was proposed. A comprehensive method was adopted to evaluate the performance of the image process algorithm,including star detection ratio,false-alarm ratio,star detection limit magnitude,single star position accuracy and timing sequence. The experiment results prove that the proposed algorithm is effective and universal for star detection,also robust for stray light interference.