In coherent optical communication system,the effect of Costas loop in homodyne detection system on phase errors of the entire system was theoretically studied. The impact of receiver sensitivity,power splitting ratio,the number of erbium doped fiber amplifiers on laser line width was analyzed. When the bit rate and bit error rate reached certain conditions,the relationships between laser line width and the receiver sensitivity,the power split ratio,the number of the erbium doped fiber amplifier were obtained through simulation,which provide certain reference for laser line width control.