To improve the space anti-radiation performance of the optical satellite communication system,the research on the space radiation environment′s influence on system bit error ratio（BER）is carried out.The radiation-induced BER model is established,based on which the BER numerical calculations after the radiation damage of either optical source or optical detector are developed.The calculation results have proven that both γ radiation and proton radiation can degrade the system BER by several order of magnitude.It points out that some suitable radiation shielding measures or homodyne BPSK mode are needed.