Fiber Bragg grating (FBG) has been gradually applied in high speed optical fiber communication systems with the advantages of convenient operation,low cost and high reliability,whose dispersion compensation abilities have attracted much attention. Transmission properties of three kind of on-off keying (OOK) modulation formats,including Not Return to Zero(NRZ),33% Return to Zero(33%RZ) and Carrier Suppressed Return to Zero(CSRZ),were comparatively studied in FBG dispersion pre-compensation,symmetric-compensation and post-compensation structures. Simulation results show that OOK modulation formats have the best transmission performances in FBG dispersion symmetric-compensation structure. 33%RZ has better transmission characteristics when separately considering nonlinear effects and polarization mode dispersion (PMD),and CSRZ has the best system transmission performances when synthetically considering the residual dispersion,nonlinear effect and PMD.