The “quantization noise-wavelength error” transformation model is proposed to improve the optical fiber Bragg grating (FBG) wavelength demodulation accuracy. The model is simulated and verified based on the Monte Carlo method. A wavelength demodulation experimental circuit is established by using a high precision 16-bit analog to digital converter (ADC). The error analysis show that the FBG wavelength demodulation error increases exponentially from 0.34 pm to 41.37 pm as the bit-length numbers of ADC decreases from 16 bits to 6 bits. The experiments and the simulation results show that the proposed model can accurately analyze the influence of quantization error on FBG wavelength demodulation accuracy,which provides a reference for the ADC resolution selection in FBG wavelength interrogator.