For the large-scale infrared detector with high frame rate generates the infrared images of large data with the requirements of high-speed transmission and real-time processing, a high performance infrared image real-time processing system is proposed in this paper. The system is designed with the framework of CPCI which has the characteristics of high-speed transmission, good scalability, reliability and stability. The high-speed signal processing platform FPGA and DSP are used in this system to complete the functions of high-speed transmission of the infrared images of large data, real-time processing, information fusion, target detecting and target tracking. The framework of the system is simple, flexible and scalable. This system fully combines the advantages of different processors and has the features of high processing capability, high-speed data transmission, reliable and convenient interface and flexible programming. At present, the system has been applied in the project and can meet the requirement of the infrared images of large data real-time processing.