In order to detect the moving infrared dim targets in real-time under starry sky background,a method was proposed based on the spatio-temporal directional filter and the least square(LS)prediction. Firstly,the theoretical derivation in frequency domain of both the traditional directional filter and the bi-directional filter was introduced,and the advantages of bi-directional filter were illustrated through contrast. Then,on the basis of the bi-directional filter,a new directional filter algorithm is proposed,and the advantages of time consumption were analyzed. Secondly,after analyzing the performance of the new directional filter algorithm,the image segmentation method based on threshold rate was illustrated. Finally,the theory of LS prediction was introduced,and the corresponding tracking algorithm was given. The experimental results show that the average location deviation of object tracking is about 1 pixel,and it will take about 1.137 seconds to detect and track the image with 20 frames when the image size is 256×256 pixels. The detection and tracking results meet the requirement of precision,and the detecting time can basically meet the real-time requirement of the infrared system imaging at extreme distance.