A novel infrared pedestrian tracking algorithm based on template difference method was proposed. The algorithm uses the mean intensity of the template instead of the template border,and then the positive pixel region and negative pixel region were gotten through the difference of the previous frame and the current frame. Based on this,the vector of positive region centroid to negative region centroid was calculated,and the template was moved with single pixel along the vector. When the target’s template is approximately coincide with the human body in current frame,infrared pedestrian tracking is achieved. The experimental results show that the proposed algorithm basically solves the distinction problem of pedestrian intersection in infrared pedestrian tracking.