Detection of infrared dim small target becomes rather difficult if the target lies in a complicated background. Background prediction is one of the important methods for the detection of dim small target in infrared image,and the determination of weight matrix in model of background prediction is of great importance to the performance of prediction and detection. Three different neighborhoods around the predicted pixel are defined,prediction effects of different neighborhoods are discussed and detection performance is compared. Simulation indicated that the detection performance using middle neighborhood is more efficient and middle neighborhood approach is less computationally expensive than multineighborhood method.