In order to detect and track moving targets in aerial video, a tracking system was established. For the characteristic of background movement of the aerial video, a sparse sample !based method containing Taylor expansion of local compensation error function was used to estimate the background movement model. A motion!vector clustering algorithm was presented to locate the potential targets in the video, relevant trackers were then constructed to select true moving targets and perform long !term tracking of them. In order to improve the precision of the background movement model parameter, the result of image matching was clustered. Using the fragmentary information of the moving targets to gain the targets edge, the precision of locating the targets was improved. The results show that the method is effective even in environment with heavy noise and light!effects.