In order to extract the edge and directional information of the source images effectively,a multiscale directional local extrema filter image fusion algorithm based on local extrema and nonsubsampled directional filter bank is proposed. The proposed method takes full use of the feature of edge-preserving filter and the ability of the nonsubsampled directional filter bank,which can preserve the edges and extracts the directional information of the source images. The low-frequency subbands and the high-frequency directional detail subbands are obtained by decomposing the source images. The low-frequency subband coefficients are fused by an adaptive sparse representation(ASR) rule combined spatial frequency and l1 norm. The high-frequency directional detail subband coefficients are fused by an adaptive selection and average strategy based on the match degree of the sum of modified Laplacian. Experimental results show that the proposed method could extract the detail information of the source images such as edges. The fusion results have higher contrast and better subjective visual effect. The proposed method also outperforms the conventional image fusion methods in terms of the objective evaluation criteria.