A nonlinear hyperspectral unmixing algorithm based on the optimal endmember subset is given to overcome the endmember variability and nonlinear mixed characteristics of hyperspectral data. Endmember sets of mixed pixels are optimized by adding a shadow endmember. The spectral of optimal endmember subset are decomposed by bilinear spectral decomposition algorithm based on hierarchical Bayesian model. Simulated experiments and real data show that OE-GBM algorithm can solve shadow effect of hyperspectral images,and the accuracy of OE-GBM is better than that of FCLS and GBM algorithm.