In this paper, we present a fast block matching algorithm by making use of the correlation between layers of the sum pyramid for a block. To speed our method, an algorithm is also developed to predict the initial motion vector for a template block. Using the elimination criteria and predicted motion vector, our method can further reduce the computational complexity significantly. Compared with the full search block matching algorithm, our approach can reduce the computing time by a factor of 9.9–28.2 with the peak signal-to-noise ratio (PSNR) degradation of 0.00–0.03 dB. Compared to WinUpMI, which is the fastest block matching algorithm preserving global optimality as far as we know, our method can reduce the computing time by 4.9–52.5% with the PSNR degradation of 0.00–0.03 dB. Our method can reduce the computing time of predictive search area approach for fast block motion estimation (PSAFBME) by 50–96% with about the same PSNR.