We propose a novel motion analysis algorithm by using the mean-shift segmentation and motion estimation technique. Mean shift algorithm is frequently used to extract objects from video according to its efficiency and robustness of non-rigid object tracking. For diminishing the computational complexity in searching process, an efficient block matching algorithm: cross-diamond-hexagonal search algorithm was used. In the motion analysis procedure, the stick figure of object obtained by thinning process is treated as guidance to gather the statistics of motion information. The experimental results show that the proposed method can provide precise description of the behavior of object in several video sequences and extrapolate human motion seamlessly by combining different motion clips obtained from other video sequences.