With the progress of network technology, online video is widely used by users and popular. Image and video retargeting techniques have been proposed in the recent past by researchers around the world, low-complexity algorithms allow video to be transmitted more quickly. Our Balanced Error Distribution (BED) retargeting method follows the same concept like uniform scaling, but only removes unimportant regions, balancing main objects’ error distribution. Priority map is proposed in this paper to maintain the structure of straight lines and irregular shape of objects without deforming complex image contents, which may be altered in traditional seam carving for complex image or video. In addition, the proposed mechanism adopts the general temporal coherence algorithm with strict condition (two continuous seams must be close as possible) to maintain visual continuity, such that the resulting video will not look shaky due to sudden changes in the background. The proposed method not only resizes the video with the retention of important information but also maintains the structural properties of objects in different kinds of videos.