Vision Transformer (ViT) has demonstrated exceptional performance in image classification tasks across large-scale datasets. However, its application in domain-specific or small-scale datasets remains a challenge. This research explores an alternative approach to image patch generation, replacing the fixed-size patch mechanism in ViT with semantic-aware segmentation using the Segment Anything Model (SAM). We focus on applying this technique to datasets such as marine biology, animals, and plants, where semantic consistency plays a more critical role. The segmented patches are compared to the conventional 16×16 patches used in ViT to evaluate their potential to enhance semantic representation. Preliminary results suggest that using SAM-based patches can introduce better localized and meaningful features, providing a foundation for performance enhancement in downstream tasks.
