In recent years, Transformer models have achieved remarkable success in natural language processing, yet their enormous number of parameters and high computational complexity restrict their application in resource-constrained environments. This thesis proposes a lightweight Transformer variant, termed SFFTT, which replaces the traditional self-attention mechanism with Fast Fourier Transform (FFT) in the first half of the encoder and employs parameter sharing along with attention threshold filtering in the latter encoder layers and the decoder. Additionally, we introduce SFFTTwithDyT by substituting all Layer Normalization layers with Dynamic Tanh normalization to enhance training stability and model expressiveness. Experimental results demonstrate that the SFFTT series models maintain competitive performance while significantly reducing parameter count and computational cost, offering an effective solution for lightweight Transformer applications.
