Screening a good-performance infrared (IR) nonlinear optical (NLO) material with a large band gap, appropriate second harmonic generation (SHG) response and moderate birefringence has inspired the study of A8BS6 (A = Cu, and Ag; B = Si, Ge, and Sn). The A8BS6 series exhibits apparently hierarchical changes in the optical properties with ion-substitution under the same symmetry. In this paper, by the first-principles method and SHG-density method, the electronic structure, the optical properties and the contribution of respective ions and ion groups have been investigated. This reveals that the substitution from Ag to Cu with isostructural compounds induces the apparent enhancement of the SHG responses, which is mainly attributed to the different intensity of dp hybridization between the A cation and S. In addition, tetrahedra formed by B-site cations contribute little to the SHG, mainly because they offset each other. For the tiny changes in band gaps, it is found that the synergetic effect of B-site ion size, compound volume size and dp hybridization of A–S plays an important role. This study provides a feasible way to design and synthesize good-performance IR NLO materials.