The ultrafast dynamics of photoexcited carriers closely depends on the excitation processes pertaining to the energy band of the materials and the relevant relaxation pathway relies on the interactions between hot carriers and lattice phonons. By using ultrafast optical-pump terahertz (THz)-probe spectroscopy with an ion-gel gate to tune the Fermi energy level EF in graphene, we are able to reveal the relaxation dynamics of hot carriers at different EF. It is found that the relaxation time increases while the pump-induced differential transmission decreases as EF approaches the Dirac point. Through self-consistent model calculations, we quantitatively interpret that a temperature-dependent scattering rate is responsible for a negative photoinduced conductivity, and the relaxation transient directly manifests the Dirac spectrum dependence of the optical phonon emission and the carrier scattering rate. More interestingly, the scattering rate of hot carriers also exhibits a strong EF dependence, which is the most likely to originate from charged impurities inevitably present in graphene. The diminution of photoresponse efficiency across the Dirac point implies that the graphene-based optoelectronic devices may be operable only in the highly doped regime.