We consider Chern-Simons gauge theory on a torus with both nonrelativistic and relativistic matter. It is shown that the Hamiltonian and two total momenta commute among themselves only in the physical Hilbert space. We also discuss relations among degenerate physical states, degenerate vacua, and the existence of multicomponent Schrödinger wave functions.