Converting a Gaussian mode to a vortex beam is much more inconvenient in the terahertz (THz) region than in the near-infrared (NIR) region due to underdevelopment of THz components and strong THz diffraction. This Letter reports the direct generation of THz vortex pulses by optical difference-frequency between two NIR chirped pulses with different topological charges (TCs). By designing a passive and transmissive device for a collinear NIR pulse pair with conjugated TCs, we have experimentally obtained stable THz vortex pulses with a TC value of 2 or -2. The process needs no THz components and so is flexible to be realized and has promising applications in the THz field.