Construction of brain templates is generally carried out using a two-step procedure involving registering a population of images to a common space and then fusing the aligned images to form a template. In practice, image registration is not perfect and simple averaging of the images will blur structures and cause artifacts. In diffusion MRI, this is further complicated by intra-voxel inter-subject differences in fiber orientation, fiber configuration, anisotropy, and diffusivity. In this paper, we propose a method to improve the construction of diffusion MRI templates in light of inter-subject differences. Our method involves a novel q-space (i.e., wavevector space) patch matching mechanism that is incorporated in a mean shift algorithm to seek the most probable signal at each point in q-space. Our method relies on the fact that the mean shift algorithm is a mode seeking algorithm that converges to the mode of a distribution and is hence robust to outliers. Our method is therefore in effect seeking the most probable signal profile at each voxel given a distribution of signal profiles. Experimental results show that our method yields diffusion MRI templates with cleaner fiber orientations and less artifacts caused by inter-subject differences in fiber orientation.