Previous behavioral and scanning studies have suggested that handedness is associated with differences in brain morphology as well as in anatomical and functional lateralization. However, little is known about the topological organization of the white matter (WM) structural networks related to handedness. We employed diffusion tensor imaging tractography to investigate handedness- and hemisphere-related differences in the topological organization of the human cortical anatomical network. After constructing left hemispheric/right hemispheric weighted structural networks in 32 right-handed and 24 left-handed healthy individuals, we analyzed the networks by graph theoretic analysis. We found that both the right and left hemispheric WM structural networks in the two groups possessed small-world attributes (high local clustering and short paths between nodes), findings which are consistent with recent results from whole-brain structural networks. In addition, the right hemisphere tended to be more efficient than the left hemisphere, suggesting a high efficiency of general information processing in the right hemisphere. Finally, we found that the right-handed subjects had significant asymmetries in small-world properties (normalized clustering coefficient γ, normalized path length λ, and small-worldness σ), while left-handed subjects had fewer asymmetries. Our findings from large-scale brain networks aid in understanding the structural substrates underlying handedness-related and hemisphere-related differences in cognition and behavior.