Previous studies have indicated that amputation induces reorganization of functional brain network. However, the influence of amputation on structural brain network remains unclear. In this study, using diffusion tensor imaging (DTI), we aimed to investigate the alterations in fractional anisotropy (FA) network after unilateral upper-limb amputation. We acquired DTI from twenty-two upper-limb amputees (15 dominant-side and 7 nondominant-side amputees) as well as fifteen healthy controls. Using DTI tractography and graph theoretical approaches, we examined the topological changes in FA network of amputees. Compared with healthy controls, dominant-side amputees showed reduced global mean strength, increased characteristic path length, and decreased nodal strength in the contralateral sensorimotor system and visual areas. In particular, the nodal strength of the contralateral postcentral gyrus was negatively correlated with residual limb usage, representing a use-dependent reorganization. In addition, the nodal strength of the contralateral middle temporal gyrus was positively correlated with the magnitude of phantom limb sensation. Our results suggested a degeneration of FA network after dominant-side upper-limb amputation.