The optical properties of plasmonic semiconductor devices fabricated by focused ion beam (FIB) milling deteriorate because of the amorphisation of the semiconductor substrate. This study explores the effects of combining traditional 30 kV FIB milling with 5 kV FIB patterning to minimise the semiconductor damage and at the same time maintain high spatial resolution. The use of reduced acceleration voltages is shown to reduce the damage from higher energy ions on the example of fabrication of plasmonic crystals on semiconductor substrates leading to 7-fold increase in transmission. This effect is important for focused-ion beam fabrication of plasmonic structures integrated with photodetectors, light-emitting diodes and semiconductor lasers.