In order to establish a relationship between microstructure and mechanical properties, we systematically annealed a Zr-based bulk metallic glass (BMG) at 100 ~ 300 °C and measured their mechanical and thermal properties. The as-cast BMG exhibits some ductility, while the increase of annealing temperature and time leads to the transition to a brittle behavior that can reach nearly-zero fracture energy. The differential scanning calorimetry did not find any significant changes in crystallization temperature and enthalpy, indicating that the materials still remained fully amorphous. Elastic constants measured by ultrasonic technique vary only slightly with respect to annealing temperature and time, which does obey the empirical relationship between Poisson's ratio and fracture behavior. Nanoindentation pop-in tests were conducted, from which the pop-in strength mapping provides a "mechanical probe" of the microscopic structural heterogeneities in these metallic glasses. Based on stochastically statistic defect model, we found that the defect density decreases with increasing annealing temperature and annealing time and is exponentially related to the fracture energy. A ductile-versus-brittle behavior (DBB) model based on the structural heterogeneity is developed to identify the physical origins of the embrittlement behavior through the interactions between these defects and crack tip.