We investigate the role of defects in laser-induced damage of fused silica and of silica coatings produced by e-beam and PIAD processes which are used in damage resistant, multi-layer dielectric, reflective optics. We perform experiments using 1053 nm, 1-60 ps laser pulses with varying beam size, number of shots, and pulse widths in order to understand the characteristics of defects leading to laser-induced damage. This pulse width range spans a transition in mechanisms from intrinsic material ablation for short pulses to defect-dominated damage for longer pulses. We show that for pulse widths as short as 10 ps, laser-induced damage properties of fused silica and silica films are dominated by isolated absorbers. The density of these precursors and their fluence dependence of damage initiation suggest a single photon process for initial energy absorption in these precursors. Higher density precursors that initiate close to the ablation threshold at shorter pulse widths are also observed in fused silica, whose fluence and pulse width scaling suggest a multiphoton initiation process. We also show that these initiated damage sites grow with subsequent laser pulses. We show that scaling laws obtained in more conventional ways depend on the beam size and on the definition of damage for ps pulses. For this reason, coupling scaling laws with the density of precursors are critical to understanding the damage limitations of optics in the ps regime.