We evaluate thermal noise (Brownian motion) in a rigid reference cavity used for frequency stabilization of lasers, based on the mechanical loss of cavity materials and the numerical analysis of the mirror-spacer mechanics with the direct application of the fluctuation dissipation theorem. This noise sets a fundamental limit for the frequency stability achieved with a rigid frequency-reference cavity of order 1 Hz/ square root Hz (0.01 Hz/ square root Hz) at 10 mHz (100 Hz) at room temperature. This level coincides with the world-highest level stabilization results.