Color constancy is often modeled on the assumption that color appearance in natural scenes is a function of the visual system's estimates of surface reflectance. Some stimuli, however, do not look like illuminated surfaces. Instead, they appear to be self-luminous. We hypothesized that the appearance of luminosity occurs when the visual system estimates a reflectance spectrum that is outside the gamut of physically realizable surfaces. To test this idea, we measured luminosity thresholds as a function of stimulus chromaticity and illuminant spectral power distribution. Observers adjusted the luminance of a test patch until it just appeared self-luminous. The test patch was spot illuminated by a computer-controlled projection colorimeter viewed in an experimental room lit diffusely by computer-controlled theater lamps. Luminosity thresholds were determined for a number of test patch chromaticities under five experimental illuminants. The luminosity thresholds define a surface in color space. The shape of this surface depends on the illuminant. We were able to describe much of the luminosity threshold variation with a simple model whose parameters define an equivalent illuminant. In the context of our model, the equivalent illuminant may be interpreted as the illuminant perceived by the observer. As part of our model calculations we generalized the classic notion of optimal stimuli by incorporating linear-model constraints. Given the equivalent illuminant, the model predicts that a patch will appear self-luminous when it is not consistent with any physically realizable surface seen under that illuminant. In addition, we show that much of the variation of the equivalent illuminant with the physical illuminant can be modeled with a simple linearity principle. The fact that our model provides a good account of our data extends the physics-based approach to judgements of self-luminosity. This in turn might be taken as support for the notion that the visual system has internalized the physics of reflectance.