In the context of sedentary middle-aged adults, the present study examined the relationships among domain-specific and global levels of self-esteem over the course of a 20-week exercise program. Additionally, the roles played by physical fitness, body composition, self-efficacy, and exercise participation as possible contributors to changes in physical self-worth were examined. Significant improvements in self-esteem at all levels were discovered with global esteem, physical self-worth, and perceptions of physical condition and attractive body increasing. Tests of the hierarchical structure of self-esteem showed greater improvements in physical condition and physical self-worth than global esteem and the relationships between global esteem and subdomain levels were shown to be mediated by physical self-worth. Hierarchical regression analyses showed changes in ratings of importance to have little impact on changes in physical self-worth. Both changes in efficacy and aerobic capacity were demonstrated to account for modest but significant variation in physical self-esteem. Results are discussed in terms of contemporary models of self-esteem, potential mediators of exercise effects on esteem, and the need to measure the constructs of interest appropriately.