We present a technique for constructing a "clean" texture map of a partially occluded building facade from a series of images taken from a moving camera. Building regions blocked by trees, signs, people, and other foreground objects in a minority of views can be recovered via temporal median filtering on a registered image mosaic of the planar facade. However, when such areas are occluded in the majority of camera views, appearance information from other visible portions of the facade provides a critical cue to correctly complete the mosaic. In this paper, we apply a robust measure of spread to infer whether a particular mosaic pixel is occluded in a majority of views, and introduce a novel spatiotemporal timeline-based inpainting algorithm that uses appearance and motion cues in order to fill the texture map in majority-occluded regions. We describe methods for automatically training appearance-based classifiers from a coarse motion-based segmentation to efficiently recognize foreground and background patches in static imagery. Results of recovered building facades are shown for various sequences.