A new method for filtering the coherence map issued from synthetic aperture radar (SAR) interferometric data is presented. For each pixel of the interferogram, an adaptive neighborhood is determined by a region-growing technique driven by the information provided by the amplitude images. Then pixels in the derived adaptive neighborhood are complex averaged to yield the filtered value of the coherence, after a phase-compensation step is performed. An extension of the algorithm is proposed for polarimetric interferometric SAR images. The proposed method has been applied to both European Remote Sensing (ERS) satellite SAR images and airborne high-resolution polarimetric interferometric SAR images. Both subjective and objective performance analysis, including coherence edge detection, shows that the proposed method provides better results than the standard phase-compensated fixed multilook filter and the Lee adaptive coherence filter.