Multispectral photoacoustic oximetry imaging (MPOI) is an emerging hybrid modality that enables the spatial mapping of blood oxygen saturation (SO2) to depths of several centimeters. To facilitate MPOI device development and clinical translation, well-validated performance test methods and improved quantitative understanding of physical processes and best practices are needed. We developed a breast-mimicking blood flow phantom with tunable SO2 and used this phantom to evaluate a custom MPOI system. Results provide quantitative evaluation of the impact of phantom medium properties (Intralipid versus polyvinyl chloride plastisol) and device design parameters (different transducers) on SO2 measurement accuracy, especially depth-dependent performance degradation due to fluence artifacts. This approach may guide development of standardized test methods for evaluating MPOI devices.