In an optical filter based VSB-DD transmission system, semiconductor optical amplifier (SOA) is a promising option to enhance system optical power margin. While, in practical system, the low input saturation power makes the SOA-amplified signal susceptible to the pattern effect, which causes a considerable spectral broadening, thereby influencing the design of VSB filter. In this paper, the relationship between SOA-induced pattern effect and the requirements of the VSB filter is systematically investigated. Firstly, qualitative analysis is given and upper sideband (USB) is proved better than lower sideband (LSB) owing to the suppression of SOA-induced pattern effect. Then, 56Gbps IM/DD PAM4 transmission is experimentally conducted. With respective optimal filter configuration, performance of USB signal is superior to LSB signal in all cases. Results show that USB signal has 1dB sensitivity superiority to LSB signal for 56Gb/s PAM4 after 40km transmission. And in 80km case, only by using USB signal, can HD-FEC limit (3.8 × 10-3) be achieved. Also, we study requirements on other filter parameters, including redundant bandwidth and filter steepness.