Carrier-less amplitude and phase (CAP) modulation can be a good candidate for short range optical communications for considerable computational complexity reduction and simple system structure. In this paper, a detailed investigation on the digital filters in CAP modulation system is presented. An adaptive equalizer based on cascaded multi-modulus algorithm (CMMA) is used for the demodulation at the receiver. The impact of digital filter taps on system performance is investigated through comprehensive simulations and a 10 Gb/s CAP16 modulation system is demonstrated experimentally. The BER performance for different length of fiber link is measured. Compared with back-to-back (BTB) transmissions, 2 dB and 3.5 dB receiver power penalty are observed at BER of 10(-3) for 20 km and 40 km fiber link respectively. It clearly demonstrates the feasibility of the CAP16 modulation for the short range transmission systems.