The proton exchanged (PE) planar and channel waveguides in a 500 nm thick single-crystal lithium niobate thin film (lithium niobate on insulator, LNOI) were studied. The mature PE technique and strong confinement of light in the LN single-crystal thin film were used. The single mode and cut-off conditions of the channel waveguides were obtained by finite difference simulation. The results showed that the single mode channel waveguide would form if the width of the PE region was between 0.75 μm and 2.1 μm in the β(4) phase. The channel waveguide in LNOI had a much smaller mode size than that in the bulk material due to the high-refractive-index contrast. The mode size reached as small as 0.6 μm(2). in simulation. In the experiment, the refractive index and phase transition after PE in LNOI were analyzed using the prism coupling method and X-ray diffraction. Three different width waveguides (5 μm, 7 μm and 11 μm) were optically characterized. Near-field intensity distribution showed that their mode sizes were 3.3 μm(2).,5 μm(2). and 7 μm(2). The propagation losses were evaluated to be about 16 dB/cm, 12 dB/cm and 11 dB/cm, respectively. The results indicate that PE is a promising method for building more complicated photonic integrated circuits in single-crystal LN thin film.