Total internal reflection (TIR) lenses are optical components that are used to collimate the light or to generate a desired uniform illumination. However, most TIR designs do not pay attention to color uniformity, an increasingly important issue for the quality of lighting, and have a considerable dimension, which also limits their applications. In this study, we proposed an effective design principle of color mixing, and a phosphor-converted white LED module integrated with a compact modified free form TIR component was presented and optimized to achieve compact size and high angular color uniformity (ACU). Optimization results indicated that modified LED packages could achieve a smaller size in vertical height (~0.52) within the same horizontal radius, compared with LEDs integrated with classic TIR lenses. Meanwhile, the enhancement of ACU with an optimum appropriate divergence half angle (65°) reached as high as 84% in terms of normalized standard deviation of yellow-blue ratio from 0.888 to 0.429.