The purpose of this paper was to extend the extended cardiac-torso (XCAT) series of computational phantoms to include a detailed lung architecture including airways and pulmonary vasculature. Eleven XCAT phantoms of varying anatomy were used in this paper. The lung lobes and initial branches of the airways, pulmonary arteries, and veins were previously defined in each XCAT model. These models were extended from the initial branches of the airways and vessels to the level of terminal branches using an anatomically-based volume-filling branching algorithm. This algorithm grew the airway and vasculature branches separately and iteratively without intersecting each other using cylindrical models with diameters estimated by order-based anatomical measurements. Geometrical features of the extended branches were compared with the literature anatomy values to quantitatively evaluate the models. These features include branching angle, length to diameter ratio, daughter to parent diameter ratio, asymmetrical branching pattern, diameter, and length ratios. The XCAT phantoms were then used to simulate CT images to qualitatively compare them with the original phantom images. The proposed growth model produced 46369 ± 12521 airways, 44737 ± 11773 arteries, and 39819 ± 9988 veins to the XCAT phantoms. Furthermore, the growth model was shown to produce asymmetrical airway, artery, and vein networks with geometrical attributes close to morphometry and model based studies. The simulated CT images of the phantoms were judged to be more realistic, including more airways and pulmonary vessels compared with the original phantoms. Future work will seek to add a heterogeneous parenchymal background into the XCAT lungs to make the phantoms even more representative of human anatomy, paving the way towards the use of XCAT models as a tool to virtually evaluate the current and emerging medical imaging technologies.