As the immunotherapy of cancer comes of age, adding immunotherapeutic agents to radiation therapy has the potential to improve the outcomes for patients with a wide variety of malignancies. Despite the enormous potential of such combination therapy, laboratory data has been lacking and there is little guidance for pursuing novel treatment strategies. Animal models have significant limitation in combining radiation therapy with immunotherapy and some of the limitations of preclinical models are discussed in this article. In addition to the preclinical challenges, radiation therapy and immunotherapy combinations may have overlapping toxicities, and for both types of therapy, early and late manifestations of toxicity are possible. Given these risks, special attention should be given to the design of the specific Phase I clinical trial that is chosen. In this article, we describe several Phase I design possibilities that may be employed, including the 3 + 3 design (also known as the cohort of 3 design), the continual reassessment method (CRM), and the time-to-event continual reassessment method (TITE-CRM). Efficacy end points for further development of combination therapy must be based on multiple factors, including disease type, stage of disease, the setting of therapy and the goal of therapy. While the designs for future clinical trials will vary, it is clear that these two successful modalities of therapy can and should be combined for the benefit of cancer patients.