Accurate and efficient dose calculation is an important prerequisite to ensure the success of radiation therapy. However, all the dose calculation algorithms commonly used in current clinical practice have to compromise between calculation accuracy and efficiency, which may result in unsatisfactory dose accuracy or highly intensive computation time in many clinical situations. The purpose of this work is to develop a novel dose calculation algorithm based on the deep learning method for radiation therapy. In this study we performed a feasibility investigation on implementing a fast and accurate dose calculation based on a deep learning technique. A two-dimensional (2D) fluence map was first converted into a three-dimensional (3D) volume using ray traversal algorithm. 3D U-Net like deep residual network was then established to learn a mapping between this converted 3D volume, CT and 3D dose distribution. Therefore an indirect relationship was built between a fluence map and its corresponding 3D dose distribution without using significantly complex neural networks. Two hundred patients, including nasopharyngeal, lung, rectum and breast cancer cases, were collected and applied to train the proposed network. Additional 47 patients were randomly selected to evaluate the accuracy of the proposed method through comparing dose distributions, dose volume histograms and clinical indices with the results from a treatment planning system (TPS), which was used as the ground truth in this study. The proposed deep learning based dose calculation algorithm achieved good predictive performance. For 47 tested patients, the average per-voxel bias of the deep learning calculated value and standard deviation (normalized to the prescription), relative to the TPS calculation, is 0.17%±2.28%. The average deep learning calculated values and standard deviations for relevant clinical indices were compared with the TPS calculated results and the t-test p-values demonstrated the consistency between them. In this study we developed a new deep learning based dose calculation method. This approach was evaluated by the clinical cases with different sites. Our results demonstrated its feasibility and reliability and indicated its great potential to improve the efficiency and accuracy of radiation dose calculation for different treatment modalities.