Breast tumor classification with multiple medical reports such as B-ultrasound, Mammography (X-ray) and Nuclear Magnetic Resonance Imaging (MRI) is crucial to the intelligent cancer diagnosis system. Unlike the other domain texts, the medical reports have latent hierarchical syntactic structures and have hidden rich semantic information about the entities and relationships, which poses a great challenge of breast cancer classification. In this article, we proposed a Knowledge-powered Deep Breast Tumor Classification model (KDBTC), which takes the semantic information as a kind of prior knowledge and incorporated it into deep neural networks. Specially, our proposed model first uses Hierarchical Attention Bidirectional Recurrent Neural Networks (HA-BiRNNs) to encode the syntax-aware representation of medical reports in a hierarchical way. In the HA-BiRNN, a hierarchical neural network structure, consisting in two encoder layers of BiRNN (Bidirectional Recurrent Neural Networks), mirrors the hierarchical structure of medical reports, and a hierarchical attention mechanism, consisting of two levels attentions, attends to important elements within clinical report with word-level attention and sentence-level attention. Secondly, our model obtains the semantic information relevant to the medical reports from the clinical domain semantic tree, and encodes the semantic representation of medical reports by using Tree Structured Recurrent Neural Network with gated recursive units (Tree-GRUs). Finally, we classify breast tumors by combining both the syntax and semantic representations of medical reports. We evaluate our method on the real-world breast cancer medical reports, and results show that our method achieves higher performance on breast cancer classification.