We study the in vitro dynamics of the malignant brain tumor glioblastoma multiforme. The growing tumor consists of a dense proliferating zone and an outer less dense invasive region. Experiments with different types of cells show qualitatively different behavior: one cell line invades in a spherically symmetric manner, but another gives rise to branches. We formulate a model for this sort of growth using two coupled reaction-diffusion equations for the cell and nutrient concentrations. When the ratio of the nutrient and cell diffusion coefficients exceeds some critical value, the plane propagating front becomes unstable with respect to transversal perturbations. The instability threshold and the full phase-plane diagram in the parameter space are determined. The results are in a qualitative agreement with experimental findings for the two types of cells.