Glioblastoma (GBM) invasion and migration are key biological behaviors leading to refractoriness to current therapies and infiltration into the non‑tumor brain parenchyma. GBM cell migration is strongly dependent on tumor architecture in vivo, which is absent in traditional two‑dimensional (2D) monolayer culture. The present study applied a three‑dimensional (3D) hydrogel model to rebuild the tumor architecture in vitro. Treatment with NSC23766, a specific inhibitor of Ras‑related C3 botulinum toxin substrate 1 (Rac1), inhibited the mesenchymal invasiveness however triggered the amoeboid motility called mesenchymal‑amoeboid transition (MAT). Notably, NSC23766 stimulated U87 GBM cell migration in the 3D hydrogel. However, this compound inhibited cell motility in 2D monolayer culture without tumor architecture for MAT, suggesting the advantage of 3D hydrogel to investigate tumor cell invasion. Due to the inverse interaction of Rac1 and Ras homolog family member A (RhoA) signaling in the transition between mesenchymal and amoeboid morphology, simultaneous treatment of NSC23766 and Y27632 (selective Rho associated coiled‑coil containing protein kinase 1 inhibitor), abolished U87 GBM cell migration through inhibiting MAT and amoeboid‑mesenchymal transition. In addition, Y27632 induced integrin expression which gave rise to the focal adhesion to facilitate the mesenchymal invasion. The results of the present study demonstrated that the 3D hydrogel was a preferable model in vitro to study tumor cell invasion and migration. The combined inhibition of Rac1 and RhoA signaling would be a promising strategy to suppress GBM invasion.