Multipotent mesenchymal stem cells (MSCs) have attracted substantial attention in stem cell therapy and tissue engineering due to their ability to be cultured for successive passages and multilineage differentiation. Carbon nanotubes (CNTs) have been proposed to be used as potential biomedical structures for bone formation. Therefore, it is important to study the mechanisms of interaction between MSCs and CNTs. We demonstrated that carboxylated single-walled carbon nanotubes (SWCNTs) and carboxylated multiwalled carbon nanotubes (MWCNTs) inhibited the proliferation, osteogenic differentiation, adipogenic differentiation, and mineralization of MSCs. Oxidative stress assay indicated that reactive oxygen species (ROS) may not be responsible for the observed cytotoxicity of carboxylated CNTs. Quantitative real-time polymerase chain reaction (Q-PCR) experiments confirmed that the expression of osteoblast specific genes and adipocyte differentiation specific genes was greatly attenuated during the differentiation of MSCs in the presence of carboxylated CNTs. TEM images revealed that CNTs might interact with proteins located on the cell membrane or in the cytoplasm, which have a further impact on subsequent cellular signaling pathways. Q-PCR results and Western blot analysis together verified that the inhibition of proliferation and osteogenic differentiation of MSCs may be modulated through a Smad-dependent bone morphogenetic protein (BMP) signaling pathway.