The human serine/threonine protein kinases, Mst1 and Mst2, share considerable homology to Ste20 and p21-activated kinase (Pak) throughout their catalytic domains. However, outside the catalytic domains there are no significant homologies to previously described Ste20-like kinases or other proteins. To understand the role of the nonhomologous regions, we performed a structure/function analysis of Mst1. A series of COOH-terminal and internal deletions indicates that there is an element within a central 63-amino acid region of the molecule that inhibits kinase activity. Removal of this domain increases kinase activity approximately 9-fold. Coimmunoprecipitation assays, the yeast two-hybrid procedure, and in vitro cross-linking analysis indicate that Mst1 homodimerizes and that the extreme COOH-terminal 57 amino acids are required for self-association. Size exclusion chromatography indicates that Mst1 is associated with a high molecular weight complex in cells, suggesting that other proteins may also oligomerize with this kinase. While loss of dimerization alone does not affect kinase activity, a molecule lacking both the dimerization and inhibitory domains is not as active as one which lacks only the inhibitory domain. Comparison of Mst1 and Mst2 indicates that both functional domains lie in regions conserved between the two molecules.