The regulation of collagenase gene expression by serotonin and progesterone was investigated in primary cultures of rat uterine smooth muscle cells. Northern blot analysis demonstrates that serotonin (5-hydroxytryptamine (5-HT)), when administered to cells in serotonin-depleted medium, causes 6-8-fold increases in levels of collagenase mRNA. Selective serotonin 5-HT2 receptor agonists were able to mimic the effect of the natural hormone, while the induction by serotonin could be blocked by 5-HT2 receptor antagonists. Addition of phorbol ester (PMA) to 5-HT-depleted cultures fully mimicked the effect of 5-HT on collagenase mRNA induction. Treatment with progesterone analogs caused a decrease in collagenase mRNA, even in the presence of saturating levels of serotonin or PMA. In all experiments, levels of secreted collagenase were observed to correspond to levels of collagenase mRNA. Experiments with cycloheximide demonstrate that serotonin- and PMA-induced increases in collagenase mRNA are dependent on protein synthesis. Furthermore, nuclear run-on analysis shows that mRNA increases are accompanied by increases in initiation of transcripts. These data indicate that transcription of collagenase mRNA in myometrial smooth muscle cells is stimulated by serotonin, possibly via activation of protein kinase C, but is in some way prevented by the negative influence of progesterone.