It has been suggested that changes in brain 5-hydroxytryptamine3 receptor function may contribute to some behavior disorders, such as anxiety, schizophrenia and drug abuse. We are using the whole-cell version of the patch-clamp technique to study the function of 5-hydroxytryptamine3 channels in neurons freshly dissociated from rat nodose ganglion. In these cells, 5-hydroxytryptamine elicits an inward current over the concentration range of 0.25-100 microM (EC50 = 2.62 microM) by activating 5-hydroxytryptamine3 receptors. The muscarinic cholinergic antagonist atropine reduced the amplitude of 5-hydroxytryptamine activated inward current in a concentration-dependent manner. Other muscarinic antagonists, scopolamine, dexetimide, the M1 muscarinic receptor antagonist pirenzepine, the M2 receptor antagonist methoctramine and the M3 receptor antagonist 4-DAMP methiodide also inhibited 5-hydroxytryptamine-induced inward current. Atropine did not appear to change the reversal potential of this current. In the presence of 5 microM atropine, the concentration-response curve for 5-hydroxytryptamine current was shifted to the right in a parallel fashion. The EC50 value for 5-hydroxytryptamine was increased from 2.62 to 8.76 microM. Schild plots of increasing atropine and 5-hydroxytryptamine concentrations revealed a pA2 value of 5.74 for atropine (apparent KD = 1.8 microM). These observations suggest that atropine competitively antagonizes the activation of a receptor for the neurotransmitter serotonin, a novel action of muscarinic antagonists in the nervous system. This effect of atropine may contribute to the clinical symptoms seen in severe atropine intoxication.