When people make decisions quickly, accuracy suffers. Traditionally, speed-accuracy tradeoffs (SATs) have been almost exclusively ascribed to changes in the amount of sensory evidence required to support a response ("response caution") and the neural correlates associated with the later stages of decision making (e.g., motor response generation and execution). Here, we investigated whether performance decrements under speed pressure also reflect suboptimal information processing in early sensory areas such as primary visual cortex (V1). Human subjects performed an orientation discrimination task while emphasizing either response speed or accuracy. A model of choice behavior revealed that the rate of sensory evidence accumulation was selectively modulated when subjects emphasized accuracy, but not speed, suggesting that changes in sensory processing also influence the SAT. We then used fMRI and a forward encoding model to derive orientation-selective tuning functions based on activation patterns in V1. When accuracy was emphasized, the extent to which orientation-selective tuning profiles exhibited a theoretically optimal gain pattern predicted both response accuracy and the rate of sensory evidence accumulation. However, these relationships were not observed when subjects emphasized speed. Collectively, our findings suggest that, in addition to lowered response thresholds, the performance decrements observed during speeded decision making may result from a failure to optimally process sensory signals.