To disentangle the temporal profiles of the diffusion and BOLD components of diffusion-weighted functional MRI (DfMRI) during visual activation, we extracted the raw signal from an anatomically defined volume of interest encompassing the visual cortex of 16 subjects. Under the assumption of a linear, time invariant system we were able to define an intrinsic diffusion response function (DRF) from neural tissue, as a counterpart to the hemodynamic response function (HRF) commonly used in BOLD-fMRI. The shape of the DRF response was found to be very similar to the time courses of optical imaging transmittance signals, thought to originate from local geometric changes in brain tissue at the microscopic scale. The overall DfMRI signal response was modeled as the convolution of the stimulation paradigm time course with a DhRF, which is the sum of the DRF and a fractional HRF resulting from residual tissue T2-BOLD contrast. The contribution of the HRF to the DfMRI signal was found to be 26% at peak amplitude, but the DRF component which has a much steeper onset contributed solely at beginning of the response onset. The suitability of this model over the canonical HRF to process DfMRI data was then demonstrated on datasets acquired in 5 other subjects using a rapid event-related design. Some non-linearities in the responses were observed, mainly after the end of the stimulation.