In traditional imaging system design, the optical lens is often optimized toward the artificial optimization target like modulation transfer function and field-of-view (FoV). This usually leads to complex stacks of lenses. In order to reduce the complexity, we propose an end-to-end single lens imaging system design method. First, the imaging and processing model is established, whose input end is the ground truth image, and the output end is the restored image by Res-Unet. Then, with the optimization target of minimizing the difference between the restored image and the ground truth image, the parameters of the lens surface and the parameters of the restoration algorithm are optimized simultaneously by deep learning. In order to realize the end-to-end design, the imaging model is required to be differentiable to the lens parameters, so a fast differentiable ray tracing model is proposed. A single lens imaging system with high-quality large FoV (47°) has been designed by the end-to-end method. This method will have a wide application prospects in the design of light and small optoelectronic imaging systems.