Atrial fibrillation (AF) is the most prevalent cardiac arrhythmia. The atrial beat is irregular during AF, which causes blood flow hardly. This may cause blood clot formation and cardioembolic strokes. Computer-aided devices may assist cardiologists in diagnosing heart rhythm disorders better. From this viewpoint, we attempt to identify the premature atrial complexes (PACs) to predict the occurrence of AF by using electrocardiogram (ECG) spectrograms. Convolutional neural networks (CNN) models such as ResNet and Wide-ResNet are used to predict the prelude of AF. Regularization constraints are used to deal with the imbalanced and small number of samples in the minority premature AF class. Sensitivity regularization investigates small variations in premature AF samples. It highlights more representative features that distinguish the PACs from the normal rhythm. On the other hand, orthogonality regularization removes the interference between negatively correlated feature weights. It places constraints on capturing similar patterns with slight differences. This constraint allows convergence to a better feature representation with fewer weight redundancies. We propose a combination of sensitivity and orthogonality penalty terms to the cost function of ResNet to decrease the overfitting and obtain a superior representation. The re-sampling class distribution method is also utilized to mitigate the issue of imbalanced data. The proposed method shows better AF prediction for highly imbalanced data with a small number of samples.