We have shown that far-infrared (FIR) spectroscopy is a sensitive probe for detecting the anion-cation interaction strength in imidazolium-based ionic liquids (ILs). Supported by density functional theory (DFT) calculations of frequencies and energies for large IL aggregates, it is observed that frequency shifts stem mainly from varying interaction strength rather than from different reduced masses of the anions. The strongest interaction is observed for the IL containing the acetate anion which is well-known for its effective dissolution power for cellulose.