Incomplete polymerization, volumetric shrinkage, and shrinkage stress are among the primary disadvantages of current resin-based dental composites. Generally, any attempt to increase final double bond conversion only exacerbates polymerization shrinkage and stress. The use of dimer acid-derived dimethacrylate (DADMA) monomers in novel dental resin formulations is examined in this article as a potential means to address these disparate goals. A series of high molecular weight DADMA monomers with different functional groups used to connect the C36 dimer acid core and the methacrylates were formulated with urethane dimethacrylate (UDMA) and/or ethoxylated bisphenol A dimethacrylate (Bis-EMA) at various compositions to manipulate comonomer compatibility and polymeric mechanical properties. Along with reaction kinetics, dynamic polymerization shrinkage and shrinkage stress were assessed. Specific DADMA monomers demonstrated limited miscibility with either Bis-EMA or UDMA. Appropriate ternary resin formulations produced homogeneous monomeric mixtures capable of controlled polymerization-induced phase separation (PIPS) to yield heterogeneous final polymers. Reduced polymerization shrinkage and stress along with higher conversion was observed for DADMA ternary systems compared with a bisphenol A glycidyl methacrylate (Bis-GMA)/triethylene glycol dimethacrylate (TEGDMA) resin control. The PIPS process resulted in a modest volume recovery and stress relaxation in the later stages of polymerization. These results indicate that certain dimer acid-derived dimethacrylates possess the potential to replace TEGDMA as a reactive diluent in dental resins that display a favorable and unique combination of properties.