In this work, the spinodal phase demixing of branched comb polymer nanocomposite (PNC) melts is systematically investigated using the polymer reference interaction site model (PRISM) theory. To verify the reliability of the present method in characterizing the phase behavior of comb PNCs, the intermolecular correlation functions of the system for nonzero particle volume fractions are compared with our molecular dynamics simulation data. After verifying the model and discussing the structure of the comb PNCs in the dilute nanoparticle limit, the interference among the side chain number, side chain length, nanoparticle-monomer size ratio and attractive interactions between the comb polymer and nanoparticles in spinodal demixing curves is analyzed and discussed in detail. The results predict two kinds of distinct phase separation behaviors. One is called classic fluid phase boundary, which is mediated by the entropic depletion attraction and contact aggregation of nanoparticles at relatively low nanoparticle-monomer attraction strength. The second demixing transition occurs at relatively high attraction strength and involves the formation of an equilibrium physical network phase with local bridging of nanoparticles. The phase boundaries are found to be sensitive to the side chain number, side chain length, nanoparticle-monomer size ratio and attractive interactions. As the side chain length is fixed, the side chain number has a large effect on the phase behavior of comb PNCs; with increasing side chain number, the miscibility window first widens and then shrinks. When the side chain number is lower than a threshold value, the phase boundaries undergo a process from enlarging the miscibility window to narrowing as side chain length increases. Once the side chain number overtakes this threshold value, the phase boundary shifts towards less miscibility. With increasing nanoparticle-monomer size ratio, a crossover of particle size occurs, above which the phase separation is consistent with that of chain PNCs. The miscibility window for this condition gradually narrows while the other parameters of the PNCs system are held constant. These results indicate that the present PRISM theory can give molecular-level details of the underlying mechanisms of the comb PNCs. It is hoped that the results can be used to provide useful guidance for the future design control of novel, thermodynamically stable comb PNCs.