This study explores the effects of native prosodic system and segmental context on the perception of Cantonese tones by Mandarin and Japanese listeners. In Experiment 1, 13 Mandarin and 13 Japanese subjects took part in a two-alternative forced-choice discrimination test of Cantonese tones in different segmental contexts (familiar vs unfamiliar). In Experiment 2, 20 Mandarin listeners participated in a perceptual assimilation task that examined the cross-language perceptual similarity between Mandarin and Cantonese tones. Results showed that Mandarin listeners were comparable to Japanese counterparts in discriminability, but the former attended more to pitch contour differences while the latter were more sensitive to pitch height. Moreover, the effect of segmental context was significant exclusively in the Mandarin group, whereas the Japanese group performed stably across syllables in discriminating Cantonese tones. It seemed that unfamiliar context rendered lower perceptual similarity, which further hindered corresponding discrimination by the Mandarin group. In addition, segmental effects were mainly observed in the assimilation patterns of category goodness or uncategorized-categorized. These findings suggested that non-native tone perception could be modulated by listeners' native prosodic structures in a finer way.