The International Commission on Radiological Protection (ICRP) reference phantoms, developed based on computed tomography images of human bodies, provide much more realism of human anatomy than the previously used MIRD5 (Medical Internal Radiation Dose) mathematical phantoms. It has been, however, realized that the ICRP reference phantoms have some critical limitations showing a considerable amount of holes for the skin and wall organs mainly due to the nature of voxels of which the phantoms are made, especially due to their low voxel resolutions. To address this problem, we are planning to develop the polygon-surface version of ICRP reference phantoms by directly converting the ICRP reference phantoms (voxel phantoms) to polygon-surface phantoms. The objective of this preliminary study is to see if it is indeed possible to construct the high-quality polygon-surface phantoms based on the ICRP reference phantoms maintaining identical organ morphology and also to identify any potential issues, and technologies to address these issues, in advance. For this purpose, in the present study, the ICRP reference male phantom was roughly converted to a polygon-surface phantom. Then, the constructed phantom was implemented in Geant4, Monte Carlo particle transport code, for dose calculations, and the calculated dose values were compared with those of the original ICRP reference phantom to see how much the calculated dose values are sensitive to the accuracy of the conversion process. The results of the present study show that it is certainly possible to convert the ICRP reference phantoms to surface phantoms with enough accuracy. In spite of using relatively less resources (<2 man-months), we were able to construct the polygon-surface phantom with the organ masses perfectly matching the ICRP reference values. The analysis of the calculated dose values also implies that the dose values are indeed not very sensitive to the detailed morphology of the organ models in the phantom for highly penetrating radiations such as photons and neutrons. The results of the electron beams, on the other hand, show that the dose values of the polygon-surface phantom are higher by a factor of 2-5 times than those of the ICRP reference phantom for the skin and wall organs which have large holes due to low voxel resolution. The results demonstrate that the ICRP reference phantom could provide significantly unreasonable dose values to thin or wall organs especially for weakly penetrating radiations. Therefore, when compared to the original ICRP reference phantoms, it is believed that the polygon-surface version of ICRP reference phantoms properly developed will not only provide the same or similar dose values (say, difference <5 or 10%) for highly penetrating radiations, but also provide correct dose values for the weakly penetrating radiations such as electrons and other charged particles.