Format

Send to

Choose Destination
Med Image Anal. 2018 Aug;48:162-176. doi: 10.1016/j.media.2018.06.004. Epub 2018 Jun 15.

Dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks.

Author information

1
The Hamlyn Centre for Robotic Surgery, Imperial College London, London, UK; Department of Computing, Imperial College London, London, UK. Electronic address: xjtuljy@gmail.com.
2
The Hamlyn Centre for Robotic Surgery, Imperial College London, London, UK; Wellcome/EPSRC Centre for Interventional & Surgical Sciences (WEISS), University College London, London, UK; Centre for Medical Image Computing, University College London, London, UK; Department of Computer Science, University College London, London, UK; Department of Surgery and Cancer, Imperial College London, London, UK. Electronic address: n.clancy@ucl.ac.uk.
3
The Hamlyn Centre for Robotic Surgery, Imperial College London, London, UK; Department of Surgery and Cancer, Imperial College London, London, UK.
4
The Hamlyn Centre for Robotic Surgery, Imperial College London, London, UK; Department of Computing, Imperial College London, London, UK.
5
Department of Otolaryngology, Northwick Park Hospital, Harrow, UK.
6
Wellcome/EPSRC Centre for Interventional & Surgical Sciences (WEISS), University College London, London, UK; Centre for Medical Image Computing, University College London, London, UK; Department of Computer Science, University College London, London, UK.
7
Division of Medical and Biological Informatics, German Cancer Research Center, Heidelberg, Germany.
8
The Hamlyn Centre for Robotic Surgery, Imperial College London, London, UK; Department of Surgery and Cancer, Imperial College London, London, UK. Electronic address: daniel.elson@imperial.ac.uk.

Abstract

Surgical guidance and decision making could be improved with accurate and real-time measurement of intra-operative data including shape and spectral information of the tissue surface. In this work, a dual-modality endoscopic system has been proposed to enable tissue surface shape reconstruction and hyperspectral imaging (HSI). This system centers around a probe comprised of an incoherent fiber bundle, whose fiber arrangement is different at the two ends, and miniature imaging optics. For 3D reconstruction with structured light (SL), a light pattern formed of randomly distributed spots with different colors is projected onto the tissue surface, creating artificial texture. Pattern decoding with a Convolutional Neural Network (CNN) model and a customized feature descriptor enables real-time 3D surface reconstruction at approximately 12 frames per second (FPS). In HSI mode, spatially sparse hyperspectral signals from the tissue surface can be captured with a slit hyperspectral imager in a single snapshot. A CNN based super-resolution model, namely "super-spectral-resolution" network (SSRNet), has also been developed to estimate pixel-level dense hypercubes from the endoscope cameras standard RGB images and the sparse hyperspectral signals, at approximately 2 FPS. The probe, with a 2.1 mm diameter, enables the system to be used with endoscope working channels. Furthermore, since data acquisition in both modes can be accomplished in one snapshot, operation of this system in clinical applications is minimally affected by tissue surface movement and deformation. The whole apparatus has been validated on phantoms and tissue (ex vivo and in vivo), while initial measurements on patients during laryngeal surgery show its potential in real-world clinical applications.

KEYWORDS:

3D reconstruction; Deep learning; Hyperspectral imaging; Intra-operative imaging; Structured light; Super-spectral-resolution

PMID:
29933116
DOI:
10.1016/j.media.2018.06.004
[Indexed for MEDLINE]

Supplemental Content

Full text links

Icon for Elsevier Science
Loading ...
Support Center