|
|
Lect Notes Comput Sci. Author manuscript; available in PMC 2008 June 26. Published in final edited form as: Lect Notes Comput Sci. 2006; 4179(LNCS): 920–932. | PMCID: PMC2440722 NIHMSID: NIHMS20371 |
Robust Tracking of Migrating Cells Using Four-Color Level Set Segmentation ![[large star]](/corehtml/pmc/pmcents/x2605.gif) Sumit K. Nath, Filiz Bunyak, and Kannappan Palaniappan MCVL, Department of Computer Science, University of Missouri-Columbia, MO, USA Understanding behavior of migrating cells is becoming an emerging research area with many important applications. Segmentation and tracking constitute vital steps of this research. In this paper, we present an automated cell segmentation and tracking system designed to study migration of cells imaged with a phase contrast microscope. For segmentation the system uses active contour level set methods with a novel extension that efficiently prevents false-merge problem. Tracking is done by resolving frame to frame correspondences between multiple cells using a multi-distance, multi-hypothesis algorithm. Cells that move into the field-of-view, arise from cell division or disappear due to apoptosis are reliably segmented and tracked by the system. Robust tracking of cells, imaged with a phase contrast microscope is a challenging problem due to difficulties in segmenting dense clusters of cells. As cells being imaged have vague borders, close neighboring cells may appear to merge. These false-merges lead to incorrect trajectories being generated during the tracking process. Current level-set based approaches to solve the false-merge problem require a unique level set per object (the N-level set paradigm). The proposed approach uses evidence from previous frames and graph coloring principles and solves the same problem with only four level sets for any arbitrary number of similar objects, like cells. Understanding behavior of migrating cells is becoming an emerging research area with many important applications. Behavior of migrating cells are important parameters of interest in understanding basic biological processes such as tissue repair, metastatic potential, chemotaxis, differentiation or analyzing the performance of drugs. Accurate segmentation and tracking of cells are vital steps in any cell behavior study. In this paper, we present an automated cell segmentation and tracking system designed to study migration of cells imaged with a phase contrast microscope. Segmentation is performed using active contour level set methods with a novel extension that efficiently prevents false-merge problem. Tracking is done by resolving frame to frame correspondences between multiple cells using a multi-distance, multi-hypothesis algorithm. Cells that move into the field-of-view, arise from cell division or disappear due to apoptosis are reliably segmented and tracked by the system. Simultaneous tracking of multiple cells imaged with a phase contrast microscope is a challenging problem due to difficulties in segmenting dense clusters of cells. As cells being imaged have vague borders, close neighboring cells may appear to merge. These false-merges lead to incorrect trajectories being generated during the tracking process. Other challenges for tracking include high number of cells, non-linear motion, lack of discriminating features, mitosis (cell division), and fragmentation during segmentation. In [ 1], Chan and Vese presented an algorithm to automatically segment an image I( y) into two distinct regions (or phases) by minimizing a minimal partition Mumford-Shah functional. A multiphase variant of the same algorithm was also proposed to handle 2 n unique phases [ 2]. However, as observed by Zhang et al., [ 3], Dufour et al., [ 4] and, Zimmer and Olivo-Marin [ 5], the two variants of the Chan and Vese algorithm are unsuitable for reliable cell segmentation due to the problem of apparent merges in cells. Zhang et al., proposed a N–level set framework with an implicit coupling constraint to reliably segment cells in an image sequence [ 3]. While this alleviates the problem of apparent merging of cells, it is computationally expensive to implement. The approach we propose uses evidence from previous frames and graph coloring principles and solves the same problem with only four level sets for any arbitrary number of similar objects. The organization of this paper is as follows. Section 2 describes the segmentation module. Salient features of our four-color level set segmentation algorithm are presented along with the related work, variants of the Chan and Vese level set algorithms and N–level set variant of Zhang et al., algorithm [ 3]. Section 3 describes the tracking module. Comparative results and a discussion are presented in Section 4, while a conclusion is presented in Section 5. 2 Cell Segmentation Using Active Contour Level Set Methods Accurate segmentation of individual cells is a crucial step in robust tracking of migrating cells as both over-segmentation (fragmentation) and under-segmentation (cell clumping) produce tracking errors (i.e., spurious or missing trajectories and, incorrect split and merge events). In this section, we describe three different level set segmentation methods and compare their performance for separating closely adjacent and touching cells in a dense population of migrating cells. The three techniques described in this section are all based on the “active contour without edges” energy functional with appropriate extensions, and include multi-phase Chan and Vese [ 2] (CV2LS), N-level sets with energy-based coupling by Zhang et al., [ 3] (ZZNLS), and our novel four-color level sets with energy-based and explicit topological coupling [ 6] (NBP4LS-ETC). The latter two techniques use coupling constraints in order to prevent the merging of adjacent cells when they approach or touch each other. - – CV2LS: In order to segment multiple (i.e., N distinct) objects, Vese and Chan extended their previous 2-phase algorithm [1] by using
![[left ceiling]](/corehtml/pmc/pmcents/lceil.gif) log2N![[right ceiling]](/corehtml/pmc/pmcents/rceil.gif) level sets [2]. The corresponding energy functional Epc (c, Φ)
where, N is the number of phases (i.e., regions in the image) associated with log 2N level set functions, I is the gray-level image being segmented, Φ is a vector of level set functions, c is a vector of mean gray-level values (i.e., ci = mean( I) in the class i), χi is the characteristic function for each class i formed by associated Heaviside functions H( ![[var phi]](/corehtml/pmc/pmcents/x03C6.gif) i), and ( μi, νi) are constants associated with the energy and length terms of the functional, respectively. The log 2N level set formulation improves on the performance of a single level set function, as more number of objects with varying intensities can be efficiently classified. But does not prevent under-segmentation (i.e. incorrect merges), when the objects have similar intensities (i.e. cells). - – ZZNLS: To overcome the drawbacks of classical Chan and Vese level set formulations, while at the same time solving the problem of apparent merging of cells during tracking, Zhang et al., [3] proposed a new model using N–level sets for segmenting cells. Here N is the number of cells at a given time instance. An a priori knowledge that cells do not merge during the evolution process was used to guide the segmentation process. This was achieved by a pair-wise energy-based coupling constraint on the level sets evolution process. A similar formulation was used by Dufor et al., in 3D cell segmentation [4].
The energy functional, Enls( cin, cout, Φ), used to solve the evolution of N–level sets is given by [ 3]: Here, Φ = [ i:i=1…N] represents N–level sets associated with N cells in the image; cin represents average intensities of cells for H( i) ≥ 0 while cout is the average intensity of the background 1. The first term of the functional penalizes pairwise couplings between level sets, while the second term controls the length of i. μin, μout, γ, ν are constants associated with the functional. - – NBP4LS-ETC: The N–level set formulation described previously is able to overcome the apparent merging of neighboring cells. However this approach is not very scalable and is computationally expensive since for N objects it requires N2/2 couplings. To overcome the computational cost, while still preventing the incorrect merges, we propose an optimized version of the N–level set algorithm described previously.
Our optimization is based on the fact that only neighboring cells can potentially merge. Through Delaunay triangulation the cell-to-cell neighborhood relationships are identified and represented in a graph where vertices represent the cells and edges represent the neighborhood relations. The four-color theorem [ 7, 8, 9] states that any planar graph is four-colorable such that no two neighboring vertices have the same color. Thus, four rather than N–level sets would suffice to classify N–objects (i.e., cells) in an image while insuring that neighboring objects do not share the same level set. In order to evolve the four level sets we propose minimizing an energy functional, Efc( cin, cout, Φ), shown in Eq. 2. The first two terms of the right-hand side of Eq. 2 are used to compute average intensities ( cin, cout) within each level set, and outside all level sets, respectively. Using an a priori assumption that all the foreground objects (i.e., cells) in the image have very similar characteristics, we use a single average intensity cin (i.e., ∀ i,
 ). Whereas Zhang et al., model of computing average intensities for each cell ( Eq. 2). The third term helps in minimizing the length of all level sets; the fourth term is the energy-based coupling constraint, used previously, in Eq. 2. The last term enforces the constraint of |Δ i| = 1, thus helping us avoid explicit redistancing of level sets during the evolution process [ 10]. Regularized Heaviside and Dirac-delta functions, proposed by Chan and Vese in [ 1], are also used in our energy functional. μin, μout, ν, γ, η, are constants associated with the functional. The four Euler-Lagrange evolution equations associated with the minimization of Eq. 2 are as follows ( i = 1, 2, 3, 4): where, Δ is the Laplacian operator. In addition to the energy-based coupling technique of ZZNLS [ 3] to penalize overlaps between level sets, we use an explicit topological coupling technique. First, we compute δ( i); i ![[set membership]](/corehtml/pmc/pmcents/x2208.gif) [ 1, 4]. As we use a narrow-band approach (i.e., δ( i) > sthresh) to update the level set curves we check the saliency of δ( i)i.e., δ( i) > δ( i); j ≠ i. This helps us identify pixels on the front of the current level set that may lie on narrow-band fronts of other level sets. A pixel on the front of a current level set is updated only if this saliency test is satisfied. If however a “collision” is detected between cells, then the evolution of level sets near the “collision” region stops. To speed up convergence as in [ 11] and [ 12] we use level set segmentation from a previous frame as an initial estimate for a current frame. For details on implementing the four-level set algorithm, we direct the reader to [ 6]. 3 Multiple Cell Tracking Using Correspondence Graphs Tracking is a fundamental step in the analysis of long term behavior of migrating cells. In this section we present a detection based cell tracking algorithm that extends our previous work in [ 13]. Tracking is done by resolving frame to frame correspondences between multiple cells segmented using active contour level set methods as described in Sec.2. 3.1 Summary of Tracking Algorithm - For each frame I(y, t) at time t, the tracking module receives four foreground mask layers, Ωk(t), that correspond to level sets from the “four-color” segmentation algorithm. In order to refine cell boundaries and remove spurious regions, morphological operations (e.g., opening, closing) may be performed on these foreground layers.
- Connected component analysis is performed on all refined foreground layers such that each Ωk(t) is partitioned into nk disjoint regions
that ideally correspond to nk individual cells. - For each disjoint region Ωk, i, features such as bounding box, centroid, area, and support map are extracted. Region information from all four layers are combined and relabeled as
without merging connected regions from different foreground layers. Keeping regions from different foreground layers distinct, even when they are spatially connected, preserves the identities of previously disjoint cells, thus preventing false trajectory merges. - Relabeled region information is arranged in an “Object-Match” graph structure
 that is used in tracking (Fig. ??). Nodes in the graph represent objects Ωi(t), while edges represent object correspondences. - Correspondence analysis searches for potential object (cell) matches in consecutive frames.
 is updated by linking nodes corresponding to objects in frame I(y, t) with nodes of potential corresponding objects in frame I(y, t − 1). The confidence value
 for each match is stored with each link. This is explained further in the following sub-section. - The trajectory generation and validation module analyses the
 graph and generates valid cell trajectories.
3.2 Cell-to-Cell Correspondence Cell-to-cell matching (correspondence) is performed using a multi-stage overlap distance
 , which consists of three distinct distance functions
 ,
 , and
 for three different ranges of cell motion. - – The inter-bounding-box distance
 quantifies the long-range displacement between two regions (cells) in consecutive frames, by the distance between their bounding boxes defined as the minimum Euclidean distance between corner pairs of the two bounding boxes.
Let
 and
 , indicate the kth and lth corners of the bounding box covering Ω i(t) and Ω j(t − 1).
 can then be defined as where, ηb is a constant. - – The inter-mask distance
 quantifies the mid-range displacement between two regions (cells) in consecutive frames, by the minimum contour to contour distance. This distance is computed in terms of the minimum number of dilations needed to overlap the two regions as
where, ![[plus sign in circle]](/corehtml/pmc/pmcents/x2295.gif) k denotes k-times dilation, sI denotes unit structuring element, and ηm is a constant. - – The tonal-weighted overlap distance
 quantifies the small-range displacement between two regions (cells) by the degree of their overlap, in terms of shape and tonal dissimilarities. In order to emphasize overlap in nuclei (i.e., dark regions with low intensity values), and to de-emphasize cytoplasm overlap (i.e., light regions with high intensity values), overlapping and non-overlapping regions are weighted by local tonal differences between two regions as
where, the intensity images Ii( y) = Ii( y, t) and Ij ( y) = Ij ( y, t − 1), and are scaled such that I [0, 1]. ηo is a constant. The first two terms in the numerator of Eq. 6 account for the distance due to uncovered regions in frames at time instants t and t − 1, respectively. The complement of intensity images are used to obtain higher distances for uncovered low intensity regions (i.e., nuclei). The third term in the numerator accounts for the intensity dissimilarity within the overlapping region. The denominator is used to normalize the distance by the area of the two cells being compared.
 can be assigned any of the three distance measures described above, as per the following rules, The proposed multi-stage distance measure depends on size and shape similarity of the compared regions, besides their proximity, and thus have several advantages over the widely used centroid distance measure. A particularly important case for cell tracking is mitosis (i.e., cell division). During mitosis, epithelial cells often become elongated, subsequently splitting across the minor axis. This produces a big increase in the centroid distance and the distances between a cell and its children become comparable to the distances between a cell and its neighboring cells (). In such a scenario, a low gating threshold would result in parent-to-children matches being discarded, resulting in discontinuities in cell trajectories. However, if a high gating threshold is used, correspondence ambiguities may arise. The proposed multi-stage overlap distance measure overcomes these problems. In addition to object separation based measures, shape (contour) similarity metrics can also be added to the distances described above as additional matching criteria. Since for cell tracking we are primarily interested in the displacement parameter, those metrics are not used in this study but they will be considered in future. During tracking a match matrix
 and a confidence matrix
 for each frame, I( y, t), are produced.
 indicates whether the ith object in I( y, t), corresponds to the jth object in I( y, t − 1).
 indicates the confidence of this match and consist of two components, - – Similarity confidence,
 , is a measure of the similarity between the matched objects and is defined as
where,
 is a user defined constant used to normalize
 . - – Separation confidence,
 , measures the competition between possible matches for the current object and is defined as
where, Ωj indicates the current candidate being compared with Ωi, and Ωj* is its closest competitor in terms of distance. This measure favors matches without competitors, and matches with competitors having higher distances. Unfeasible correspondences are eliminated using confidence values. Absolute pruning eliminates matches whose confidence values are below a certain threshold, while relative pruning eliminates matches whose confidence values are below a percentage of the confidence for the best match. 3.3 Trajectory Generation and Validation Trajectory segments are generated from
 using a multi-hypothesis testing approach with delayed decision. Besides one-to-one object matches, the proposed tracking algorithm supports many-to-one, one-to-many, many-to-many, one-to-none, or none-to-one matches that may result from false detections or associations, segmentation errors, occlusion, entering, exiting, or division of cells. The segment generation module analyzes match information by classifying the nodes of
 (cells) into nine types; single, source, source-split, sink, inner, split, sink-merge, merge-split, merge, based on the number of parent and child objects. A data structure (Segment-List) is formed by identifying and organizing a linked list (Trajectory-Segments) of inner objects starting with a source or split type cell and ending with a merge or sink type cell. Extracted segments are labeled using a method similar to connected component labeling. Not all the detected segments correspond to actual cell trajectories. Trajectory validation unit checks the validity of each segment based on criteria such as duration, length, linearity, size of the corresponding object, parent and children segments etc. and filters out invalid segments. Trajectories are formed by linking unfiltered segments sharing the same label. Discontinuity resolution is also done in this unit using Kalman filter prediction. The proposed cell segmentation and tracking system has been tested on a wound healing image sequence consisting of 136 frames of dimensions 300×300 (40 μm×40 μm) with image intensities I [0, 255]. The sequence has been obtained using a monolayer of cultured pig epithelial cells, as described by Salaycik et al., in [ 14]. Images were sampled uniformly over a 9:00:48 hour period and acquired using a phase contrast microscope, with a 10× objective lens, and at a resolution of approximately 0.13 μm per pixel. Three segmentation algorithms have been implemented: a multi-phase Chan and Vese level set algorithm (CV2LS); our four-level set algorithm with only energy-based coupling (NBP4LS-EC); and our four-level set algorithm with energy-based and explicit topological coupling NBP4LS-ETC. The tracking algorithm described in Sec. 3 has been applied to the three sets of masks obtained from these segmentation algorithms, and the results have been compared. For all three segmentation algorithms the following parameters have been used: μin = 1, μout = 1 ,ν = 1.0/(255.0) 2 and the number of iterations for each frame has been set to a fixed number K = 15. For the segmentation algorithms with energy-based coupling constraint γ has been set to 0.1. During tracking, a relative pruning rate of 85% has been used and matches with confidence values below 85% of
 have been pruned. (
 indicates the confidence of the best match for the current object). The tracking results have been filtered by object size and segment length. Size threshold Tos has been set to 30 pixels, and duration threshold Tsl has been set to 5 frames. Incomplete objects at the image borders (i.e., within 20 pixels from each side) have been excluded from the statistics. Representative results for our segmentation (NBP4LS-ETC) and tracking algorithms are given in and . shows segmented cells and their neighborhood relationships in the form of a Delaunay graph. shows the cell trajectories obtained after tracking. Table 1 shows tracking results obtained using the tracking algorithm described in Section 3, on masks produced by three different segmentation algorithms, CV2LS, NBP4LS-ECand NBP4LS-ETC. T-O indicates the total number of disjoint objects (i.e., cells) detected in all frames of the image sequence. T-S indicates the number of trajectory segments that split from another trajectory segment. Such splits may result from cell division, or fragmentation during segmentation. T-M indicates number of trajectory segments that merge with other trajectory segments. T-A and T-D indicate numbers of trajectory segments that appear or disappear, unexpectedly. These may occur when no evidence of cells in the current frame exists in previous frames, or when no match exists in future frames for cells present in the current frame. The NBP4LS-ETC algorithm does a better job than either the NBP4LS-EC or the CV2LSalgorithm in preventing false-merges of cells. Fragmentation is a problem with both the NBP4LS-ETC and NBP4LS-EC algorithms. But as shown in Table 1, the NBP4LS-ETCresults in a smaller number of splits than either NBP4LS-ECor CV2LS algorithms. As shown in , false splits and merges may arise when a cell is fragmented in the current frame, or becomes fragmented in a future frame. Proposed method removes fragments at the image level through post-processing of small objects using morphology and at the trajectory level through filtering short segments. | Table 1Tracking results, when using three different segmentation algorithms. The number of frames in the sequence = 136, with dimensions of each frame equal to 300×300. |
and show two cases that demonstrate the advantage of our NBP4LS-ETC method. shows the evolution of three cells labeled as A, B, and C. In the sequence from frame #78 to #111 two of the cells undergo mitosis (cell division). Cell C divides first followed by B; between frames #78 and #103 cell C splits into children cells C1 and C2; between frames #103 and #111 cell B splits into children cells B1 and B2. Both algorithms CV2LS (2nd row), and NBP4LS-ETC (3rd row) correctly identify the mitosis event of cell C. But only NBP4LS-ETCcorrectly identifies the mitosis event of cell B. Using the CV2LSmethod, at frame #103 cell B that is clearly distinct in frame #78, gets falsely merged with cell A, because cell B becomes indistinct (the nucleus is not as thick as preparation for DNA replication) and at the same time the region between cells A and B gets darker as these cells move closer together. This false merge causes additional tracking complications when cell B undergoes subsequent mitosis. As seen in frame #111, cell A appears to undergo mitosis and is incorrectly associated with the children A1 and A2. Cell A2 is associated with parent cell A instead of parent B due to a correspondence error; it should be labeled as cell B2. Object A1 is associated with parent object A due to both correspondence and segmentation errors; the segmentation of object A1 merges two actual cells (labeled A and B1 in row 3) which leads to the association error during tracking. However using the NBP4LS-ETCalgorithm the explicit topological coupling constraint prevents cells A and B from merging and the mitosis of cell B is correctly detected and tracked. Effects of fragmentation such as the small region above cell A in frames #103 and #111 are handled by postprocessing and trajectory filtering and did not cause any tracking errors. depicts an “absorption event” where one level set pushes a neighboring level set out of its own path of evolution. Using only an energy-based coupling term (2nd row - NBP4LS-EC) leads to the shifting of the boundary between adjacent objects. Ultimately one object “ absorbs” the other without a merge event (where adjacent level sets merge together during contour evolution). Using the proposed additional explicit topological coupling term as described, the location of the boundary is maintained which prevents absorption. The absorption process may be order dependent but we have found that randomization of level set processing order is not sufficient to prevent absorption. We have presented an automated cell segmentation and tracking system designed to study migration of cells imaged with a phase contrast microscope. Cells that move into the field-of-view, arise from cell division or disappear due to apoptosis are reliably segmented and tracked by this system. The novel four-color level set formulation introduced to deal with the false-merge problem in segmentation and tracking of dense cell clusters, is very scalable and significantly reduces the computational complexity of N -level set formulation of Zhang et al., [ 3]. Experimental results show that segmentation with the proposed four-level set formulation, with an explicit topological coupling constraint, greatly improves accuracy of trajectories obtained during cell tracking. Further research on trajectory validation and behavior analysis is currently in progress. 1. Chan T, Vese L. Active contours without edges. IEEE Trans Image Process. 2001;10:266–277. [PubMed] 2. Vese L, Chan T. A multiphase level set framework for image segmentation using the Mum-ford and Shah model. Intern J Comput Vis. 2002;50:271–293. 3. Zhang B, Zimmer C, Olivo-Marin J-C. Tracking fluorescent cells with coupled geometric active contours. Proc 2nd IEEE Int Symp Biomed Imaging (ISBI); Arlington, VA. 2004. pp. 476–479. 4. Dufour A, Shinin V, Tajbakhsh S, Guillén-Aghion N, Olivo-Marin JC, Zimmer C. Segmenting and tracking fluorescent cells in dynamic 3-D microscopy with coupled active surfaces. IEEE Trans Image Process. 2005;14:1396–1410. [PubMed] 5. Zimmer C, Olivo-Marin JC. Coupled parametric active contours. IEEE Trans Pattern Anal Machine Intell. 2005;27:1838–1842. 6. Nath S, Palaniappan K, Bunyak F. Cell segmentation using coupled level sets and graph-vertex coloring. In: Larsen R, Nielsen M, Sporring J, editors. LNCS-Proc MICCAI 2006; Springer-Verlag. 2006. 7. Appel K, Haken W. Every planar map is four colorable. Part I. discharging. Illinois J Math. 1977;21:429–490. 8. Appel K, Haken W, Koch J. Every planar map is four colorable. Part II. reducibility. Illinois J Math. 1977;21:491–567. 9. Robertson N, Sanders DP, Seymour PD, Thomas R. The four color theorem. J Combin Theory, Ser B. 1997;70:2–44. 10. Li C, Xu C, Gui C, Fox D. Level set evolution without re-initialization: A new variational formulation. Proc IEEE Conf Computer Vision Pattern Recognition. 2005;1:430–436. 11. Mukherjee DP, Ray N, Acton ST. Level set analysis for leukocyte detection and tracking. IEEE Trans Image Process. 2001;13:562–672. [PubMed] 12. Paragiois N, Deriche R. Geodesic active contours and level sets for the detection and tracking of moving objects. IEEE Trans Pattern Anal Machine Intell. 2000;22:266–280. 13. Bunyak F, Palaniappan K, Nath SK, Baskin TI, Dong G. Quantitive cell motility for in vitro wound healing using level set-based active contour tracking. Proc 3rd IEEE Int Symp Biomed Imaging (ISBI); Arlington, VA. 2006. 14. Salaycik KJ, Fagerstrom CJ, Murthy K, Tulu US, Wadsworth P. Quantification of micro-tubule nucleation growth and dynamics in wound-edge cells. J Cell Sci. 2005;118:4113–4122. [PubMed]
|
