Logo of pnasPNASInfo for AuthorsSubscriptionsAboutThis Article
Proc Natl Acad Sci U S A. 1999 Jul 6; 96(14): 7950–7955.
Cell Biology

Time-resolved analysis and visualization of dynamic processes in living cells


Recent development of in vivo microscopy techniques, including green fluorescent proteins, has allowed the visualization of a wide range of dynamic processes in living cells. For quantitative and visual interpretation of such processes, new concepts for time-resolved image analysis and continuous time–space visualization are required. Here, we describe a versatile and fully automated approach consisting of four techniques, namely highly sensitive object detection, fuzzy logic-based dynamic object tracking, computer graphical visualization, and measurement in time–space. Systematic model simulations were performed to evaluate the reliability of the automated object detection and tracking method. To demonstrate potential applications, the method was applied to the analysis of secretory membrane traffic and the functional dynamics of nuclear compartments enriched in pre-mRNA splicing factors.

The development of in vivo microscopy techniques and fluorescent reagents has stimulated interest in studying the dynamics of cellular processes (for review, see refs. 1 and 2). These types of experiments generate large and complex data sets and require tools for visual and quantitative analysis of the observed dynamic processes in space and time.

Imaging fast moving vesicles in living cells at high speed and high spatial resolution generally implies a low signal-to-noise ratio, hampering accurate object detection. As a consequence of the optical aperture problem, tracking of small objects based on visual similarity criteria is difficult because many objects appear very similar (3). Highly sensitive object detection and tracking has been recognized as crucial for an accurate evaluation of such data. However, a quantitative interpretation of trafficking vesicles has been generally based on manual evaluation of a user-biased selection of objects with apparently highest motility. Such an evaluation is very time-consuming and also is limited by the perception of the manual inspector.

Processes in the cell nucleus are much slower and need to be observed over a longer period of time. To avoid disruptions of nuclear processes, the total light exposure during in vivo observation must be minimized. Thus, the signal-to-noise ratio and, more importantly, the number of time series taken in a particular experiment is considerably reduced, leading to a loss in spatio-temporal resolution. Displaying time series as movies is a widely used method for visual interpretation. However, this approach does not improve temporal resolution; that is, additional information about the continuous development of the observed processes between the imaged time steps (subpixel resolution in time) is not obtained. Furthermore, quantitative information is not revealed by such a visual approach. In a first approach to quantitatively describe nuclear dynamics in vivo, single particle tracking (4) has been used to estimate the diffusion of chromatin in living cells of different species (5).

Here, we describe a novel technique for fully automated analysis and time–space visualization of time series from living cells, which involves segmentation and tracking of cellular structures as well as continuous visualization and measurement of quantitative parameters with subpixel resolution in time and space. It will be demonstrated that this approach can reconstruct resolution both in time and space. Hence, quantitative and visual information unachievable with traditional methods can be accessed.


Highly Sensitive Object Detection.

Imaging structures in living cells at high speed and high spatial resolution generally implies a low signal-to-noise ratio (Fig. (Fig.11A), hampering accurate object detection. We developed a two-step procedure for object detection. First, a constrained image smoothing by anisotropic diffusion is performed that removes noise without disturbing essential edge information (Fig. (Fig.11B). Anisotropic diffusion selectively diffuses an image I in regions where the signal is

equation M1

of constant mean, in contrast to those regions where a rapid signal change occurs. The smoothing process within the Perona-Malik model is monitored by an abstract time-scale t; that is, higher values imply stronger filtering (6). The diffusion process depends solely on local image properties and is governed by the shape of the so-called “edge-stopping” function g. Here, an object scale-dependent edge-stopping function based on Tukey’s biweight robust estimator ρ was applied because diffusion with the Tukey norm produces sharper boundaries than the Lorentzian (Perona-Malik) norm (7).

equation M2

equation M3

equation M4

For segmentation, an edge-oriented technique using a concept of local orientation was applied. Based on the smoothed representation, candidate edge pixels are determined by a modified form of the standard non-maximum-suppression algorithm (8), including a weak hysteresis formulation. A pixel is classified as an edge pixel if it has a potential predecessor and successor (hysteresis) and if the magnitude of the gradient is maximal compared with the two neighbors in direction of the gradient (nonmaxima suppression). The first condition assists in the formation of unbroken contours whereas the second inhibits multiple responses to a single edge present in the data (Fig. (Fig.11C).

Figure 1
Detection of vesicles in human chromogranin B–GFP-transfected Vero cells. After release of secretion, 40 images were recorded with a time-lapse of 0.5 sec. (A) Unprocessed image at initial time step. (B) After diffusion filtering, the noise level ...

To obtain closed borderlines, edges were assumed to separate two neighboring homogeneous regions. A region was considered homogenous if its intensity values could be modeled by a Gaussian distribution. Edges were traced based on two parameters, namely local orientation and equal probability of belonging to adjacent regions. For small scale structures, local orientation can be approximated by the direction of gradient. This induces a local coordinate system such that the orthogonal axis aligns with the isophote (line of constant intensity) parallel to the borderline. Because borderlines induce a partition of the image, the second condition implies that edge pixels need to belong to adjacent partitions with equal probability. As a result, closed borderlines enclosing homogeneous regions are obtained. Based on the image partition, a region neighborhood graph is built. Each node of the graph is associated with one region and is assigned morphological parameters such as mean intensity, shape, and size of the respective region. Regions of interest are finally detected as regions with locally maximal intensity (Fig. (Fig.11D).

Dynamic Object Tracking.

For dynamic analysis, cellular objects need to be tracked in time–space. Based on object features such as size, shape, total intensity, or texture, tracking an object amounts to finding its best match in consecutive images. According to the continuity equation of optical flow, corresponding objects in consecutive images should be similar. However, distortions in the imaging process, such as noise, bleaching, and illumination differences, as well as changes in focal position might considerably distort the time–space continuity assumption. Standard region-based matching techniques (9) do not give satisfying results in general. Our methods use a fuzzy logic-based system for image sequence analysis (10) based on the assumption that object features are conserved in an indistinct (fuzzy) sense.

Fuzzy theory assumes that all things are matter of degree (11). Fuzzy systems behave as associative memories mapping close inputs to close outputs without requiring a mathematical description of how the output functionally depends on the input. A fuzzy system relies on a linguistic “rule” encoded in a numerical fuzzy associative memory mapping, the fuzzy associative memory rule. According to a dynamic particle model, the velocity of an object is assumed to remain relatively constant. To compare two objects in consecutive images, differences in velocity and deviation of expected extrapolated position from the potential new position are measured. In addition, differences in total intensity and area are computed and translated into fuzzy rules.

Each of these four parameters activates each fuzzy associative memory rule to different degree mANT. The scalar activation value actj of the fuzzy associative memory rules’ consequent equals the minimum of the four antesequent conjuncts’ values. With correlation-product, encoding the value of the consequent is multiplied by the activation value. By computing the fuzzy centroid, the output is defuzzified to a single numerical value, the composite similarity measure csim. Objects in one image correspond to the object with the highest “defuzzified” similarity measure in the consecutive image.

equation M5

equation M6

equation M7

equation M8

In case no corresponding object with a similarity value below a certain threshold has been found, this object remains unmatched, and the respective object track ends. A correspondence map together with the binarized images are the output of the image sequence analysis module and are used for continuous visualization and quantification in time–space.

Continuous Time–Space Reconstruction.

We developed two different approaches for continuous time–space reconstruction. The dynamics of objects with constant shape over time are well described by the dynamic repositioning of their gravity centers. According to the correspondence map provided by the particle, tracking module discrete trajectories are formed in time–space. By cubic b-spline interpolation between corresponding gravity centers, they are subsequently transformed into continuous trajectories (Fig. (Fig.22A). Cubic b-splines were chosen for interpolation between corresponding points because they are stable in a geometric sense: That is, they do not tend to oscillations even for a large number of sample points. Furthermore, continuous derivatives exist up to the order of two. Note that the first and second derivative correspond to velocity and acceleration, respectively, which are crucial for quantification of dynamics (see below). The reconstruction procedure was embedded in a powerful multidimensional viewer (Open Inventor Scene Viewer, Silicon Graphics, Mountain View, CA; available for almost all hardware platforms), providing a differentiated visualization of a large number of trajectories in time–space (Fig. (Fig.22B).

Figure 2
Time–space tracking of vesicles for the cell shown in Fig. Fig.1.1. (A) Tracking and interpolation between consecutive time steps is demonstrated for 4 of 40 sections at indicated time steps. The sequence of original images is embedded ...

The second approach was designed for visualization of objects dynamically changing their shape over time. This problem accounts for a continuous shape reconstruction from series of two-dimensional images in time–space (12, 13). In a first step, the binarized object representation is transformed into a parameterized contour representation (Fig. (Fig.33 AC). Thereafter, corresponding boundary points in adjacent time sections are found by a global optimization scheme. Under the assumption that the transformation of one contour k1(u) in its adjacent contour k2(v) is sufficiently smooth and leaves the order of contour points,

equation M9

equation M10

equation M11

equation M12

equation M13

unchanged, the optimal transformation is found by minimizing the integral over all Euclidean distances between corresponding contour points.

Figure 3
(AC) A series of three images after induction of transcription of BK-virus was taken at the indicated time steps. Detected speckles are outlined in green. After in vivo imaging, the induced RNA was visualized by fluorescence in situ hybridization. ...

For minimization of this energy term, a recursive contour splitting approach was chosen. A continuous surface reconstruction in time–space is obtained by b-spline interpolation of corresponding boundary points (Fig. (Fig.33D) and is visualized within the graphical scene viewer. Triangles are essential primitives for computer graphical display. Hence, continuous surfaces need to be approximated by triangular meshes. To reduce the complexity of the triangle mesh while maintaining a close approximation to the original b-spline interpolation, a multiresolution strategy for visualization was developed. At each level of resolution, triangles are formed according to the correspondence map in time–space. To obtain a homogenous triangulation in time–space, a triangle is further subdivided only if its maximal displacement from the b-spline interpolated surface exceeds a preset threshold (14).

Quantitative Measurements.

Although multidimensional visualization is crucial for a qualitative evaluation of dynamic processes, morphological and dynamic parameters are required for quantitative measurements. Based on object outlines and the time–space correspondence map, a quantification module was developed. Within this module, morphological parameters such as size and shape as well as dynamic parameters such as path length, velocity, acceleration, mean squared distances, and diffusion coefficients are computed in a fully automated way. An interface to standard statistic software facilitates further evaluation and display of parameters (see below).


Dynamics of Secretory Membrane Traffic.

In a first study, we examined the motility of secretory vesicles mediating biosynthetic transport from the trans-Golgi network to the plasma membrane (15). Green fluorescent protein (GFP) was tagged to the secretory protein human chromogranin B in Vero cells followed by in vivo time-lapse microscopy. After adaptive smoothing and segmentation (Fig. (Fig.1),1), the object information was passed to the image sequence analysis tool for object tracking. Image analysis, graphical preprocessing, and computation of dynamic parameters involving >500 vesicles in 40 time sections was performed in a fully automated way within 60 min on a standard Pentium PC or Silicon Graphics workstation. In comparison, manual evaluation consumed >4 hours of user interaction for a small subset of 40 user selected vesicles with highest motility.

Within the time–space reconstruction module, the resulting 391 trajectories were categorized as stationary, unidirectional, or bi-directional according to their degree of motility (Fig. (Fig.22B). A trajectory was considered as stationary if its mean velocity was below the average velocity of all trajectories. The group of remaining vesicles was subdivided into two classes: vesicles performing unidirectional motion and vesicles reversing their direction of motion. A vesicle was considered as reversing its direction at a particular time step if the velocity vector averaged over three time steps before it differed at least 130° in direction as compared with three time steps afterward.

Trajectories were color-encoded for easy interpretation of motility (Fig. (Fig.22C). Dynamic parameters such as mean squared distances, diffusion coefficients, and velocities were computed (Fig. (Fig.44A). The vast majority (64%) of vesicles were found to be stationary whereas 31% showed periods of fast directed movements interspersed by periods of slow random motion. The average velocity of nonstationary vesicles was 0.587 μm/sec as compared with 0.252 μm/sec for stationary vesicles. The maximum velocity measured for this cell was 1.23 μm/sec (i.e., 6 pixels/time frame; see Table Table1).1). A smaller fraction (5%) showed a reverse in direction, where half of these vesicles reverted their direction by 176–180°.

Figure 4
(A) Time-dependent mean squared distances for the cell shown in Figs. Figs.11 and and22 and a nocodazole treated cell with disrupted microtubules (not shown). The mean squared change in distances was computed as 〈Δd2 ...
Table 1
Simulation and time-space tracking of vesicles

The plots of mean squared distances and diffusion constants show that stationary vesicles in normal cells as well as vesicles in cells with disrupted microtubules after addition of nocodazole undergo Brownian motion (Fig. (Fig.44A). These findings support a trial-and-error model for transport of secretory vesicles. According to this model, vesicles are transported fast but on random tracks simply trying any direction to find a target membrane. The model further suggests that vesicles perform fast directed movements if associated with microtubules and slow random search movements toward the next microtubule after dissociation. Of interest, we also observed small clusters of vesicles apparently performing the same kind of random and directed movement bringing up the question whether those vesicles are associated with larger transport intermediates. The described techniques provide a fast and powerful tool to investigate how secretory traffic is regulated: for example, to study the effect of cell motility or cell–cell contact formation on the transport of secretory vesicles.

Comparison with Model Simulations.

In an attempt to quantitatively access the accuracy of the dynamic object tracking module, we performed a systematic simulation of trafficking vesicles under different dynamic parameter settings. Vesicles were simulated as spots with a Gaussian gray value distribution. In an initial configuration, a fixed number of vesicles was randomly placed into the image. The localization of a vesicle in the next time step depends on four parameters: its initial velocity (vini), its acceleration (a), its direction of motion, and the maximally allowed velocity (vmax). Although settings for initial and maximal velocity as well as acceleration differed among vesicles and experiments, a vesicle was allowed to perform any form of change in its directional motion in all simulations.

A sequence of 40 time steps was simulated under different parameter settings (Table (Table1).1). After automated object detection, vesicles were tracked by the object tracking module as described above (see Methods). Automatically found trajectories were systematically compared with simulated trajectories (Fig. (Fig.44B). Simulation of vesicles with dynamic parameters set to the experimentally determined maximal velocities and acceleration for vesicles in membrane traffic (Table (Table1)1) showed that 72.5% of the simulated vesicles could be exactly tracked by the dynamic object tracking module. Considering the high dynamics of vesicles and the frequent intersection of trajectories in time–space (Fig. (Fig.5),5), only 17.5% of the simulated trajectories were unmatched; the remaining 10% of trajectories were subdivided into two or three independent trajectories:, that is, at one or two time steps, a vesicle was lost. Notably, the overall dynamics as reflected by mean velocities of vesicles was accurately estimated by the automated tracking module (deviation to exact mean velocities <3%; see Table Table1).1).

Figure 5
Simulation and time–space tracking of vesicles under experimental dynamic conditions determined for vesicular membrane traffic (see Fig. Fig.44A and Table Table1).1). (A) Three sections of a series of forty simulated sections at ...

Dynamics of Transcription and Pre-mRNA Splicing.

In a second study, the described technique was applied to study gene expression events in vivo, in particular, the tempo-spatial and functional relationship between transcription and pre-mRNA splicing. In the mammalian cell nucleus, most splicing factors are concentrated in 20–40 distinct domains called speckles. GFP was fused in-frame to the amino terminus of the essential splicing factor 2/alternative splicing factor (SF2/ASF) and was visualized by time-lapse microscopy. Time-lapse microscopy shows that those speckles are highly dynamic structures (16). BKT-1B cells, transfected with GFP–essential splicing fator 2/alternative splicing factor cAMP-inducible early genes of BK virus, were triggered in vivo, followed by time-lapse microscopy.

After automated image analysis, outlines of speckles and BK-induced RNA were computed (Fig. (Fig.33 AC), followed by a continuous time–space reconstruction and computation of surface dynamics for these speckles (Fig. (Fig.33D). The continuous reconstruction shows that one of the speckles extends toward the BK virus gene and intersects the gene signal (Fig. (Fig.33E). Notably, the surface dynamics of all neighboring speckles measured by the average acceleration of surface points (data not shown) at each interpolated time step increases after transcriptional activation and reaches its peak when the first speckle hits the gene. Thereafter, the speckles show a rapid slow-down in surface dynamics and a slight reduction in surface area.

In a different cell, speckles were imaged after addition of α-amanitin, a specific inhibitor of RNA polymerase II. Notably, the speckles rounded up and showed only low surface dynamics (Fig. (Fig.33F). These findings suggest that speckles supply pre-mRNA splicing factors to nearby activated sites of transcription and that the dynamics of speckles is directly related to nearby transcriptional events. We are currently using these tools to quantitatively characterize the surface dynamics and intranuclear positioning of the splicing factor compartments.


We have developed a widely applicable technique for accurate analysis of dynamic processes in living cells. In the first application, we quantitatively characterized vesicular motility in secretory membrane traffic. Here, the method proved particularly useful because it allowed for a fully automated analysis of even complex data involving >500 vesicles in a series of 20 images. In a comparison with simulated data, we quantitatively accessed the reliability of the dynamic object tracking module. Under experimental conditions, the vast majority of vesicles could be accurately tracked. Furthermore, the small population of unmatched vesicles did not influence the overall dynamics, as reflected by the velocity distribution of vesicles. For the study of nuclear processes, the time–space interpolation technique proved particularly useful for investigating the continuous assembly of functional compartments in response to metabolic requirements.

Photon count limitations and/or limitations in total exposure time during in vivo imaging implies restrictions both in spatial and temporal resolution. Our results indicate that the developed methods help to reconstruct tempo-spatial resolution. For fast-moving vesicles, the highly sensitive object detection module reveals vesicles that are difficult or even impossible to detect by visual interactive inspection. For nuclear processes, the visualization module provides a time–space interpolation of intermediate time steps, thus allowing the prediction of dynamic processes between measured image steps.

The mechanisms and forces involved in the assembly and dynamics of functional subcellular compartments in response to metabolic requirements are not well understood at present and need further investigations. The present method has been shown to be highly suited for an accurate quantitative and qualitative study of nuclear and cellular processes with improved tempo-spatial resolution.

Supplementary Material

Supplemental Movies:


We thank B. Jähne for generously supporting the development of the particle tracking program and M. Markoutsakis for technical assistance. We are particularly grateful to W. Jäger for his support and helpful comments concerning mathematical image analysis. This work was supported by Bundesministerium für Bildung und Forschung Grant 01 KW 9621 and Deutsche Forschungsgemeinschaft Grant Ja 395/6-2. D.L.S. is supported by National Institute of General Medical Sciences Grant 42694, W.T. is supported by the Graduiertenkolleg at the Interdisziplinäres Zentrum für Wissenschaftliches Rechnen, and T.M. is supported by the Roche Research Foundation.


GFPgreen fluorescent protein


1. Lamond A I, Earnshaw W C. Science. 1998;280:547–553. [PubMed]
2. Lippincott-Schwartz J L, Smith C L. Curr Opin Neurobiol. 1997;7:631–639. [PubMed]
3. Bornfleth H, Sätzler K, Eils R, Cremer C. J Microsc (Oxford) 1998;189:118–136.
4. Qian H, Sheetz M P, Elson E L. Biophys J. 1991;60:910–921. [PMC free article] [PubMed]
5. Marshall W F, Straight A, Marko J F, Swedlow J, Dernburg A, Belmont A, Murray A W, Agard D A, Sedat J W. Curr Biol. 1997;7:930–939. [PubMed]
6. Perona P, Malik J. Proceedings of the IEEE Computer Society Workshop on Computer Vision. Washington, DC: IEEE Computer Society Press; 1987. pp. 16–22.
7. Black M J, Sapiro G, Marimont D, Heeger D. IEEE Trans Image Process. 1998;7:421. [PubMed]
8. Nevatia R, Babu K R. Comput Graph Image Process. 1980;13:257–269.
9. Anandan P. Int J Comput Vis. 1989;2:283–310.
10. Hering F, Leue C, Wierzimok D, Jähne B. Exp Fluids. 1997;23:472–482.
11. Nauck D, Klawonn F, Cruse R. Foundations of Neuro-Fuzzy-Systems. New York: Wiley; 1997.
12. Eils R, Dietzel D, Bertin E, Schröck E, Speicher M R, Ried T, Robert-Nicoud M, Cremer C, Cremer T. J Cell Biol. 1996;135:1427–1440. [PMC free article] [PubMed]
13. Eils R, Sätzler K. In: Handbook of Computer Vision and Applications. Jähne B, Haussecker H, Geissler P, editors. San Diego: Academic Press; 1999.
14. Hoppe H. Progressive Meshes: Proceedings of SIGGRAPH ’96, Computer Graphics Proceedings, Annual Conference Series. Reading, MA: Addison–Wesley; 1996. pp. 99–108.
15. Wacker I, Kaether C, Krömer A, Migala A, Almers W, Gerdes H H. J Cell Sci. 1997;110:1453–1463. [PubMed]
16. Misteli T, Cáceres J F, Spector D L. Nature (London) 1997;387:523–527. [PubMed]

Articles from Proceedings of the National Academy of Sciences of the United States of America are provided here courtesy of National Academy of Sciences
PubReader format: click here to try


Save items

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...


Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...