Accurate measurement of longitudinal changes of anatomical structure is important and challenging in many clinical studies. Also, for identification of disease-affected regions due to the brain disease, it is extremely necessary to register a population data to the common space simultaneously. In this paper, we propose a new method for simultaneous longitudinal and groupwise registration of a set of longitudinal data acquired from multiple subjects. Our goal is to 1) consistently measure the longitudinal changes from a sequence of longitudinal data acquired from the same subject; and 2) jointly align all image data (acquired from all time points of all subjects) to a hidden common space. To achieve these two goals, we first introduce a set of temporal fiber bundles to explore the spatial-temporal behavior of anatomical changes in each longitudinal data of the same subject. Then, a probabilistic model is built upon the hidden state of spatial smoothness and temporal continuity on the fibers. Finally, the transformation fields that connect each time-point image of each subject to the common space are simultaneously estimated by the expectation maximization (EM) approach, via the maximum a posterior (MAP) estimation of probabilistic models. Promising results are obtained to quantitatively measure the longitudinal changes of hippocampus volume, indicating better performance of our method than the conventional pairwise methods.