Rapid interpretation of physiological time-series data and accurate assessment of patient state are crucial to patient monitoring in critical care. Algorithms that use artificial intelligence techniques have the potential to help achieve these tasks, but their development requires well-annotated patient data. In this study, we designed a data acquisition system for synchronized collection of physiological time-series data and clinical event annotations at the bedside to support the evaluation of alarm algorithms in real time, and implemented this system in a pediatric intensive care unit (ICU). This system captured vital sign measurements at 1 Hz and 325 clinical alarms generated by the bedside monitor and the 2 instances of false negatives during a monitoring period of 196 hours. The alarm annotations in real time at the bedside indicate that about 89% of these alarms were clinically-relevant true positives; 6% were true positives without clinical relevance; and 5% were false positives. These findings show an improved specificity of the alarm algorithms in the newer generation of bedside monitoring systems and demonstrate that the designed data acquisition system enables real-time evaluation of patient monitoring algorithms for critical care.