Send to

Choose Destination
AMIA Annu Symp Proc. 2014 Nov 14;2014:333-42. eCollection 2014.

Developing an eBook-Integrated High-Fidelity Mobile App Prototype for Promoting Child Motor Skills and Taxonomically Assessing Children's Emotional Responses Using Face and Sound Topology.

Author information

Department of Biomedical Informatics, Columbia University, New York, NY ; HIV Center for Clinical and Behavioral Studies, New York Psychiatric Institute and Columbia University, New York, NY.
Department of Biomedical Informatics, Columbia University, New York, NY.
School of Nursing, Columbia University, New York, NY.
School of International and Public Affairs, Columbia University, New York, NY.


Developing gross and fine motor skills and expressing complex emotion is critical for child development. We introduce "StorySense", an eBook-integrated mobile app prototype that can sense face and sound topologies and identify movement and expression to promote children's motor skills and emotional developmental. Currently, most interactive eBooks on mobile devices only leverage "low-motor" interaction (i.e. tapping or swiping). Our app senses a greater breath of motion (e.g. clapping, snapping, and face tracking), and dynamically alters the storyline according to physical responses in ways that encourage the performance of predetermined motor skills ideal for a child's gross and fine motor development. In addition, our app can capture changes in facial topology, which can later be mapped using the Facial Action Coding System (FACS) for later interpretation of emotion. StorySense expands the human computer interaction vocabulary for mobile devices. Potential clinical applications include child development, physical therapy, and autism.

[Indexed for MEDLINE]
Free PMC Article

Supplemental Content

Full text links

Icon for PubMed Central
Loading ...
Support Center