Send to

Choose Destination
Ann Clin Transl Neurol. 2019 Dec;6(12):2555-2565. doi: 10.1002/acn3.50952. Epub 2019 Dec 3.

Reproducibility of cognitive endpoints in clinical trials: lessons from neurofibromatosis type 1.

Author information

Murdoch Children's Research Institute, Royal Children's Hospital, Parkville, Victoria, Australia.
Department of Paediatrics, Faculty of Medicine, Dentistry and Health Sciences, University of Melbourne, Melbourne, Victoria, Australia.
Center for Neuroscience and Behavioral Medicine, Children's National Health System, Washington, DC.
Division of Oncology, The Children's Hospital of Philadelphia, Philadelphia, Pennsylvania.
Kids Neuroscience Centre, The Children's Hospital at Westmead, Westmead, New South Wales, Australia.
Children's Hospital Education Research Institute, The Children's Hospital at Westmead, Westmead, New South Wales, Australia.
The University of Sydney Children's Hospital Westmead Clinical School, University of Sydney, Westmead, New South Wales, Australia.
Department of Neurology, Boston Children's Hospital, Boston, Massachusetts.
Department of Preventative Medicine, University of Alabama at Birmingham, Birmingham, Alabama.
School of Public Health, University of Alabama at Birmingham, Birmingham, Alabama.
Division of Neurology, The University of Chicago Medicine Comer Children's Hospital, Chicago, Illinois.
Department of Genetics, University of Utah, Salt Lake City, Utah.
Center for Neuropsychology, Boston Children's Hospital, Boston, Massachusetts.
Human Genetics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio.
Department of Psychology, University of Alabama at Birmingham, Birmingham, Alabama.
Department of Pediatrics, University of Texas Southwestern Medical Center, Dallas, Texas.
Department of Neurology, Washington University School of Medicine, St Louis, Missouri.
Department of Neurology, Children's Hospital of Los Angeles, Los Angeles, California.
Department of Genetics, University of Alabama at Birmingham, Birmingham, Alabama.
National Institutes of Health, National Human Genome Research Institute, Bethesda, Maryland.



Rapid developments in understanding the molecular mechanisms underlying cognitive deficits in neurodevelopmental disorders have increased expectations for targeted, mechanism-based treatments. However, translation from preclinical models to human clinical trials has proven challenging. Poor reproducibility of cognitive endpoints may provide one explanation for this finding. We examined the suitability of cognitive outcomes for clinical trials in children with neurofibromatosis type 1 (NF1) by examining test-retest reliability of the measures and the application of data reduction techniques to improve reproducibility.


Data were analyzed from the STARS clinical trial (n = 146), a multi-center double-blind placebo-controlled phase II trial of lovastatin, conducted by the NF Clinical Trials Consortium. Intra-class correlation coefficients were generated between pre- and post-performances (16-week interval) on neuropsychological endpoints in the placebo group to determine test-retest reliabilities. Confirmatory factor analysis was used to reduce data into cognitive domains and account for measurement error.


Test-retest reliabilities were highly variable, with most endpoints demonstrating unacceptably low reproducibility. Data reduction confirmed four distinct neuropsychological domains: executive functioning/attention, visuospatial ability, memory, and behavior. Test-retest reliabilities of latent factors improved to acceptable levels for clinical trials. Applicability and utility of our model was demonstrated by homogeneous effect sizes in the reanalyzed efficacy data.


These data demonstrate that single observed endpoints are not appropriate to determine efficacy, partly accounting for the poor test-retest reliability of cognitive outcomes in clinical trials in neurodevelopmental disorders. Recommendations to improve reproducibility are outlined to guide future trial design.

Supplemental Content

Full text links

Icon for PubMed Central
Loading ...
Support Center