Framework for Evaluation of Automated Knowledge-Based Planning Systems Using Multiple Publicly Available Prostate Routines

Purpose: To establish a framework for the evaluation of knowledge-based planning routines that empowers new adopters to select systems that best match their clinical priorities. We demonstrate the power of this framework using 4 publicly available prostate routines.

Methods and materials: Four publicly available prostate routines (CCMB, Miami, UCSD, WUSTL) were automatically applied across a 25-patient cohort using Eclipse scripting and a PTV prescription of V81 Gy = 95%. The institutions' routines differed in contouring guidelines for planning target volume (PTV) and organs at risk, beam arrangements, and optimization parameters. Model-estimated dose-volume histograms (DVHs) and deliverable postoptimization DVHs were extracted from plans to calculate average DVHs for each routine. Each routine's average calculated DVH was subtracted from the average DVH for all plans and from the model's average predicted DVH for comparison. DVH metrics for PTV (DMAX, D1%, D99%, DMIN), Rectum (DMAX, V70, V60, V40), Bladder (V75, V40), Femur (DMAX), and PenileBulb (DMEAN) were compared with the average using 2-sided paired t tests (Bonferroni-corrected P < .05). To control for contouring effects, the full analysis was conducted for 2 PTV margin schemas: 5 mm uniform and 3 mm or 7 mm posterior/else.

Results: Calculated plans generally aligned with their routine's DVH estimations, except CCMB organ-at-risk Dmaxes. Dosimetric parameter differences were not significant, with the exception of PTV DMAX (Miami = 111.1% [P < .001]), PTV D99% (Miami = 97.4% [P = .05]; UCSD = 97.4% [P = .03]; CCMB = 98.5% [P = .001]), Rectum V40 (Miami = 19.1% [P < .001]; UCSD = 22.7% [P = .003]; CCMB = 53.5% [P < .001]), and Femur DMAX (WUSTL = 48.6% [P = .001.]; CCMB = 37.9% [P < .001]). Overall, UCSD and Miami had lower rectum doses, and CCMB and WUSTL had higher PTV homogeneity. Conclusions were unchanged with different PTV margin schemas.

Conclusions: Using publicly available knowledge-based planning routines spares clinicians substantial effort in developing new models. Our results allow clinicians to select the prostate routine that matches their clinical priorities, and our methodology sets the precedent for comparing routines for different treatment sites.