OBJECTIVE: Consensus development techniques were used in the late 1980s to create explicit criteria for the appropriateness of cholecystectomy. New diagnostic and treatment techniques have been developed in the last decade, so an updated appropriateness of indications tool was developed for cholecystectomy in patients with non-malignant diseases. The validity and reliability of panel results using this tool were tested. METHODS: Criteria were developed using a modified Delphi panel judgement process. The level of agreement between the panelists (six gastroenterologists and six surgeons) was analysed and the ratings were compared with those of a second different panel using weighted kappa statistics. RESULTS: The results of the main panel were presented as a decision tree. Of the 210 scenarios evaluated by the main panel in the second round, 51% were found appropriate, 26% uncertain, and 23% inappropriate. Agreement was achieved in 54% of the scenarios and disagreement in 3%. Although the gastroenterologists tended to score fewer scenarios as appropriate, as a group they did not differ from the surgeons. Comparison of the ratings of the main panel with those of a second panel resulted in a weighted kappa statistic of 0.75. CONCLUSIONS: The parameters tested showed acceptable validity and reliability results for an evaluation tool. These results support the use of this algorithm as a screening tool for assessing the appropriateness of cholecystectomy.