New Technologies for Cervical Cancer Screening

Publication Details

Executive Summary

Invasive cervical cancer was once the leading cause of cancer death in women in the United States. Today cervical cancer accounts for less than 3 percent of all cancers in women. A 70 percent decrease in cervical cancer deaths over the past 50 years has been credited in large part to regular screening with Papanicolaou (Pap) testing. 1 Available data indicates that in the late 1930s and early 1940's cervical cancer accounted for 25-30 deaths per 100,000 women. 2 In 1992, the death rate for cervical cancer was 3 per 100,000 women. 3 Organizations including the National Cancer Institute 4 and the Centers for Disease Control and Prevention 5 state that the death rate for cervical cancer would be even lower if all women received periodic Pap tests. There are concerns about false-negative rates associated with manual screening of Pap tests among the public and medical community.

Direct consumer marketing by manufacturers of new Pap test technologies has raised questions about the comparable value of existing and new methods of testing. In addition, questions have been raised concerning the cost-effectiveness of the new techniques and their potential to decrease the rate of cervical cancer among Minnesota women.

This report examines computer-assisted screening and new slide preparation methods for cervical cancer screening and summarizes findings from the medical literature regarding their potential to affect net health outcome and the cost of Pap testing. This report has two purposes. First, it is intended to provide information to policymakers, health plans, and the public to support health policy development, coverage and purchasing decisions. Second, the report is intended to provide information to health care practitioners to assist in responding to patient questions regarding the accuracy of Pap testing and the value of the new technologies. Diagnosis and treatment of premalignant and malignant lesions of the cervix are not addressed in this report.

Findings

Computer-assisted screening and rescreening technologies

Automated screening and rescreening technologies utilize computerized systems to select or classify slides with possible abnormalities. Three technologies are currently approved by the Food and Drug Administration (FDA).

The PAPNET Testing System® is FDA-approved for supplemental quality control screening of Pap smears evaluated as negative during manual screening.

The NeoPath AutoPap 300 QC System® NeoPath's AutoPap Primary Screening System is FDA approved as a primary screening tool for Pap smears.

Studies of these methods demonstrate that computer-assisted cervical cancer screening and rescreening modestly improves detection of false-negative smears as compared with conventional manual screening. The majority of false-negative smears detected are low-grade squamous intraepithelial lesions (LGSIL), reactive or reparative changes, or atypical squamous cells of undetermined significance (ASCUS) rather than the more serious premalignant or malignant lesions. Some studies have shown that computer-assisted Pap smear screening may marginally improve health outcome for some patients. The net health benefits of computer assisted screening have not been proven. Studies examining the cost-effectiveness of the new technologies indicate that the cost-benefit of computer-assisted rescreening technologies is less favorable than any manual rescreening alternatives.

Slide Preparation Method

One semi-automated, liquid-based slide preparation method, ThinPrep® Pap Test (Cytyc Corporation), is FDA approved. The ThinPrep method produces a slide with a thin layer of cells which can also be used to test for human papilloma virus (HPV) as well as for cytology review. Data from studies reviewed showed that ThinPrep is capable of reducing the false-negative rates of Pap smear screening. The FDA has stated that study data submitted by the manufacturer produced slides that are “significantly more effective (than traditional methods) in a variety of patient populations for detecting low-grade squamous intraepithelial lesions (SIL's) and some of the more severe lesions.” 6 Several studies concluded that ThinPrep processing saves time in slide review. Costs saved with reduced slide review time for cytotechnologists may be offset by the cost of equipment and slide preparation time.

Conclusions

Each of the new technologies improves the accuracy of Pap test screening at a higher cost than manual Pap screening. Improvements in the effectiveness of a screening test do not necessarily translate into prevention of cancer, and the degree to which this reduction in false-negatives reduces cervical cancer rates has yet to be determined. Further, if costs of Pap test screening rise, fewer women may receive periodic Pap tests. This would, in turn, predictably increase cervical cancer rates.

Recommendations

The added value of the new technologies PAPNET, AutoPap, and ThinPrep in improving the net health outcome of women and preventing cervical cancer has not been determined, therefore:

  • Women should obtain periodic Pap tests, which have proven to be the best way for women to protect themselves against cervical cancer and false-negative results.
  • Clinicians should educate all women in their care about the importance of periodic Pap testing. For women concerned about Pap test accuracy, clinicians are encouraged to discuss the new technologies for cervical cancer screening and provide information about the individual benefits of the new Pap testing technologies.
  • Health plans should utilize available information systems to increase the rate of women who have periodic Pap tests. In addition, health plans should incorporate Pap testing of women into quality improvement goals for their clinics. Given the limitations on health care dollars, health plans should concentrate their resources toward encouraging women to receive periodic Pap tests.
  • The State of Minnesota should continue to implement strategies to assure that women facing financial and other barriers are able to obtain periodic Pap tests and should closely monitor changes in patterns of cervical cancer screening, morbidity and mortality.
  • Stakeholders should promote additional studies on the safety, clinical effectiveness, and cost effectiveness of the new technologies for slide preparation and computer-assisted cervical cancer screening and quality control slide review.

Introduction

Invasive cervical cancer was once the leading cause of cancer death in women in the United States. Today it accounts for less than 3 percent of all cancers in women. A 70 percent decrease in cervical cancer deaths over the past 50 years has been credited in large part to regular Papanicolaou (Pap) testing. 1 Available data indicates that in the late 1930s and early 1940's cervical cancer accounted for 25-30 deaths per 100,000 women. 2 In 1992, the death rate for cervical cancer was 3.0 per 100,000 women. 3 Organizations including the National Cancer Institute 4 and the Centers for Disease Control and Prevention 5 state that the death rate for cervical cancer would be even lower if all women received periodic Pap tests.

Direct consumer marketing by manufacturers of new Pap test screening technologies has raised questions about the comparable value of new and existing technologies. Each of the new technologies marginally improves the accuracy of Pap test screening at a higher cost than manual Pap screening. Fewer women may receive periodicaless frequently in a low-risk woman at the discretion of her physician. 7 Pap tests if costs of screening rise. This, will, in turn, predictably increase cervical cancer rates.

This report examines computer-assisted screening and new slide preparation methods for cervical cancer screening and summarizes findings in the medical literature regarding their potential to affect net health outcome and the cost of Pap testing. This report has two purposes. First, it is intended to provide information to policymakers, health plans, and the public to support health policy development, coverage and purchasing decisions. Second, the report is intended to provide information to health care practitioners to assist in responding to patient questions regarding the accuracy of Pap testing and the value of the new technologies.

Pap testing technologies approved by the FDA include computer-assisted Pap smear screening and a slide preparation method. This report discusses the following questions relating to these technologies and Pap testing:

  • What are the strategies that could lower the rate of cervical cancer?
  • Do these technologies improve detection of false-negative smears as compared with conventional manual screening?
  • Can these technologies improve net health outcome?
  • Is Pap test screening with these technologies cost-effective?
  • Are there other alternatives that could lower the rate of cervical cancer?

Background

Cervical cancer originates on the surface of the uterine cervix or in the endocervical canal. Precursor lesions are epithelial dysplasia and/or carcinoma in situ (CIS). Some of these early lesions develop into invasive cervical cancer. Generally, the progression to invasive cancer is a slow, predictable pattern. Longitudinal studies indicate that 30 percent to 70 percent of untreated patients with cervical intraepithelial neoplasia (CIN) will develop invasive carcinoma in 10-12 years. In about 10 percent of patients with CIN, lesions progress to invasive carcinoma in under one year. Prognosis for the disease is influenced by the stage, size and grade of tumor at the time of detection, the tumor's histologic type, and whether or not it has spread via the blood or lymphatic systems.8,9

The risk for cervical cancer is increased for women who have had their first sexual intercourse at a young age, multiple sex partners, a high number of pregnancies,recurrent vaginal infections, or sexually transmitted diseases including genital herpes, genital warts, and HIV. Infection with human papilloma virus (HPV) is a major risk factor for development of cervical cancer. Various types of HPV infection are associated with different levels of risk for development of malignant and premalignant lesions. 10 Although studies are currently underway to determine how HPV typing can be used to help determine therapy patterns for women infected with the virus, no therapy or follow-up protocols have been established. The risk of cervical cancer is also increased in women who smoke.10,11

Cervical cancer is not age limited, and women should continue to receive periodic Pap tests throughout their lives. Invasive cervical cancer is most common in women between the ages of 30 and 50. 12 Nationally, approximately 41 percent of deaths due to cervical cancer occur in women aged 65 and older. 1

According to national estimates, approximately 4,900 women will die of cervical cancer in 1998. The five-year survival rate for locally invasive cervical cancer is over 91 percent. The five-year survival rate for all invasive cervical cancer is 69 percent. 3 In its early stages, cervical cancer often shows no symptoms. In its later stages, cervical cancer produces unexplained vaginal bleeding, pain and bleeding after intercourse, or persistent vaginal discharge. 12-15

Occurrence of preneoplastic and premalignant cervical cellular changes are several times higher than the rate of invasive cervical cancer. Nationally, an estimated 13,700 women will be diagnosed with invasive cervical cancer in 1998. Another 65,000 will be found to have CIS. 2 The National Cancer Institute estimates that each year approximately 300,000 women are found to have premalignant cervical lesions. These premalignant lesions and CIS are most often discovered through a routine pelvic examination and Pap test. 2 The Pap is a screening test, not a diagnostic test. Abnormal Pap test results are frequently followed by cervical colposcopy with or without biopsy or larger cervical tissue sampling to determine a diagnosis.13-16 Prompt follow-up by clinicians is essential when Pap test results show malignant or premalignant cells. The American Cancer Society, 13 American College of Obstetricians and Gynecologists, 14 and the National Cancer Institute/National Institutes of Health,1,12 have developed guidelines and recommendations for appropriate treatment of women whose Pap test results indicate cervical abnormalities.

Occurrence of cervical cancer in Minnesota

Between 1988 and 1994, 1,371 Minnesota women age 15 years and older were diagnosed with invasive cervical cancer (rate of 7.5 per 100,000 women). One-hundred-seven women in Minnesota died of invasive cervical cancer during that same period (rate of 1.6 per 100,000 women). In 1994, the most recent year for which data is available,1,776 Minnesota women age 15 and older (rate of 97.6 per 100,000 women) were diagnosed with preinvasive cancer of the cervix. 17

Data from a 1996 telephone survey conducted annually by the Centers for Disease Control and Prevention (CDC) found that 68 percent of Minnesota women surveyed had received a Pap test in the past year. This rate is the same as the national average. Approximately ten percent had not had a Pap test for five or more years. 18 National statistics indicate that Pap testing rates tend to be lower among some groups than others. The elderly, uninsured, some ethnic minorities, and poor women are generally less likely to receive periodic Pap tests than other groups. 1 Reliable data from this survey were not available regarding Pap test rates for all population groups in Minnesota. Statistics from the 1996 Minnesota HMO Profile show that cervical cancer screening rates vary from approximately 46 percent to approximately 88 percent among health plans.b General Assistance Medical Care (GAMC) members tended to receive Pap tests below the rate of enrollees in other plans. 19

Cytology classification

Cells found on Pap smears were initially classified according to the Papanicolaou system, based on numerical cell class designations. In 1988, recognizing the need for standardization of cervical and vaginal cytopathology reporting and revision of the Papanicolaou system, the National Cancer Institute introduced The Bethesda System (TBS). TBS defines criteria for both specific cell classification and specimen adequacy. 16 A table describing the classification systems is attached in Appendix I.

Pap Smear Methods

Manual Screening

When properly obtained and prepared, Pap smears alert practitioners to possible abnormalities of the cervix. Manual microscopic screening of Pap smears by cytotechnologistscof Clinical Pathologists. continues to be the most widely used and accepted method of identifying irregular cells on cellular abnormalities of the cervix. During manual screening of Pap smear samples, cytotechnologists search for what may be a few pathologic cells by inspecting many fields on a slide under the microscope. Cytotechnologists must manipulate the microscope stage through visual inspection of approximately 50,000 to 300,000 epithelial and inflammatory cells per smear. Manual screening can be tedious and time-consuming since the majority of cells on a smear are normal. Even smears with serious abnormalities may contain only a few dozen malignant or premalignant cells among hundreds of thousands of normal cells.20,21

In 1998, Medicare began reimbursing Pap tests every three years for women at low risk. Annual screening is reimbursed only for those women identified at high risk for cervical cancer. 22 Medicaid covers periodic Pap testing. Most private health care plans cover manual Pap testing.

False-negative Pap smears

A false-negative Pap smear is defined in this report as a normal cytologic report for a woman with existing dysplasia, premalignant or malignant lesions of the cervix. 1 When these cells are missed, resulting delays in diagnosis can lead to progression of the disease and the need for more aggressive treatment. However, if abnormal cells are missed in one test, there is a high likelihood they will be found on the next test. False negatives can occur under the following conditions:

Cell sampling and preparation. Accuracy of Pap smear results depends in part upon how well the cervical tissue is sampled and slides are prepared. At the time of collection, samples of cells collected from the surface of the cervix must contain both ectocervical and endocervical cells. If abnormal cells are not included in the sample, abnormalities cannot be detected. False-negatives can also occur if blood, vaginal secretions, or inflammatory cells in the sample conceal the abnormal cells making them undetectable by the human eye.

Screening and interpretation. False-negative results due to screening are caused by failure to identify a relatively small number of abnormal cells in the screening process. A trained cytotechnologist typically searches hundreds of thousands of cells per smear on up to 100 slides per day. 50

Some in the medical community and consumers are concerned about the number of false-negative results generated by the manual technique. 23 The combined false-negative rate for a single test is estimated to be 20 percent. 1 The proportional contribution of sampling, screening, and interpretation on the accuracy of test results has not been fully determined. Some sources attribute the majority of false-negative Pap smear results to problems in sampling and preparation. 24 Others state that approximately half are due to inadequate specimen sampling and slide preparation. The other half are due to inaccurate interpretation or failure of screeners to identify abnormal cells on a slide. 1

There is consensus among cytology professionals that there is an irreducible false-negative rate of five to ten percent.25,26 All false-negative results do not carry the same clinical significance. Rather, false-negative results most often mean that cells of undetermined significance have been missed. In a small percentage of cases, malignant or premalignant conditions are missed. No data were found regarding the mortality rate associated with false-negative Pap smear results. False-negative results due to improper interpretation (cells interpreted as benign or normal when they are, in fact, SIL, premalignant. 5

Strategies To Lower The Rate Of Cervical Cancer

The common goal of all the strategies discussed in this section is to lower the rate of cervical cancer. One strategy addresses increasing the rate of cervical cancer screening. The remaining three strategies deal with technologies or programs designed to improve the accuracy of Pap smear.

Increasing the rate of cervical cancer screening

Public and private health organizations recognize that improved recruitment of women who do not receive periodic Pap tests is an effective means of decreasing the rate of cervical cancer. The National Institutes of Health (NIH) estimate that 50 percent of women with newly diagnosed invasive cancer of the cervix never had a Pap test done. Another 10 percent had not been screened in the past five years. 1 In its Healthy People 2000 program, the U.S. Department of Health and Human Services has included a goal to increase the proportion of all women aged 18 and older who will have received a Pap test within the preceding 3 years to at least 85 percent. 27

Reducing the rate of cervical cancer by increasing the rate of women who receive periodic Pap tests is a goal of the State of Minnesota as well. The Minnesota Breast and Cervical Cancer Control Program (MBCCCP) provides free cervical cancer screening programs to low income women. The MBCCCP also provides information on cervical cancer screening aimed at population groups in which Pap testing rates are low.

Pap test sampling and slide preparation

Several techniques have been developed since the introduction of the Pap test to improve screening and interpretation accuracy through improvements in sampling and slide preparation methods. These improvements include the development of the Ayre and extended tip spatulas and cervical brush to aid in collecting sufficient samples of both endocervical and ectocervical cells. 28 In addition, research continues on the most effective means of obtaining optimum samples using combinations of spatulas, brushes, and slide staining techniques.24,29

Some sampling and preparation problems, such as improper sample thickness and air drying artifacts on the slide, are controllable and may be avoided. Educational programs directed at reducing problems associated with Pap smear sampling may be effective in improving the quality of specimens. These improvements, in turn, will aid cytotechnologists in slide review and interpretation.

Cytology laboratory oversignt

The Clinical Laboratory Improvement Amendments (CLIA) of 1988 were enacted by the United States Congress to reduce the incidence of false-negative Pap smear results occurring in cytology laboratories. Under CLIA, 30 the daily workload of a cytotechnologist is limited to a maximum rate of 100 slides per day. CLIA also requires cytotechnologists and cytopathologists to undergo regular proficiency testing. No testing program had been put in place as of December 1998. In addition, laboratories must rescreen 10 percent of slides previously designated as normal for quality control. Rescreened slides must be randomly selected from normal cases as well as from patients classified as high-risk by clinical history. 31 Requirements of CLIA have increased the cost of Pap testing for patients and the public. A summary of the CLIA regulations is attached in Appendix II.

New technologies in slide preparation and screening

This section describes only those technologies approved by the FDA prior to June 1998. These technologies include an alternative slide preparation system and two computer-assisted screening systems. Several other liquid-based cell preparation systems and computer-assisted screening technologies are under review by the FDA.

Alternative slide preparation systems

Concern about false-negative Pap smear results due to unsatisfactory slide preparation has led to the development of a method of preparation designed to improve slide quality. ThinPrep, was approved by the FDA in May 1996. Pap smear cell samples are collected with a special cervical brush developed by the manufacturer. Cells are rinsed into a vial of alcohol-based fixative (PreservCyt®) solution rather than transferring the sample directly to a slide. The vial is sent to a cytology laboratory equipped with a Thin Prep 2000 Processor. The processor mixes the solution in the vial breaking up non-diagnostic materials including blood, mucus, and inflammatory cells. The cells are then collected on a special filter and a thin layer of cells is transferred to a glass slide which is deposited in a fixative solution. ThinPrep prepared slides cannot be read by computer-assisted screening or quality control devices. They must be manually evaluated by a cytotechnologist or pathologist. Specific staff training is needed for the preparation and assessment of ThinPrep slides. Even experienced cytotechnologists require additional training before becoming proficient in assessing single layer preparations. 32

The FDA has stated that the study data, submitted by the manufacturer, produced slides that are “significantly more effective (than traditional methods) in a variety of patient populations for detecting low-grade squamous intraepithelial lesions (SIL's) and some of the more severe lesions.” 6 Improvements in effectiveness of a particular test do not necessarily translate into prevention of cancer, and the degree to which this reduction in false-negatives reduces cervical cancer rates has yet to be determined.

Sampling and preparation supplies are provided to clinics by laboratories performing the slide analysis. Samples can be sent only to cytology laboratories equipped with the ThinPrep 2000 processor, which carries a purchase price of approximately $39,000. 33 A number of managed care plans throughout the country have begun to reimburse for ThinPrep processing. Reimbursement rates for manual Pap tests by private health plans average approximately $15. In Minnesota, Medicare and Medicaid reimbursement for manual Pap tests is $7.15, and $11.00 for ThinPrep processing. 34 ThinPrep processing adds $15-20 to the laboratory charges for a Pap test. 35

Computer-assisted rescreening

The table below summarizes the equipment requirements and processes of the two computer-assisted systems that have received FDA approval for quality control rescreening of Pap smears manually evaluated as normal, the PAPNET system and the AutoPap system.

Table 1. Comparison of Computer-Assisted Quality Control Rescreening Technologies.

Table 1

Comparison of Computer-Assisted Quality Control Rescreening Technologies.

Computer-assisted devices are designed to reduce false-negative Pap smear interpretations using computerized systems that assist the cytotechnologist in identifying Pap smear abnormalities. Computerized rescreening technologies provide added value in their ability to analyze all cells on slides without fatigue becoming a factor and the systems' ability to consistently and objectively analyze slides. 36 Automated systems have also been shown to reduce the number of false-negative Pap smears compared to manual rescreening. Automated systems also allow for targeted selection of slides more likely to be abnormal for the mandated CLIA rescreen. This may be more effective than the current random selection method.

Drawbacks of computerized systems include their cost and an increase in the number of false-positive Pap smears. Slides identified by computerized systems, including false-positives, must be reviewed by a cytotechnologist or cytopathologist for classification. False-positive determinations will likely result in a cascade of events including slide review and retesting that may create unnecessary patient anxiety and cost to the health care system.

It is not known to what extent added costs associated with computerized screening may reduce costs over time if the technology improves early detection of cervical cancer. No studies were found for this report which examined the potential effects of false-positive Pap test results on patient outcomes or health care costs. Follow-up of false-positive Pap results will increase costs as retesting, colposcopy, biopsy, or other procedures may be done, depending upon the types of cellular changes indicated by the results. There is concern among some medical and consumer groups that added costs and perceptions of added benefit of computer-assisted screening technologies will result in decreased Pap testing 35,37

Computer-assisted primary screening

The AutoPap Primary Screening system was approved by the FDA for primary or initial cervical cytology screening in May 1998. The system is designed to assist cytotechnologists in reviewing conventionally prepared slides by scoring and ranking each slide according to the likelihood that a slide contains abnormalities. All successfully processed slides are categorized as either “no further review,” “review,” or “quality control review.” The system identifies up to 25 percent of the slides at lowest risk. These slides require no human review, and can be immediately archived. The remaining are more likely to contain abnormalities and require manual review by the cytotechnologist. Only those slides on which no abnormalities are detected and which are determined to be at lowest risk for an abnormality are immediately archived. If abnormalities are detected on all slides in a group, the system will return all with the “review” designation. AutoPap also selects 15 percent “most abnormal” slides from the remaining slides and designates them as quality control review. This “enriched” sample of slides can replace the 10 percent random selection of slides for the CLIA mandated rescreen. The AutoPap Primary Screening System is not intended for use with high risk patients, and is not compatible with liquid-based slide preparation and all staining methods. 41

Reimbursement of computer-assisted Pap test technologies

Nationally, private insurers have begun to cover these technologies, although coverage varies. Patients considering whether to have a Pap test screened or rescreened using a computer-assisted method should check with their health plan and their clinician. If a health plan or insurer does not cover the full charges of PAPNET or Auto Pap, the added charges are passed on to patients. In Minnesota, Medicare and Medicaid reimburse automated rescreening of Pap smears at $7.15, the same rate at which manually-screened Pap smears are reimbursed. 42

Review of Peer-Reviewed, Published Literature

Methodology

Searches were conducted on peer-reviewed articles published between 1993 and March 20, 1998 on Medline, Current Contents, FDCR, and the National Cancer Institute PDQ on the key words Pap, AutoPap, PapNet, ThinPrep, vaginal smears and diagnosis, mass screening, vaginal smears, cytodiagnosis, and cytology.

Data were evaluated from six PapNet clinical trials, four AutoPap clinical trials, and seven ThinPrep clinical trials. Data from six PapNet clinical trials and two ThinPrep clinical trials are not discussed due to poor study methodology or outdated research.

Literature summary

Clinical trial results indicate that computer-assisted Pap test screening is effective in reducing the rate of false-negative results typically associated with Pap testing. AutoPap studies are more often designed to demonstrate the improvement in selecting a sample of slides for quality control review, whereas the PapNet studies focus on the change in detection rates. One study examined AutoPap as a primary screening device demonstrating improved detection rates without loss of specificity. Sensitivities and specificities reported in the literature are calculated using various methods making it difficult to compare results across studies. AutoPap literature reports sensitivities ranging from 35 to 77, and increased detection rates by thresholds ranging from three to eight. This variation reflects, in part, the differing selected review rates tested in the AutoPap studies. Additionally, improved detection rates of these technologies varied depending on the cell classifications studied. In general, the studies demonstrate greater detection rates for abnormal cell types, with lesser accuracy in detecting atypical glandular cells of unknown significance (AGUS).

Study results should be interpreted carefully due to limitations and biases in the study designs. Several of the studies used “enriched” study samples containing a high number of positive smears, and thus did not represent the realities of daily practice. Also, heightened vigilance of the cytotechnologists reviewing slides in a research environment may have impacted study results.

Four studies examined the potential cost-effectiveness of PAPNET and AutoPap 300 QC System.43-46 Two studies43,44 conclude that increased costs are not justified given the marginal rates of improvement in detection of false-negative slides. Two studies45,46 found computerized rescreening of Pap smears to be cost-effective if performed on a triennial basis. No cost studies were available at the time of this report on the AutoPap 300 Primary Screening device which may reduce cytologists' work load. Screening women who have not previously had a Pap test may prove to be more cost-effective than computer-assisted screening in reducing the incidence of cervical cancer. However, cost-effectiveness of many of these programs is unknown.

The cost-effectiveness of automated rescreening is dependent upon the prevalence of cervical cancer in the screened population, the sensitivity of initial screening, and by the cost of rescreening. Laboratories screening population at high risk for cervical cancer or those which have a higher false-negative rate on initial screening, will find computer-assisted Pap screening more cost-effective than laboratories which screen patients at low or average risk or have a low false-negative rate. As the computer-assisted Pap screening technology becomes integrated into clinical practice, reductions in the cost of the systems will also add to the cost-effectiveness of the devices. Additional well designed studies are needed to assess the value of these new technologies as they operate as quality control devices and as primary screening systems.

For this report, two studies comparing ThinPrep with biopsy confirmed diagnosis and six studies comparing ThinPrep to conventional preparation were reviewed. The data suggest that ThinPrep offers a marginal improvement in sensitivity, and similar specificity as compared with conventional Pap test slide preparation. Generally, higher number of low grade and more severe lesions were detected by ThinPrep in the studies. Another proposed advantage is a decrease in the number of slides designated as atypical squamous cells of unknown significance ASCUS). In addition, improved specimen adequacy reduces the number of cases classified as satisfactory but limited by (SBLB).32,47 Human papilloma virus testing can also be done with the ThinPrep system. 47 Thus, ThinPrep is capable of reducing the false-negatives of Pap screening; however, the benefit to the population in reducing cervical cancer rates has yet to be determined. Several studiesconcluded that ThinPrep processing saves time in slide review. 48 Costs saved with reduced slide review time may be offset by the cost of equipment and slide preparation time. Summaries of the medical literature reviewed for this report are attached in Appendix III.

Other Technology Reports

American College of Obstetricians and Gynecologists (ACOG)

In August 1998, the American College of Obstetricians and Gynecologists (ACOG) Committee on Gynecological Practice published a Committee Opinion on new Pap test screening technologies. ThinPrep slide preparation and rescreening by AutoPap and PapNet were assessed. The procedure for primary screening of slides using the AutoPap System was reviewed, however, no data were available at the time of publication regarding its efficacy or cost-effectiveness as a primary screening tool.

The ACOG committee acknowledged that these new technologies improve sensitivity of cervical cytology review and reduce false-negative results. However, the committee cited two disadvantages of the new technologies. First, the technologies raise the cost of Pap testing and second, the improvements in accuracy involved identification of an increased number of patients with low grade rather than premalignant or malignant lesions. The ACOG committee concluded that new Pap test screening technologies are not a standard of care and that appropriate use of these technologies require further investigation.

The ACOG committee stated that efforts to reduce the rate of false-negative Pap tests “should not detract attention from the most important way to prevent cervical cancer - participation in a screening program.” The committee also noted that manual Pap test screening for all women should be the goal of Pap testing efforts.

The Canadian Coordinating Office for Health Technology Assessment (CCOHTA)

The Canadian Coordinating Office for Health Technology Assessment (CCOHTA) conducted a cost-effectiveness analysis comparing PapNet and AutoPap to a strategy of 10 percent manual rescreening. The model assumed a 25 percent false-negative rate for conventional manual Pap screening, a rescreening sensitivity of 83 percent for PapNet and 80 percent for AutoPap. Based on the Canadian dollar, the direct cost per slide for rescreening with the AutoPap system was $7; the direct cost per slide of rescreening with the PapNet system was estimated at $14. The resultant cost for each additional abnormal case found by 10 percent random rescreening was estimated at $250. The cost rises to over $400 per additional abnormal smear detected with the AutoPap system and to over $10,000 utilizing the PapNet device. The findings were highly sensitive to the cost of the computerized rescreening devices. The report states that the cost for AutoPap may be reasonable, but for PapNet, the cost does not appear warranted. The CCOHTA concludes that computer-assisted cervical screening is more effective, and more expensive than manual rescreening. The report also acknowledges that such technologies will have no impact on improving Pap test accuracy in women who are rarely or never screened.

Blue Cross/Blue Shield Association (BCBSA)

The Blue Cross/Blue Shield Association Technology Evaluation Center (TEC) completed a cost-effectiveness analysis of AutoPap, PapNet, and the ThinPrep system in February 1998. The study determined that, although he automated devices are more accurate than conventional Pap testing, the improvements in diagnostic accuracy are modest considering the natural history of the disease. Since cervical cancer, in most cases, is a slow-growing cancer that progresses over a period of years before ultimately becoming a malignant invasive neoplasm, the cost analysis concludes that compared to standard Pap testing, AutoPap, PapNet, and ThinPrep “are likely to increase life expectancy by, at most, one or two days; and, for women who are screened frequently, only by hours.” 49 The authors suggested that a much larger increase in life expectancy may result from screening women who have not previously had a Pap test. The higher cost of automated cervical slide preparation and computer-assisted Pap screening may render current Pap testing methods cost prohibitive, thus causing a decrease in the number of women screened.49,50

Conclusions

The Pap test, one of the most successful cancer screening tools, has reduced the mortality rate for cervical cancer by 70 percent since its introduction in 1941. Yet, even today, not all women who should be screened receive Pap tests. For example, nearly one-third of women in Minnesota do not receive annual Pap tests. In addition, Pap testing is lower in some population groups than the general population. These groups include the elderly, some ethnic minorities, the uninsured and underinsured. Strategies to improve the effectiveness of the Pap test and thereby further reduce the rate of cervical cancer include reducing the rate of false-negative Pap smears and increasing the number of women who receive periodic Pap tests.

New technologies designed to reduce the rate of false negative Pap smears include computer-assisted Pap smear screening and a method for monolayer slide preparation that improves sample quality. While computer-assisted Pap smear screening reduces false-negative results as compared with random manual screening, the improvement in accuracy is modest in that the majority of false negatives found are low grade rather than premalignant or malignant lesions. In addition, computer-assisted screening is expected to only minimally improve net health outcome in most women who undergo periodic Pap tests. This is because treatment is effective in controlling cervical cancer, generally a slowly-progressing disease, in nearly 100 percent of cases where it is detected early. The monolayer preparation method is also capable of reducing the false-negatives of Pap screening; similarly, the improvement in net health outcome of this technology to the population has yet to be determined.

In contrast, recruitment of women who have not previously had a Pap test, or are rarely screened, and ensuring that they receive periodic Pap tests could significantly reduce the number of cervical cancer cases. None of the new technologies for cervical cancer screening will affect detection of cervical cancer in women who are not periodically screened.

A few studies examining the cost-effectiveness and cost-benefit of the new technologies for cervical cancer screening have been completed. They indicate that the cost-benefit of computer-assisted technologies is less favorable than any manual rescreening alternatives and that screening of women who have not previously had a Pap test may prove to be more cost-effective than computer-assisted screening in reducing the incidence of invasive cervical cancer.

Recommendations

The added value of the new technologies, including PapNet, AutoPap and ThinPrep, in improving the net health outcome of women and preventing cervical cancer has not been determined, therefore:

  • Women should obtain periodic Pap tests, which have proven to be the best way for women to protect themselves against cervical cancer and false-negative results.
  • Clinicians should educate all women in their care about the importance of periodic Pap testing. For women concerned about Pap test accuracy, clinicians should discuss the new technologies for cervical cancer screening and ensure that they receive information about the individual benefits of the new Pap testing technologies.
  • Health plans should utilize available information systems to increase the rate of women who have periodic Pap tests. In addition, health plans should incorporate Pap testing of women into quality improvement goals for their clinics. Given the limitations on health care dollars, health plans should concentrate their resources toward encouraging women to receive periodic Pap tests.
  • The State of Minnesota should continue to implement strategies to assure that women facing financial and other barriers are able to obtain periodic Pap tests and should closely monitor changes in patterns of cervical cancer screening, occurrence, and mortality as new slide preparation and computer assisted screening technologies are more widely used. This data will be beneficial as policy makers evaluate the most effective use of health care resources.
  • Stakeholders should promote additional studies on the safety, clinical effectiveness, and cost effectiveness of the new technologies for slide preparation and computer-assisted cervical cancer screening and quality control slide review.

Disclaimer of Endorsement

Reference to any product, process, or service by trade name, trademark, manufacturer or otherwise does not constitute or imply any endorsement or recommendation by the State of Minnesota, the Health Technology Advisory Committee or any of their employees or contractors.

Disclaimer of Liability

With respect to this document and information, neither the State of Minnesota, the Health Technology Advisory Committee, nor their employees or contractor make any warranty, express or implied, including warranties of merchantability and fitness for a particular purpose with respect to the data available. Additionally, the State of Minnesota, the Health Technology Advisory Committee, nor their employees or contractors assume no legal liability for the accuracy, completeness, or usefulness of any information, apparatus, product, data, or process disclosed herein and do not represent that use of such information, apparatus, product, data, or process would not infringe on privately owned rights.

Appendix I: Classification Systems for Cervical Cytology

Pap smears are most often classified according to The Bethesda System (TBS). It was developed in 1988 to replace numerical designations in the Papanicolaou System developed in the 1930s. Two dysplasia classification systems were developed by Richart and Reagan in 1953 and 1973 respectively. The National Cancer Institute (NCI) introduced TBS to standardize and expand designations and facilitate management decisions for further testing and treatment of cervical abnormalities.

Appendix II: Summary of Clinical Laboratory Regulations of the Clinical Laboratory Improvement Amendments of 1988 (CLIA)

The Clinical Laboratory Improvement Amendments of 1988 (CLIA) specifies rules in several categories including personnel standards, laboratory inspections, proficiency testing, and quality control. CLIA applies to all laboratories that perform tests on human tissue if results are used for the diagnosis, prevention, or treatment of disease or impairment. These include hospital and clinic based laboratories as well as reference or specialty laboratories. CLIA established specific regulations for cytology laboratories. A summary of the regulations follows.

CLIA is administered by the Health Care Financing Administration (HCFA), and the Centers for Disease Control and Prevention (CDC), which is largely responsible for proficiency testing guidelines. The HCFA website (http://www.hcfa.gov/clia/clia1.htm) provides updates on CLIA regulations and laboratory compliance.

The Commission of Office Laboratory Accreditation is the accrediting organization under CLIA. U.S. Department of Health and Human Services has granted deeming authority to additional accrediting organizations. These organizations are the American Society for Histocompatability and Immunogenetics, Joint Commission on Accreditation of Healthcare Organizations (JCAHO), College of American Pathologists (CAP), American Osteopathic Society (AOA), and the American Association of Blood Banks (AABB).

Cytology Quality Control

CLIA establishes a workload limitation of no more than 100 slides (gynecologic and nongynecologic or both) in a 24-hour period for individuals reviewing cytologic preparations by nonautomated technologies. This does not include previously reviewed slides.

Laboratories must manually review 10 percent of all gynecologic cases initially interpreted to be negative.

Cytology Proficiency testing

Annual monitored on-site proficiency testing (PT) is required of individuals engaged in the examination of cervicovaginal smears. The initial PT consists of a set of 10 slides. Individuals who fail the initial test are retested within 45 days with a different 10 slide set. Those who fail a second test must be provided with documented, remedial training and education. They are subsequently retested with a 20-slide set. All slides examined by those who require retesting must be reexamined.

Personnel Standards

A cytotechnologist must possess a current state license (if required by individual states) as a cytotechnologist and must be certified in cytotechnology by the American Society of Clinical Pathologists.

The technical supervisor of a cytology laboratory must be a MD or DO, board certified or board eligible in histopathology.

Appendix III: Findings from the Medical Literature

Six studies on the efficacy of the PapNet re-screening system are summarized below. Three studies of AutoPap as a rescreening device, and one study of the use of AutoPap in primary screening are summarized, along with eight studies of the ThinPrep Pap Test. Many of the studies reviewed below were either sponsored by the manufacturers of these systems or included researchers affiliated with the manufacturers.

PapNet re-screening

Rosenthal et al. University of California, Los Angeles and the University of Southern California, Los Angeles

Rosenthal et al. 21 investigated the PapNet system's ability to detect significant areas on clinically important false-negative smears. Pap smears obtained from a 1987 case-control study were rescreened by two different teams, at two different time periods, at separate institutions. One team of blinded researchers at the University of Southern California (USC, Los Angeles, CA) manually rescreened the smears as part of the original study. Slides that were considered adequate for grading were assigned a Pap classification of I through V. More recently, blinded researchers at the University of California, Los Angeles (UCLA) used the PapNet system to rescreen the same smears. The Cytotechnologist triaged each slide using the PapNet system as “review”, inadequate or negative”, then manually rescreened slides classified as “review” or “inadequate”. The cytotechnologist interpreted the slides and referred those with abnormality or atypia to a cytopathologist for further review.

Sixty two smears from the original study were evaluated. Twenty nine were classified as “negative” in the original study. However, the 19 women from whom these 29 smears were obtained were subsequently diagnosed with invasive squamous cell carcinoma. Another 33 smears taken from a previous study were obtained from women who did not develop cervical cancer.

The blinded, manual review of the 62 smears by the USC cytotechnologist resulted in the reclassification of 31 percent of the case smears taken from women with invasive squamous cell carcinoma (nine of the 29) and 6 percent of the control smears taken from women without cervical cancer (two of 33) as Class II to V.

When the UCLA cytotechnologist triaged these same smears using the PapNet system, 83 percent of the case smears taken from women with invasive squamous cell carcinoma (24 of 29) and 73 percent of the control smears taken from women without cervical cancer (24 of 33) were classified as review and referred for manual rescreening. Following manual rescreening 21 case smears from women with cervical cancer were referred to the cytopathologist who ultimately, reclassified 41 percent of these (12 of 29) as abnormal. Thirteen of the control smears were referred to the cytopathologist who reclassified 15 percent (5 of 33) as abnormal.

The study results showed that the PapNet system detected three additional case smears and three additional control smears, than did manual rescreening. Based on these data, the authors concluded that PapNet testing is an effective system for the routine re-screening of clinically significant false-negative PAP smears. However, study biases may have affected the results. The author commented that the reviewing PapNet cytotechnologist knew that some of the cases were false-negative and, therefore, was especially diligent in her search for abnormalities and requested a high proportion of slides be manually reviewed by the cytologist. Also, archival slides were rescreened in this study. A prospective review of slides obtained in clinical practice would have made the authors' results more applicable to general practice. A more valid comparison of the efficacy of the two technologies could have been made if the same personnel had conducted manual rescreening as well as PapNet rescreening. Additionally, the study sample was small and the number of abnormal smears that were reclassified as a result of PapNet testing was not statistically significant. Negative PapNet smears were not referred to a cytopathologist for validation in this study - only smears classified as “review” or “inadequate” were referred. No accurate conclusions can be made from this trial regarding the sensitivity, specificity, or predictive value of PapNet testing as compared with manual screening.

Farnsworth et al. Sydney, Australia

Cytopathologists at a large laboratory in Sydney, Australia, 51 compared the results of PapNet testing with manual testing in the screening of Pap smears. Initially, 60,317 smears were manually screened; 54,658 were classified as normal, and were then rescreened using PapNet. Additionally, of the 3840 smears classified as abnormal on manual screening, 1022 were randomly chosen to be rescreened using the PapNet system.

PapNet rescreening reclassified 266 of the 54,658 conventionally evaluated normal smears as abnormal and classified an additional 43 as technically unsatisfactory, representing a 7 percent increase in abnormal smear detection. PapNet identified all the high-grade intra epithelial lesions (HSIL) and squamous cell carcinomas. Additionally, PapNet reclassified 112 of the previously considered “abnormal” smears as normal. Histology follow-up verified that, of the 17 smears classified by PapNet as HSIL, 15 smears (88 percent) contained HSIL. In total, PapNet detected an15 histologically confirmed high-grade lesions not detected by manual screening.

Farnsworth and colleagues concluded that PapNet was capable of detecting more abnormal smears than manual screening, without a loss of specificity., and could feasibly be integrated into the routine practice of a laboratory. This two-armed, blinded trial provides some promising data regarding the efficacy of PapNet testing in the classification of conventionally evaluated Pap smears. However, no data was provided concerning the sensitivity, specificity, or predictive value of PapNet rescreening compared with manual rescreening.

Jenny et al.; Zurich, Switzerland

To determine the reproducibility and accuracy of the PapNet system in detecting and presenting abnormal and atypical cells, Jenny et al. 52 scanned 1200 Pap smears. Included in the 1200 smears were 516 smears taken from women with positive histologic diagnoses. Of the remaining 684 smears, 114 were suspicious smears and 570 smears were negative smears from women with negative follow-up over the previous 2 years.

PapNet rescreening was conducted at two separate laboratories by different personal who were blinded to the microscopic designations. At the first lab, the smears were screened with the PapNet device and reviewed by a cytotechnologist. The same smears were then reviewed in the same manner at the second laboratory by a cytopathologist. Smears were categorized by each laboratory as normal, suspicious or positive. To ascertain the reliability and consistency of PapNet screening, the concordance between the results of both PapNet series was calculated. To evaluate the efficacy of manual screening as compared with PapNet screening, the results of manual screening were compared to the two PapNet reviews.

Upon test completion, manual screening resulted in a false-negative rate of 5.6 percent; 29 of 516 smears were falsely classified as negative. The first PapNet screening resulted in a false-negative rate of 0.4 percent, and the second resulted in a false-negative rate of 0.8 percent. Statistical analysis determined that the differences between the results of manual screening and the two PapNet reviews were statistically insignificant when smears were classified as cervical intraepithelial neoplasia (CIN 1 or CIN 2). However, for invasive squamous carcinoma smears, the reduction in false-negatives by PapNet testing was deemed highly statistically significant. Upon completion of the trial, the sensitivity of manual screening was calculated to be 94.4 percent for malignant and premalignant lesions of the cervix; sensitivity was increased by the PapNet 1 and PapNet 2 testings to 99.6 percent and 99.2 percent, respectively. The concurrence of the first and second PapNet screening was 92.2 percent. The authors concluded that PapNet scanning of Pap smears is a reproducible and accurate method of finding and presenting abnormal and atypical cells.

This large, double-blinded trial provides some insight regarding the sensitivity and consistency of PapNet testing in the evaluation of Pap smears. Since slides designated by the PapNet system as normal were not histologically confirmed, no determination regarding the number of truly positive cases occurring in the study slides can be drawn. To obtain a more accurate comparison of PapNet to manual rescreening, smears should have been rescreened using both PapNet testing and manual methods by the same personnel.

O'Leary et al. Armed Forces Institute of Pathology; Washington, DC

O'Leary et al. 44 used PapNet to review 5,478 smears that were classified as “within normal limits” or “benign cellular changes” on both primary manual screening and 10 percent random rescreen. Of the 5,478 smears, 3,864 (71 percent) were classified normal upon PapNet screening and 1614 (29 percent) required further microscopic review. A consensus panel ultimately reviewed 11 cases for potentially atypical cells and reclassified 5 cases as ASCUS and 1 as AGUS. PapNet screening detected a small number of abnormal smears (0.11 percent) that were not identified on single manual screening. The majority of additional abnormalities detected were ASCUS or AGUS; both of these types of abnormalities have uncertain prognostic importance and uncertain management implications. Follow-up determined that the patient with the designation of AGUS on rescreening actually had LSIL.

O'Leary et al. acknowledges that this study detected fewer false-negatives than previous studies. The authors attribute the difference to their study design. Previous studies contained a relatively high percentage of abnormal smears, such as known false-negatives, smears originally classified as atypical squamous cells of undetermined significance, or cases that had not been previously screened. In comparison, the 5,478 smears studied by O'Leary et al. were expected to have a small percentage of false-negatives since they had previously undergone primary screening and a 10 percent random rescreen.

O'Leary et al. also performed a cost analysis estimating the costs and expected rate of increased detection of abnormal slides. Using $7.50 as the laboratory cost for performing PapNet rescreening, the analysis found the cost of PapNet screening to be $9.38 per slide, and the marginal cost of PapNet rescreening over 100 percent manual rescreening to be $6.38 per slide. When they used the PapNet quoted price of $40.00 per case, they estimated the marginal cost of PapNet over 100 percent rescreening to $33,781 per case of AGUS or ASCUS, and $101,343 per case of LSIL.

The authors acknowledged that the utility of PapNet rescreening will be sensitive to the cost of the PapNet test, and the prevalence of the disease in the screened population. The authors conclude that PapNet is capable of detecting a few additional cases of ASCUS as compared with manual rescreening, but at a high cost, and that other strategies such as population-based cervical screening programs to recruit rarely or never screened women may be more cost-effective in reducing cervical cancer morbidity and mortality.

Schechter; Mt. Sinai School of Medicine, New York, New York; and Neuromedical Systems

Using a model adapted from a government study of the cost-effectiveness of cervical cancer screening in elderly women, Schechter 45 evaluated the cost-effectiveness of supplemental PapNet testing in women 20 to 64 years of age. Based on current data for manually screened smears, Schechter assumed a false-negative rate of 25 percent for low-grade squamous intraepithelial lesions and 15 percent for high-grade squamous intraepithelial lesions. Using data submitted by the manufacturer of the PapNet system (Neuromedical Systems, Inc.) to the FDA, Schechter applied an increased sensitivity of 30 percent when using PapNet. Applying these data, Schechter estimates the cost per life-year saved among biennially PapNet-screened women to be $48,474. Similarly, the cost per life-year saved among triennially PapNet-screened woman is $25,185. Schechter calculated the cost per life-year saved among women biennially screened with conventionally evaluated Pap smears to be $9880, a figure substantially lower than biennial PapNet screening. The author concludes that PapNet rescreen is an economical approach to decreasing cervical cancer and that the cost is similar to other widely practice health interventions such as screening for prostate cancer.

The author of this study uses a sensitivity rate of 30 percent, a higher detection rate than reported elsewhere found in the literature. This assumption alone may have had significant impact on the favorable study results. In performing a sensitivity analysis, the author downgrades the PapNet performance to 24 percent, which increases the cost per life year $71,907. If the sensitivity analysis had included detection rate lower than reported in the literature, it is likely that the results would have been less favorable.

Kok and Boon; Leiden Cytology and Pathology Laboratory, The Netherlands

To determine the consistency of PapNet findings and the cytotechnologists using it, Kok and Boon 53 collected data from PapNet screening conducted from 1992 to 1994. The article reports on the daily screening performance of seven cytotechnologists, and does is not reflect a study situation according to the authors. The cytotechnologists prospectively evaluated 25,767 smears with manual screening and 65,527 smears with the aid of PapNet. The cytotechnologists did not review the same slides; incoming smears were randomly divided. Using TBS, the number of atypia of undetermined significance (Positive I), low-grade precursor lesions (Positive II), and high-grade lesions or invasive carcinomas (Positive III) were calculated for both screening methods. The mean positive individual scores and coefficients of variability per screening method and per positive group were calculated. To establish the consistency of scores for the seven cytotechnologists for each method, the differences between individual PapNet positive scores and individual conventional positive scores were calculated. Statistical analysis determined whether the mean individual positive scores were significantly different between the two screening methods. A statistical difference would indicate that one screening method was more accurate for identifying abnormal cells than the other.

The study found that statistically, the amount of inconsistency between the cytotechnologists was significantly lower for PapNet screening than for conventional screening. Analysis also determined that the statistical results of the screening process, particularly for high-grade Positive III lesions, showed significantly less dependence on the technologist performing the screening for PapNet than for conventional screening. Overall, the mean positive scores of the cytotechnologists were higher for PapNet than for conventional screening, indicating that cervical abnormalities or dysplasia were more readily identified with the PapNet system.

Despite the favorable findings in this “study”, several design weaknesses need to be considered. Only total PapNet positive scores for the groups of cytotechnologists were calculated. All the PapNet designations were not histologically validated; only the positive III smears were histologically confirmed. The true number of histologically proven positive smears was not determined. The actual number of histologically positive slides may have varied by cytotechnologist, allowing for a wide variation in positive smears. Since the same slides were not reviewed by all the cytotechnologists and histologically confirmed, no conclusions regarding the sensitivity, specificity, and predictive value of PapNet testing can be made.

AutoPap

Colgan et al.; Toronto Hospital and University of Toronto, Toronto, Ontario, Canada

Colgan et al. 54 compared the sensitivity of the AutoPap 300 QC System and random selection in determining false-negatives for quality control review. Under the trial protocol, 3487 slides previously classified to be within normal limits (WNL) were rescreened by the AutoPap QC 300 System and subsequently by a cytotechnologist blinded to the AutoPap results. Smears determined to be false-negative by the cytotechnologist who performed the first manual review, where then classified by one external and one internal review panel to determine final cytologic diagnosis.

Analysis was performed using multiple quality control thresholds for the AutoPap device that corresponded to review rates and 10, 15, and 20 percent. Extensive data results are reported in the article. The authors conclude that AutoPap detected five times as many false-negatives at the 10 percent review rate as compared to a 10 percent random quality control rescreen. Higher review rates showed less improvement. Although the data are promising, further trials are needed that reflect the use of AutoPap in a daily setting and document the sensitivity, specificity, and predictive value of AutoPap screening.

Wilbur et al.;NeoPath Inc., Redmond, Washington and University of Rochester edical Center, Rochester, New York; Kyto Diagnostics, New York, New York

To assess the sensitivity of the AutoPap 300 QC System to 86 cases of HSIL and invasive cervical cancer, Wilbur et al. 55 compared AutoPap findings with surgical biopsy results. Databases from previous AutoPap clinical trials were used to obtain biopsy results from patients in whom the cytologic designation was HSIL or greater. Wilbur et al. then compared the total number of cases deemed “review” by the AutoPap device with biopsy-confirmed cases. A sensitivity figure was calculated for the device's ability to detect cases at the level of HSIL or greater when the designated quality control rescreening level was established at 10 and 20 percent review thresholds.

The study found inclusion rates of 77 and 74 percent in the 10 and 20 percent review rates respectively. This represented an eight fold increase in the ability of PapNet to include these cases for quality control rescreening compared to random 10 percent rescreen. The highest rates of concordance were in the HSIL group (83 percent) at a 10 percent review threshold, and in any positive diagnosis match (87 percent) at the 20 percent review level. The calculated sensitivity demonstrated by the AutoPap QC 300 System ranged from 63 to 77 percent for 10 and 20 percent review levels respectively, for HSIL and invasive carcinomas. The overall sensitivity for detection of all false negative cases was 35.5.

Stevens et al.;NeoPath Inc., Redmond, Washington and Institute of Medical and Veterinary Science, Adelaide, South Australia

Stevens et al. 56 conducted a study to: 1) evaluate the performance of the AutoPap system and compare its detection rate to that of manual rescreening at various rescreening rates; 2) analyze the ability of AutoPap to detect slides previous classified as abnormal; and 3) determine the accuracy with which AutoPap can identify unsatisfactory or suboptimal smears.

In comparing AutoPap false-negative detection rates to manual rescreening, smears previously determined to be WNL were rescreened both manually and with the AutoPap 300 QC System. false-negative slides were defined as demonstrating minor non-specific changes in squamous cells or worse. Slides were evaluated by AutoPap at quality control selection rates of 10, 20, and 30 percent. Slides that were classified by the system as “quality control review”, or “review with potential abnormality”, were rescreened by a cytotechnologist. If the original manual screen designation and the AutoPap manual rescreen designation differed, the slides were examined by a review panel and a consensus diagnosis was determined.

The findings for the first goal of the study were: AutoPap identified 553 slides requiring quality control review. Overall, 26 smears were classified as false-negative-ASCUS, from which seven slides were determined to be false-negative according to the study definition. The 30 percent quality control threshold detected the highest number on false-negative slides (4 ASCUS) whereas the 10 percent threshold detected none. No smears were classified as LSIL or worse. At 30 percent and 20 percent quality control thresholds, the AutoPap device identified false-negative smears with sensitivity rates of 57.19 percent and 42.8 percent respectively. By comparison, 30 percent random manual rescreening identified false-negative smears at a sensitivity rate of 14.3 percent. No false-negative smears were identified by 20 percent random manual rescreening. Neither method identified false-negative smears at 10 percent quality control levels. The sensitivity of the AutoPap system at 30 percent and 20 percent quality control levels represented a three and fourfold improvement over random manual rescreening. The authors noted that although AutoPap was able to identify more false-negatives than random manual rescreening, the improvement was marginal due to the small study sample.

For part two of the study, 139 abnormal slides were analyzed to determine the ability of AutoPap system to detect slides previously classified as abnormal. Smear designations were confirmed either by biopsy or by unequivocal evidence of cytological abnormality. At quality control selection rates of 10 percent, 20 percent, and 50 percent, the AutoPap 300 QC system demonstrated sensitivity rates of 61.9 percent, 77 percent, and 94.2 percent respectively. No comparison was made to manual Rescreen without the assistance of AutoPap in this part of the study, so no conclusion can be drawn regarding comparable performance.

The third goal of the study was to determine the accuracy with which the AutoPap 300 QC device identifies unsatisfactory or sub-optimal smears, the sensitivity and specificity of the device to an inadequate squamous cell component (n=55) and endocervical cell component (n=1,587) was determined. At predetermined thresholds, the AutoPap's overall sensitivity to smears with an inadequate squamous cell component was 81.8 percent and to those slides with an inadequate cell component was 82.7 percent.

This study demonstrated that AutoPap detected a greater number of false-negatives than did random rescreening, but it did so only at 20 percent and 30 percent rescreening thresholds. Although the sample size was small, this study revealed no improvement in detection of additional false-negative slides over manual rescreen at the 10 percent level where both rescreening methods failed to detect the false-negative smears. Additionally, although the AutoPap system had a 94.2 sensitivity rate in the “enriched” abnormal slide sample, this occurred at a 50 percent review threshold, a much higher rescreen threshold mandated by CLIA.

AutoPap Primary Screening System

Wilbur et al.; NeoPath Inc., Redmond, Washington and University of Rochester Medical Center, Rochester, New York

A prospective, intended-use study 36 was conducted to determine: 1) whether use of AutoPap-assisted screening as both a primary and quality control device is superior to the current practice of manual screening and 10 percent mandated CLIA rescreening, and 2) how well the AutoPap System-assisted screening compares to current practice in identifying slides determined to be unsatisfactory or satisfactory but limited by (SBLB). This two-armed clinical study on 25,124 slides was conducted at five commercial laboratories.

All study slides were first manually screened by a cytotechnologist and rated as within normal limits (WNL), abnormal. or given an inadequacy rating (unsatisfactory or SBLB). A 10 percent random rescreen was then performed on those designated WNL. Thereafter, all successfully screened slides underwent AutoPap primary screening and received a designation of either no review (WNL, require no manual review, approximately 25 percent of slides), review (approximately 75 percent of slides, required manual review), or process review/rerun (slide failed processing). All slides were ranked according to the probability that they contained abnormal cells. Rankings were used by the cytotechnologist during the screening process. Additionally, approximately 15 percent of the slides designated as review, were identified for rescreening due to their high ranking. Reviewers were blinded to the results from all other reviews, and never reviewed the same slides.

The final diagnosis, or “truth determination” for each slide was determine in one of two ways. If the results of the cytotechnologist and AutoPap System review were in agreement, the result was considered the “true” cytology classification and slide adequacy. Independent external panels reviewed those slides on which the results did not agree in either slide adequacy or cytology classification. The external panel reviewers were blinded to the results of the previous reviews.

The study found that AutoPap identified more abnormal cases than did the current practice arm of the study in all disease categories (ASCUS, LSIL alone and LSIL+), and that the differences were statistically significant. From the final truth determination for each slide, the authors calculated an improvement of 13.4 percent in the positive predictive value, and reduced the false positive rates for normal slides by 15 percent. In the AutoPap arm of the study, fewer cases were referred to the pathologist than in the current practice arm. The positive and negative predictive values were not confirmed histologically. Lastly, the AutoPap System-assisted practice and current practice were similar in detecting unsatisfactory or SBLB slides.

The authors conclude that the AutoPap system as used in this study, reflecting intended use, is superior to the current practice of manual screening and 10 percent random rescreen. There was no discussion regarding the cost effectiveness of this strategy. Such studies are currently underway.

ThinPrep

Two studies compare ThinPrep with biopsy confirmed diagnosis47,48 and are reported in the table below. Data comparing ThinPrep to conventional preparation for 6 studies identified in the peer-reviewed literature is reported in a subsequent table. Sensitivities and specificities found in Tables A and B were calculated by HAYES, Inc. from data reported in the studies in order to make direct comparisons of values.

ThinPrep Results Compared With Biopsy Confirmed Diagnosis.

Table

ThinPrep Results Compared With Biopsy Confirmed Diagnosis.

Sensitivity is considered the most important parameter for a diagnostic screening test in the general population in that the maximum number of diseased patients are identified for more intensive testing. The Ferenczy, 1996 study reports an improvement in sensitivity for ThinPrep over conventional slide preparation (78 percent versus 70 percent). The sensitivity of ThinPrep is higher when ASCUS is considered a positive result (79.4 percent and 78 percent); however, specificity is compromised when ASCUS is considered a positive finding. If considering only negative results, the sensitivity for ThinPrep is higher than conventional Pap smears (79.4 percent and 78 percent versus 70 percent) while the specificity for one study is similar (73.6 percent versus 74.7 percent) and one is lower (52.4 percent versus 74.7 percent). Therefore, these data suggest that ThinPrep has fewer false-positive results than conventional slide preparation methods but has less optimal specificity when ASCUS is considered a positive result.

A number of studies evaluated the diagnostic accuracy of the ThinPrep method. Five studies used split-sample designs where both conventional and ThinPrep preparations were made from a single sample. Five studies used different study populations tested with either ThinPrep or conventional Pap smear. Some studies reported detection rates for both methods, under the assumption that a positive result by either method was a true positive. Other studies used either independent review and/or biopsy to confirm abnormal findings; in these studies the independent review or biopsy results provided the definitive diagnosis.

These studies provide evidence that the ThinPrep method of screening for cervical cancer is more sensitive than the conventional Pap smear technique, without a decrease in specificity. This difference is most apparent in the detection of LGSIL and when testing is performed in a screening center rather than a hospital based testing laboratory. Most of the studies showed improved specimen adequacy when samples were collected and prepared using the ThinPrep method rather than the conventional method. In addition, there was no increase in the number of specimens reported as inconclusive or atypical with unknown significance when ThinPrep was used.

Although there is substantial evidence that the ThinPrep system provides enhanced detection of cervical lesions with improved specimen adequacy, there are several issues that must be addressed before it can be concluded that ThinPrep should replace conventional Pap smears for routine cervical cancer screening. With the exception of one study 57 where use of the ThinPrep test as an adjunct to conventional Pap testing resulted in an alteration in clinical management, none of the studies reviewed demonstrated that use of ThinPrep alters net health outcome.

It is unclear whether the use of ThinPrep technology would lead to an increase or a reduction in the cost of screening. At the present time, ThinPrep testing costs more per sample and requires the purchase of specialized sample preparation equipment, however, the differential in cost between ThinPrep and conventional Pap smears may become less if ThinPrep improves specimen quality, reduces retesting, and improves ease of cytological evaluation.

Cost-effectiveness of ThinPrep

A core question in discussions of cost-effectiveness of ThinPrep is whether improvements in slide quality and resulting decreases in false-negative rates yield substantial improvements in detecting clinically significant disease given increased costs and charges for Pap testing and screening. In the ThinPrep system, costs are incurred by the clinic and cytology laboratories over and above costs of conventional Pap tests. The manufacturer of the ThinPrep system requires clinics to purchase their sampling and preparation devices to obtain the sample in addition to vials and preservative solution. Samples can be sent only to cytology laboratories equipped with the ThinPrep 2000 processor.

ThinPrep Compared to Conventional Preparation

Appendix IV: Public Comment

Comments received during the public comment period are available by contacting HTAC staff.

References

1.
National Institutes of Health Consensus Development Panel National Institutes of Health consensus development conference statement. cervical cancer, April 1-3, 1996. J Natl Cancer Inst Monographs. 1996:vii–xix.
2.
American Cancer Society What are the key statistics about cancer of the cervix? 1998.
3.
American Cancer Society Cancer facts & figures—1998. 1998. pp. –.
4.
National Cancer Institute Cancer facts: questions and answers about the Pap smear.
5.
Centers for Disease Control and Prevention Regulatory closure of cervical cytology laboratories: recommendations for a public health response. 1997. pp. 1–20. [PubMed: 9409538]
6.
Food and Drug Administration New devices aim at improving Pap test accuracy. FDA Consumer. 1997:(FDA) 97–4264.
7.
American College of Obstetricians and Gynecologists (ACOG) Committee Opinion: recommendations on frequency of Pap test screening. 1995. pp. –. [PubMed: 7649335]
8.
Stehman FB, Bundy BN, DiSaia PJ Carcinoma of the cervix treated with radiation therapy I: a multi-variate analysis of prognostic variables in the Gynecologic Oncology Group. Cancer. 1991;67:2276–2785. [PubMed: 2025841]
9.
Zaino RJ, Ward S, Delgado G, et al Histopathologic predictors of the behavior of surgically treated stage 1B squamous cell carcinoma of the cervix: A Gynecologic Oncology Group study. Cancer. 1992;69:1750–1758. [PubMed: 1551060]
10.
Brisson J, Morin C, Fortier M, et al Risk factors for cervical intraepithelial neoplasia: differences between low-and high-grade lesions. Am J Epidemiol. 1994;140:700–710. [PubMed: 7942772]
11.
Schiffman MH, Bauer HM, Hoover RN, et al Epidemiologic evidence showing that human papillomavirus infection causes most cervical intraepithelial neoplasia. J Natl Cancer Inst. 1993;85:958–964. [PubMed: 8388478]
12.
National Cancer Institute PDQ information for health care professionals: cervical cancer. 1997.
13.
Jones WB, Saigo PE The “atypical” Papanicolaou smear: a professional education publication of the American Cancer Society. 1986.
14.
American College of Obstetricians and Gynecologists (ACOG) Cervical cytology: evaluation and management of abnormalities. ACOG technical bulletin #183. ACOG Technical Bulletin. 1993 [PubMed: 7905444]
15.
American Cancer Society Cervical Cancer-Overview. 1998.
16.
Kurman RJ, Henson DE, Herbst AL, et al Interim guidelines for management of abnormal cervical cytology. JAMA. 1994;271:1866–1869. [PubMed: 8196145]
17.
Minnesota Department of Health, Center for Health Statistics Incidence of invasive cervical cancer and cervical carcinoma in situ, Minnesota. 1998.
18.
Centers for Disease Control and Prevention Behavioral Risk Factor Surveillance System: online prevalence data. 1998.
19.
Minnesota Department of Health 1996 Minnesota HMO Profile. 1998.
20.
Mango LJ Reducing false negatives in clinical practice: the role of neural network technology. Am J Obstet Gynecol. 1996;75:1114–1119. [PubMed: 8885796]
21.
Rosenthal DL, Acosta D, Peters RK Computer-assisted rescreening of clinically important false negative cervical smears using the PAPNET testing system. ActaCytol. 1996;40:120–126. [PubMed: 8604564]
22.
U.S. Department of Health and Human Services Screening Pap smear. Medicare Provider News. 1998;154:3–4.
23.
Rosenthal DL Computerized scanning devices for Pap smear screening: current status and critical review. Clinics in Laboratory Medicine. 1997;17:263–284. [PubMed: 9243073]
24.
Williamson SLH, Hair T, Wadehra V The effects of different sampling techniques on smear quality and the diagnosis of cytological abnormalities in cervical screening. Cytopathology. 1997;8:188–195. [PubMed: 9202894]
25.
Frable WJ Does a zero error standard exist for the Papanicolaou smear? Arch Pathol Lab Med. 1997;121:301–310. [PubMed: 9111124]
26.
DeMay RM Cytopathology of false negatives preceding cervical carcinoma. Am J Obstet Gynecol. 1996;175:1110–1113. [PubMed: 8885795]
27.
U.S. Department of Health and Human Services, Public Health Service Healthy People 2010 Eliminating racial and ethnic disparities in health. 1998.
28.
Eisenberger D, Hernandez E, Tener T, Atkinson BF Order of endocervical and ectocervical cytologic sampling and the quality of the Papanicolaou smear. Obstet Gynecol. 1997;90:755–758. [PubMed: 9351759]
29.
Buntinx F, Brouwers M Relation between sampling device and detection of abnormality in cervical smears: a meta-analysis of randomized and quasi-randomized studies. BMJ. 1998;313:1285–1290. [PMC free article: PMC2352737] [PubMed: 8942687]
30.
Health Care Financing Administration, U.S.Department of Health and Human Services Title 42: Public Health; Chapter IV Health Care Financing Administration, Department of Health and Human Services, Part 493: Laboratory requirements. 1997;42CFR493:Title42, Vol 3-Parts 430-end.
31.
Shingleton HM, Patrick RL, Johnston WW, Smith RA The current status of the Papanicolaou smear. CA Cancer J Clin. 1995;45:305–320. [PubMed: 7656133]
32.
McGoogan E, Reith A Would monolayers provide more representative samples and improved preparations for cervical screening? Overview and evaluation of systems available. Acta Cytol. 1996;49:107–119. [PubMed: 8604562]
33.
Martin EC New Pap test faces hurdles over cost: Cytyc signs deal with United HMO. Boston Business Journal. 1998
34.
Minnesota Department of Human Services Pap test goals for state HMO contracts. 1998.
35.
American College of Obstetricians and Gynecologists (ACOG) ACOG Committee Opinion: New Pap Test Screening Techniques. ACOG News Release #206. 1998. [PubMed: 9989906]
36.
Wilbur DC, Prey MU, Miller WM, et al The AutoPap System for primary screening in cervical cytology: comparing the results of a prospective, intended-use study with routine manual practice. Acta Cytol. 1998;42:214–220. [PubMed: 9479343]
37.
Intersociety Working Group for Cytology Technologies Proposed guidelines for primary screening instruments for gynecologic cytology. Acta Cytol. 1997;41:924–934. [PubMed: 9167728]
38.
Lee JSJ, Wilhelm P, Kuan L, et al AutoPap system performance in screening for low prevalence and small cell abnormalities. Acta Cytol. 1997;41:56–64. [PubMed: 9022727]
39.
Neuromedical Systems Incorporated (NSI) PapNet. Suffern, NY. 1998.
40.
NeoPath Incorporated NeoPath charges for AutoPap screening. Redmond, WA. 1998.
41.
NeoPath Incorporated AutoPap primary screening system: Product Insert. Redmond, WA. 1998.
42.
Minnesota Department of Human Services 1998.
43.
Canadian Coordinating Office for Health Technology Assessment (CCOHTA) Assessment of techniques for cervical cancer screening. 1997;CCOHTA. 1997. pp. 1–33.
44.
O'Leary TJ, Tellado M, Buckner SB, et al PAPNET-assisted rescreening of cervical smears: cost and accuracy compared with a 100% manual rescreening strategy. JAMA. 1998;279:235–237. [PubMed: 9438746]
45.
Schechter CB Cost-effectiveness of rescreening conventionally prepared cervical smears by PAPNET testing. Acta Cytol. 1996;40:1272–1282. [PubMed: 8960040]
46.
Radensky PW, Mango LJ Interactive neural network-assisted screening: an economic assessment. Acta Cytol. 1998;42:246–252. [PubMed: 9479347]
47.
Sherman ME, Schiffman MH, Lorincz AT Cervical specimens collected in liquid buffer are suitable for both cytologic screening and ancillary human papillomavirus testing. Cancer (Cancer Cytopath) 1997;81:89–97. [PubMed: 9126136]
48.
Ferenczy A, Robitaille J, Franco E, et al Conventional cervical cytologic smears vs. ThinPrep smears: a paired comparison on cervical cytology. Acta Cytol. 1996;40:1136–1142. [PubMed: 8960019]
49.
Gray Sheet Technology reimbursement proposals to private payer should include patient selection criteria, Blues plan exec says, side-by-side comparisons also sought. Gray sheet. 1998;24:–.
50.
Maremont M Cytyc test for cervical cancer offers minimal advancement study says. Wall Street Journal. 1998
51.
Farnsworth A, Chambers FM, Goldschmidt CS Evaluation of the PAPNET system in a general pathology service. Med J Aust. 1996;165:429–431. [PubMed: 8913244]
52.
Jenny J, Isenegger I, Boon M, Husain OAN Consistency of a double PAPNET scan of cervical smears. Acta Cytol. 1997;41:82–87. [PubMed: 9022731]
53.
Kok MR, Boon ME Consequences of neural network technology for cervical screening: increase in diagnostic consistency and positive scores. Cancer. 1996;78:112–117. [PubMed: 8646706]
54.
Colgan TJ, Patten SF, Lee JSJ A clinical trial of the AutoPap 300 QC system for quality control of cervicovaginal cytology in the clinical laboratory. Acta Cytol. 1995;39:1191–1198. [PubMed: 7483997]
55.
Wilbur DC, Cibas ES, Merritt S, et al ThinPrep processor: clinical trials demonstrate an increased detection rate of abnormal cervical ctyologic specimens. Am J Clin Pathol. 1994;101:209–214. [PubMed: 8116577]
56.
Stevens MW, Milne AJ, James KA, et al Effectiveness of automated cervical cytology rescreening using the AutoPap 300 QC system. Diagn Cytopathol. 1997;15:505–512. [PubMed: 9181316]
57.
Roberts JM, Gurley AM, Thurloe JK, et al Evaluation of the ThinPrep Pap test as an adjunct to the conventional Pap smear. Medical Journal of Australia. 1997;167:466–469. [PubMed: 9397059]
58.
Lee KR, Raheela A, Birdsong GG, et al Comparison of conventional Papanicolaou smears and a fluid-based, thin-layer system for cervical cancer screening. Obstet Gynecol. 1997;90:278–284. [PubMed: 9241308]
59.
Corkill M, Knapp D, Martin J, Hutchinson ML Specimen adequacy of ThinPrep sample preparations in a direct-to-vial study. Acta Cytol. 1997;41:39–44. [PubMed: 9022724]
60.
Weintraub J The coming revolution in cervical cytology: a pathologist's guide for the clinician. References en Gynecologie Obstetrique. 1997;5:1–6.
61.
Corkill M, Knapp D, Hutchinson ML Improved accuracy for cervical cytology with the ThinPrep method and the endocervical brush-spatula collection procedure. J Lower Genital Tract Disease. 1998;2:12–16. [PubMed: 25951356]
62.
Inhorn SL, Wilbur D, Zahniser D, Linder J Validation of the ThinPrep Papanicolaou test for cervical cancer diagnosis. Journal of Lower Genital Tract Disease. 1998;2:208–212. [PubMed: 25950214]
63.
Sherman ME, Mendoza M, Lee KR, et al Performance of liquid-based, thin-layer cervical cytology: correlation with reference diagnoses and human papillomavirus testing. Mod Pathol. 1998;11:837–843. [PubMed: 9758363]
64.
Bolick DR, Hellman DJ Laboratory implementation and efficacy assessment of the ThinPrep cervical cancer screening system. Acta Cytol. 1998;42:209–213. [PubMed: 9479342]
65.
Dupree WB, Suprun HZ, Beckwith DG, et al The promise and risk of a new technology: the Lehigh Valley Hospital's experience with liquid-based cervical cytology. Cancer (Cancer Cytopathology) 1998;84:202–207. [PubMed: 9723594]
66.
Papillo JL, Zarka MA, St.John TL Evaluation of the ThinPrep Pap Test in Clinical Practice: a seven month, 16,314-case experience in northern Vermont. Acta Cytol. 1998;42:203–208. [PubMed: 9479341]