(217 days)
The ScanScope® XT System is an automated digital slide creation, management, viewing and analysis system. It is intended for in vitro diagnostic use as an aid to the pathologist in the display, detection, counting and classification of tissues and cells of clinical interest based on particular color, intensity, size, pattern and shape.
The IHC ER Image Analysis application is intended for use as an aid to the pathologist in the detection and quantitative measurement of ER (Estrogen Receptor) in formalin-fixed paraffinembedded normal and neoplastic tissue.
The IHC PR Image Analysis application is intended for use as an aid to the pathologist in the detection and quantitation measurement of PR (Progesterone Receptor) in formalin-fixed. paraffin-embedded normal and neoplastic tissue.
It is indicated for use as an aid in the management, prognosis, and prediction of therapy outcomes of breast cancer.
Note: The IHC ER and PR Image Analysis applications are an adjunctive computer-assisted methodology to assist the reproducibility of a qualified pathologist in the acquisition and measurement of images from microscope slides of breast cancer specimens stained for the presence of estrogen and progesterone receptor proteins. The accuracy of the test result depends upon the quality of the immunohistochemical staining. It is the responsibility of a qualified pathologist to employ appropriate morphological studies and controls as specified in the instructions for the ER and PR reagent/kit used to assure the validity of the IHC ER and PR Image Analysis application assisted scores.
The system comprises a ScanScope® XT digital slide scanner instrument and a computer system executing Spectrum " software. The system capabilities include digitizing microscope slides at diagnostic resolution, storing and managing the resulting digital slide images, retrieving and displaying digital slides, including support for remote access over wide-area networks, providing facilities for annotating digital slides and entering and editing metadata associated with digital slides, and facilities for image analysis of digital slides, including the ability to quantify characteristics useful to Pathologists, such as measuring and scoring immunohistochemical stains applied to histology specimens, such as Dako ER/PR, which reveal the presence of ER (Estrogen Receptor) protein and PR (Progesterone Receptor) protein expression, which may be used to determine patient treatment for breast cancer.
Hardware Operation: The ScanScope XT digital slide scanner creates seamless true color digital slide images of entire glass slides in a matter of minutes. A high numeric aperture 20x, as found on conventional microscopes, is used to produce high-quality images. (When the 2X magnification changer is inserted, the effective magnification of the images is 40X.) The ScanScope XT employs a linear-array scanning technique that generates images free from optical aberrations along the scanning axis. The result is digital slide images that have no tiling artifacts and are seamless.
Software Operation: The Spectrum software is a full-featured digital pathology management system. The software runs on a server computer called a Digital Slide Repository (DSR), which stores digital slide images on disk storage such as a RAID array, and which hosts an SQL database that contains digital slide metadata. Spectrum includes a web application and services which encapsulate database and digital slide image access for other computers. The Spectrum server supports the capability of running a variety of image analysis algorithms on digital slides, and storing the results of analysis into the database. Spectrum also includes support for locally or remotely connected image workstation computers, which run digital slide viewing and analysis software provided as part of Spectrum.
Overview of System Operation: The laboratory technician or operator loads glass microscope slides into a specially designed slide carrier with a capacity of up to 120 slides. The scanning process begins when the operator starts the ScanScope scanner and finishes when the scanner has completed scanning of all loaded slides. As each glass slide is processed, the system automatically stores individual "striped" images of the tissue contained on the glass slide and integrates the striped images into a single digital slide image, which represents a histological reconstruction of the entire tissue section. After scanning is completed, the operator is able to view and perform certain analytical tests on the digital slides.
Here's a summary of the acceptance criteria and the study that proves the device meets them, based on the provided text:
1. Table of Acceptance Criteria and Reported Device Performance
The acceptance criteria are implied through the results presented, which aim to demonstrate substantial equivalence to manual microscopy. The study primarily focuses on inter-pathologist agreement for both manual microscopy and the device's image analysis, as well as agreement between manual microscopy and the device's image analysis. Precision studies also demonstrate the device's consistency.
Given the document structure, the "acceptance criteria" appear to be defined not as specific numerical thresholds prior to the study, but rather by demonstrating that the device performs comparably to manual microscopy and shows acceptable levels of precision. The reported performance shows the ranges of agreement found.
| Metric | Acceptance Criteria (Implied by Comparison) | Reported Device Performance (Range) |
|---|---|---|
| ER Percentage of Positive Nuclei | Inter-pathologist agreement comparable to manual microscopy; agreement between manual microscopy and AI comparable to inter-pathologist manual agreement. | Inter-pathologist (AI): 93.8%-98.8% |
| Inter-pathologist (Manual): 91.3%-98.8% | ||
| Manual vs. AI: 92.5%-97.5% | ||
| ER Intensity Score | Inter-pathologist agreement comparable to manual microscopy; agreement between manual microscopy and AI comparable to inter-pathologist manual agreement. | Inter-pathologist (AI): 88.8%-90.0% |
| Inter-pathologist (Manual): 55.0%-86.3% | ||
| Manual vs. AI: 63.8%-86.3% | ||
| PR Percentage of Positive Nuclei | Inter-pathologist agreement comparable to manual microscopy; agreement between manual microscopy and AI comparable to inter-pathologist manual agreement. | Inter-pathologist (AI): 85.0%-99.0% |
| Inter-pathologist (Manual): 83.8%-99.0% | ||
| Manual vs. AI: 81.3%-99.0% | ||
| PR Intensity Score | Inter-pathologist agreement comparable to manual microscopy; agreement between manual microscopy and AI comparable to inter-pathologist manual agreement. | Inter-pathologist (AI): 68.8%-88.0% |
| Inter-pathologist (Manual): 58.8%-88.0% | ||
| Manual vs. AI: 58.8%-84% | ||
| ER Percentage of Positive Nuclei Precision (Intra-system) | Low standard deviation and range across runs. | Overall SD: 0.31% (max 0.74%) |
| Avg Range: 0.71% (max 2.25%) | ||
| ER Intensity Score Precision (Intra-system) | Low standard deviation and range across runs. | Overall SD: 0.67 (max 1.45) |
| Avg Range: 1.18 (max 4.88) | ||
| PR Percentage of Positive Nuclei Precision (Intra-system) | Low standard deviation and range across runs. | Overall SD: 0.54% (max 1.47%) |
| Avg Range: 1.06% (max 4.78%) | ||
| PR Intensity Score Precision (Intra-system) | Low standard deviation and range across runs. | Overall SD: 0.9 (max 1.60) |
| Avg Range: 2.48 (max 4.27) | ||
| ER Percentage of Positive Nuclei Precision (Inter-system) | Minimal variation across different ScanScope systems. | Overall Avg SD: 0.55% (max 1.05%) |
| Avg Range: 1.44% (max 4.02%) | ||
| ER Intensity Score Precision (Inter-system) | Minimal variation across different ScanScope systems. | Overall Avg SD: 1.22% (max 3.07%) |
| Avg Range: 2.37% (max 8.91%) | ||
| PR Percentage of Positive Nuclei Precision (Inter-system) | Minimal variation across different ScanScope systems. | Overall Avg SD: 0.87% (max 1.57%) |
| Avg Range: 2.54% (max 8.13%) | ||
| PR Intensity Score Precision (Inter-system) | Minimal variation across different ScanScope systems. | Overall Avg SD: 1.35% (max 2.03%) |
| Avg Range: 4.55% (max 6.86%) |
2. Sample Size for the Test Set and Data Provenance
- ER Study Test Set: 80 formalin-fixed, paraffin-embedded breast tissue specimens.
- Data Provenance: Retrospective, from a single CLIA-qualified clinical site in the US (implied by CLIA qualification, as it's a US regulatory standard). Specimens were "selected based on their clinical scores on file."
- PR Study Test Set: 180 formalin-fixed, paraffin-embedded breast tissue specimens.
- Data Provenance: Retrospective, from two CLIA-qualified clinical sites in the US. 80 slides from the first site (selected based on clinical scores) and 100 slides from the second site (routine clinical specimens, representing the target population).
3. Number of Experts Used to Establish Ground Truth for the Test Set and Qualifications
- Number of Experts: Three board-certified pathologists at each clinical site.
- Qualifications: "Board-certified staff pathologists" at CLIA-qualified clinical sites. (No specific years of experience are mentioned).
4. Adjudication Method for the Test Set
The document describes a form of expert consensus and comparison rather than a strict adjudication to arrive at a single "ground truth" value for the test set.
- For Manual Microscopy: Three different board-certified pathologists at each clinical site performed a blinded manual review of each glass slide. They reported the percentage of positive nuclei and average intensity score. The study then uses the "manual microscopy average percentages of positive nuclei from the three pathologists" and "manual microscopy average intensity scores from the three pathologists" for comparisons. This suggests an averaging approach rather than a specific adjudication rule (e.g., 2-out-of-3 majority; a 3+1 method where a fourth expert adjudicates disagreements is not explicitly stated).
- For Image Analysis: Each of the three pathologists outlined tumor regions on digital slides (blinded from each other and from image analysis results). Image analysis was then performed on each set of outlined regions, resulting in a separate image analysis score for each of the three pathologists. No formal adjudication is described to combine these three algorithm scores into a single "ground truth" for the algorithm; rather, the agreement between the pathologists' manual scores and their respective image analysis scores is evaluated, as well as inter-pathologist agreement for both methods.
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was done, and its effect size:
- This was a type of MRMC study, as multiple readers (pathologists) evaluated multiple cases (slides) both manually and with the assistance of the device (though the AI component was run after and blinded from the pathologists' region selection).
- Effect Size: The document does not report an effect size for how much human readers improve with AI vs. without AI assistance. Instead, it reports agreement percentages between pathologists' manual scores, between pathologists' AI-assisted scores, and between a pathologist's manual score and their corresponding AI-assisted score. The study's focus was on demonstrating substantial equivalence and agreement, not on measuring reader improvement with assistance.
6. If a standalone (i.e., algorithm-only without human-in-the-loop performance) was done:
- Yes, in part. The image analysis algorithm reported the percentage of positive nuclei and average intensity score for each digital slide. However, the input to the algorithm (the tumor regions) was still defined by human pathologists. The critical step of selecting the region of interest was human-in-the-loop, even if the quantitative analysis within that region was standalone.
- The document states: "Image analysis was run in batch processing mode completely separated from the pathologists outlining the tumor regions to avoid influencing the pathologists in their choice of tumor regions." This clarifies that the numerical output of the algorithm was standalone for a given region, but the selection of that region itself was pathologist-driven.
7. The Type of Ground Truth Used
- Expert Consensus (Averaged): For comparing the device performance, the "ground truth" for manual microscopy was established by taking the average percentage of positive nuclei and average intensity scores from three board-certified pathologists. This serves as the reference against which the digital system's performance (also tied to pathologist-defined regions) is compared.
- Not Pathology, but based on Pathology Scores: While pathology slides were used, the ground truth was not solely an independent pathology report in the traditional sense, but rather the statistically combined scores of the evaluating pathologists who were part of the study.
- Not Outcomes Data: The study did not use patient outcomes data.
8. The Sample Size for the Training Set
- The document does not specify the sample size used for the training set for the image analysis algorithm. The provided information focuses entirely on the clinical validation study (test set).
9. How the Ground Truth for the Training Set was Established
- The document does not provide information on how the ground truth for the training set was established, as the training set details are not described.
§ 864.1860 Immunohistochemistry reagents and kits.
(a)
Identification. Immunohistochemistry test systems (IHC's) are in vitro diagnostic devices consisting of polyclonal or monoclonal antibodies labeled with directions for use and performance claims, which may be packaged with ancillary reagents in kits. Their intended use is to identify, by immunological techniques, antigens in tissues or cytologic specimens. Similar devices intended for use with flow cytometry devices are not considered IHC's.(b)
Classification of immunohistochemistry devices. (1) Class I (general controls). Except as described in paragraphs (b)(2) and (b)(3) of this section, these devices are exempt from the premarket notification requirements in part 807, subpart E of this chapter. This exemption applies to IHC's that provide the pathologist with adjunctive diagnostic information that may be incorporated into the pathologist's report, but that is not ordinarily reported to the clinician as an independent finding. These IHC's are used after the primary diagnosis of tumor (neoplasm) has been made by conventional histopathology using nonimmunologic histochemical stains, such as hematoxylin and eosin. Examples of class I IHC's are differentiation markers that are used as adjunctive tests to subclassify tumors, such as keratin.(2) Class II (special control, guidance document: “FDA Guidance for Submission of Immunohistochemistry Applications to the FDA,” Center for Devices and Radiologic Health, 1998). These IHC's are intended for the detection and/or measurement of certain target analytes in order to provide prognostic or predictive data that are not directly confirmed by routine histopathologic internal and external control specimens. These IHC's provide the pathologist with information that is ordinarily reported as independent diagnostic information to the ordering clinician, and the claims associated with these data are widely accepted and supported by valid scientific evidence. Examples of class II IHC's are those intended for semiquantitative measurement of an analyte, such as hormone receptors in breast cancer.
(3) Class III (premarket approval). IHC's intended for any use not described in paragraphs (b)(1) or (b)(2) of this section.
(c)
Date of PMA or notice of completion of a PDP is required. As of May 28, 1976, an approval under section 515 of the Federal Food, Drug, and Cosmetic Act is required for any device described in paragraph (b)(3) of this section before this device may be commercially distributed. See § 864.3.