(192 days)
The ScanScope® System is an automated digital slide creation, management, viewing and analysis system. It is intended for in vitro diagnostic use as an aid to the pathologist in the display, detection, counting and classification of tissues and cells of clinical interest based on particular color, intensity, size, pattern and shape.
The IHC HER2 Manual Read of Digital Slides application is intended for use as an aid to the pathologist in the detection and semi-quantitative measurement of HER-2/neu (cerbB-2) by manual examination of the digital slide of formalin-fixed, paraffin-embedded normal and neoplastic tissue immunohistochemically stained for HER-2 receptors on a computer monitor. HER2 results are indicated for use as an aid in the management, prognosis and prediction of therapy outcomes of breast cancer.
The IHC HER2 Manual Read of Digital Slides application is intended for use as an accessory to the Dako HercepTest™ to aid the pathologist in the detection and semi-quantitative measurement of HER-2/neu (cerbB-2) by manual examination of the digital slide of formalin-fixed, paraffin-embedded normal and neoplastic tissue immunohistochemically stained for HER-2 receptors on a computer monitor. When used with the Dako HercepTest™, it is indicated for use as an aid in the assessment of breast cancer patients for whom HERCEPTIN® (Trastuzumab) treatment is being considered. Note: The actual correlation of the Dako HercepTest™ to Herceptin® clinical outcome has not been established.
The ScanScope ® System is an automated digital slide creation, management, viewing and analysis system. The ScanScope® System components consist of an automated digital microscope slide scanner, computer, color monitor, keyboard and digital pathology information management software. The system capabilities include digitizing microscope slides at high resolution, storing and managing the resulting digital slide images, retrieving and displaying digital slides, including support for remote access over wide-area networks, providing facilities for annotating digital slides and entering and cditing metadata associated with digital slides, and facilities for image analysis of digital slides. Image analysis capabilities include the ability to detect and quantify characteristics useful to Pathologists, such as detecting and quantifying certain proteins revealed by immunohistochemical stains applied to histology specimens. The remote digital slide viewing capabilities of the system support reading digital slides on a computer monitor, enabling Pathologists to make clinically relevant decisions analogous to those they make using a conventional microscope. Specifically, the system supports the pathologist in the detection and semi-quantitative measurement of HER-2/neu (cerbB-2) by manual examination of the digital slide of formalin-fixed, paraffin-embedded normal and neoplastic tissue immunohistochemically stained for HER-2 receptors on a computer monitor.
Here's an analysis of the acceptance criteria and the study proving the device meets them, based on the provided text:
1. Table of Acceptance Criteria and Reported Device Performance
The document does not explicitly state pre-defined acceptance criteria (e.g., "The device must achieve X% agreement"). Instead, it focuses on demonstrating "substantial equivalence" through agreement percentages within and between pathologists and methods. The performance is presented as ranges and specific percentages.
| Acceptance Criteria (Implied) | Reported Device Performance |
|---|---|
| Clinical Comparison: Agreement between manual microscopy and manual digital slide reading (same pathologist) | Range from 61.3% to 92.5% with 95% CI from 49.7% to 97.2%. |
| Clinical Comparison: Inter-Pathologist Agreement (Manual Microscopy) | Range from 76.3% to 91.3% with 95% CI from 65.4% to 96.4%. |
| Clinical Comparison: Inter-Pathologist Agreement (Manual Digital Slide Reading) | Range from 70.0% to 86.0% with 95% CI from 58.7% to 92.1%. |
| Analytical Performance (Intra-System Precision) - Calculated HER2 scores | 100% agreement. |
| Analytical Performance (Intra-System Precision) - Cumulative percentages of 3+, 2+ and 1+ cells | Overall average standard deviation of 0.69% (max 2.46%) and average range of 1.22% (max 7.14%). |
| Analytical Performance (Inter-Day/Intra-System Precision) - Calculated HER2 scores | 100% agreement. |
| Analytical Performance (Inter-Day/Intra-System Precision) - Cumulative percentages of 3+, 2+ and 1+ cells | Overall average standard deviation of 0.67% (max 2.43%) and average range of 1.68% (max 12.07%). |
| Analytical Performance (Inter-System Precision) - Calculated HER2 scores | 100% agreement across all systems and all runs. |
| Analytical Performance (Inter-System Precision) - Cumulative percentages of 3+, 2+ and 1+ cells | Overall average standard deviation of 0.78% (max 2.41%) and average range of 1.93% (max 8.95%) when combining results from three ScanScope systems. Individual system st. dev. ranged from 0.57% to 0.69% and average range from 1.14% to 1.22%. |
| Analytical Performance (Intra-Pathologist Precision) - Manual Microscopy Outliers | 2 outliers out of 40 scores (5%). |
| Analytical Performance (Intra-Pathologist Precision) - Manual Digital Slide Reading Outliers | 3 outliers out of 40 scores (7.5%). |
| Analytical Performance (Inter-Pathologist Precision) - Manual Microscopy Outliers | 3 outliers out of 24 scores (12.5%). |
| Analytical Performance (Inter-Pathologist Precision) - Manual Digital Slide Reading Outliers | 5 outliers out of 24 scores (21%). |
2. Sample Size Used for the Test Set and Data Provenance
- Test Set Sample Size:
- Clinical Comparison Study: 180 formalin-fixed, paraffin-embedded breast tissue specimens.
- Precision/Reproducibility Study: 8 HER2 slides (with two slides per HER2 score 0, 1+, 2+, and 3+).
- Data Provenance:
- Clinical Comparison Study: 80 specimens from a "first clinical site" and 100 specimens from a "second clinical site." The text does not specify the country of origin, but it implies a clinical setting within the US given the FDA submission. The study was prospective in the sense that the readings were done as part of the study, rather than re-analyzing existing interpretations. The specimens themselves might have been collected retrospectively from existing tissue banks.
- Precision/Reproducibility Study: The 8 HER2 slides were "sampled from one of the clinical sites." Similar to the above, the country of origin is not explicitly stated but implied to be within the US.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Their Qualifications
- Clinical Comparison Study: Three (3) pathologists at each of the two clinical sites were used, totaling six pathologists for the study (3 per site). The qualifications are stated as "Pathologists," which implies they are medical doctors specialized in pathology. No specific experience level (e.g., 10 years) is given.
- Precision/Reproducibility Study (for initial slide sampling): The "rounded average score of the manual microscopy scores provided by the three pathologists" was used to sample the 8 slides.
- Precision/Reproducibility Study (Intra-Pathologist): One pathologist.
- Precision/Reproducibility Study (Inter-Pathologists): Three pathologists.
4. Adjudication Method for the Test Set
The primary clinical comparison study used a "blinded manual examination" by individual pathologists. The agreement percentages are reported between pathologists and between methods (manual microscopy vs. digital reading for the same pathologist). There is no explicit mention of an adjudication method (like 2+1 or 3+1) to establish a single, definitive ground truth for each case by consensus or a senior expert for the entire test set. Instead, the study design allows for comparing agreement rates.
For the precision studies, "outliers are defined as scores that are different from the median values of the scores provided by the pathologist over 5 runs" (Intra-Pathologist) or "scores that are different from the median values of the scores provided by the three pathologists in this study" (Inter-Pathologist). This is a method for identifying discrepancies, not necessarily for adjudicating a final ground truth for each case for the purpose of primary performance metrics.
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study Was Done, and the Effect Size of How Much Human Readers Improve with AI vs. Without AI Assistance
-
Was an MRMC study done? Yes, a form of MRMC study was performed in the "Comparison studies: a. Method comparison with predicate device." It involved multiple readers (pathologists) and multiple cases (180 specimens). It compared two reading methods: manual microscopy and manual digital slide reading.
-
Effect Size of AI assistance: The device (ScanScope XT System) is described as an "automated digital slide creation, management, viewing and analysis system." However, the "IHC HER2 Manual Read of Digital Slides application" specifically tested in the clinical comparison is for manual examination of digital slides by a pathologist. The document explicitly states: "The IHC HER2 Manual Read of Digital Slides application is intended for use as an aid to the pathologist in the detection and semi-quantitative measurement of HER-2/neu ... by manual examination of the digital slide...".
Therefore, this study does not measure the effect size of AI assistance on human readers. It rather evaluates the equivalence of reading digital slides on a monitor to reading physical slides under a microscope.
While there is a mention of "Aperio's IHC HER2 image analysis algorithm" within the precision/reproducibility section, that part of the study was not for clinical comparison or human reader improvement, but to objectively quantify system variability separate from pathologist variability. The clinical comparison specifically focuses on the manual read function of the digital slides by pathologists.
6. If a Standalone (Algorithm Only Without Human-in-the-Loop Performance) Was Done
Yes, a standalone performance assessment was conducted for the device's analytical capabilities in terms of image analysis on HER2 scores. This specifically evaluated "Aperio's IHC HER2 image analysis algorithm" (not the manual reading by pathologists). The precision studies (Intra-System, Inter-Day/Intra-System, Inter-System) were done using this algorithm, with results showing 100% agreement for calculated HER2 scores and very low standard deviations/ranges for cellular percentages. This demonstrates the performance of the algorithm without human intervention for specific analytical tasks, separate from the clinical "manual read" study.
7. The Type of Ground Truth Used
- Clinical Comparison Study (for the comparison itself): The ground truth for the comparison was essentially established by the pathologists' manual microscopic examination of the glass slides. The digital readings were compared against this traditional method, which serves as the established clinical standard. There isn't an "absolute" or "independent" ground truth like pathology reports from excisional biopsies or patient outcomes explicitly stated for each case in the comparison data. The predicate device for substantial equivalence also relies on manual microscopic assessment.
- Precision/Reproducibility Study (for sampling): The 8 slides used for precision studies were sampled based on the "rounded average score of the manual microscopy scores provided by the three pathologists," implying a consensus-based approach derived from expert readings.
8. The Sample Size for the Training Set
The document does not explicitly state a sample size for a "training set." The clinical study described is a comparison study and precision study, not a study for training or validating an AI model for clinical decision-making with direct human-in-the-loop assistance. While the text mentions "Aperio's IHC HER2 image analysis algorithm," it doesn't provide details on how this algorithm was trained or validated.
9. How the Ground Truth for the Training Set Was Established
Since no specific training set and its ground truth establishment are detailed for an AI model for clinical use in this submission, this information is not available in the provided text. The "ground truth" referenced for the analytical precision studies of the algorithm is inherent in its design to quantify cell features and use a scoring scheme mimicking pathologists' manual approaches.
§ 864.1860 Immunohistochemistry reagents and kits.
(a)
Identification. Immunohistochemistry test systems (IHC's) are in vitro diagnostic devices consisting of polyclonal or monoclonal antibodies labeled with directions for use and performance claims, which may be packaged with ancillary reagents in kits. Their intended use is to identify, by immunological techniques, antigens in tissues or cytologic specimens. Similar devices intended for use with flow cytometry devices are not considered IHC's.(b)
Classification of immunohistochemistry devices. (1) Class I (general controls). Except as described in paragraphs (b)(2) and (b)(3) of this section, these devices are exempt from the premarket notification requirements in part 807, subpart E of this chapter. This exemption applies to IHC's that provide the pathologist with adjunctive diagnostic information that may be incorporated into the pathologist's report, but that is not ordinarily reported to the clinician as an independent finding. These IHC's are used after the primary diagnosis of tumor (neoplasm) has been made by conventional histopathology using nonimmunologic histochemical stains, such as hematoxylin and eosin. Examples of class I IHC's are differentiation markers that are used as adjunctive tests to subclassify tumors, such as keratin.(2) Class II (special control, guidance document: “FDA Guidance for Submission of Immunohistochemistry Applications to the FDA,” Center for Devices and Radiologic Health, 1998). These IHC's are intended for the detection and/or measurement of certain target analytes in order to provide prognostic or predictive data that are not directly confirmed by routine histopathologic internal and external control specimens. These IHC's provide the pathologist with information that is ordinarily reported as independent diagnostic information to the ordering clinician, and the claims associated with these data are widely accepted and supported by valid scientific evidence. Examples of class II IHC's are those intended for semiquantitative measurement of an analyte, such as hormone receptors in breast cancer.
(3) Class III (premarket approval). IHC's intended for any use not described in paragraphs (b)(1) or (b)(2) of this section.
(c)
Date of PMA or notice of completion of a PDP is required. As of May 28, 1976, an approval under section 515 of the Federal Food, Drug, and Cosmetic Act is required for any device described in paragraph (b)(3) of this section before this device may be commercially distributed. See § 864.3.