(245 days)
The Virtuoso system provides automated digital slide creation, management, analysis, and viewing. It is intended for in vitro diagnostic use as an aid to the pathologist in the display, detection, counting, review and classification of tissues and cells of clinical interest based on particular morphology, color, intensity, size, pattern and shape.
The Virtuoso™ System for Ki67 (30-9) is for digital read and image analysis applications. This particular Virtuoso system is intended for use as an aid to the pathologist in the detection and semi-quantitative measurement of Ki67 (30-9) protein in formalin-fixed, paraffin-embedded normal and neoplastic tissue. This device is an accessory to the Ventana Medical Systems, Inc. CONFIRM™ anti-Ki67 (30-9) Rabbit Monoclonal Primary Antibody assay. The Ventana Medical Systems, Inc. CONFIRM™ anti-Ki67 (30-9) assay is indicated for use in assessing the proliferative activity of normal and neoplastic breast tissue. When used with this assay, the Virtuoso™ System for Ki67 (30-9) is indicated for use as an aid in the assessment of Ki-67 status in breast cancer patients (but is not the sole basis for treatment).
The Virtuoso™ System is an instrument-plus-software system designed to assist the qualified pathologist in the consistent assessment of protein expression in immunohistochemically stained histologic sections from formalin-fixed, paraffinembedded normal and neoplastic tissues. The system consists of a slide scanner (iScan), computer, monitor, keyboard, mouse, image analysis algorithms for specific immunohistochemical markers, and software with a Windows web browser-based user interface. Virtuoso is a web-based, end-to-end, digital pathology software solution that allows pathology laboratories to acquire, manage, view, analyze, share, and report digital images of pathology specimens. Using the Virtuoso software, the pathologist can view digital images, add annotations, make measurements, perform image analysis, and generate reports.
Hardware: The iScan slide scanning device captures digital images of formalin-fixed, paraffin-embedded tissues that are suitable for storage and viewing. The device includes a digital slide scanner, racks for loading glass slides. computer, scanner software, keyboard, mouse and monitor.
Software: The Virtuoso software is designed to complement the routine workflow of a qualified pathologist in the review of immunohistochemically stained histologic slides. It allows the user to select fields of view (FOVs) in the digital image for analysis and provides quantitative data on these FOVs to assist with interpretation. The software makes no independent interpretations of the data and requires competent human intervention for all steps in the analysis process.
Here's a breakdown of the acceptance criteria and the study details for the VENTANA Virtuoso™ System for IHC Ki-67 (30-9), based on the provided 510(k) summary:
1. Table of Acceptance Criteria and Reported Device Performance
The document does not explicitly state pre-defined "acceptance criteria" in terms of specific thresholds that needed to be met for agreement percentages. Instead, it presents the results of concordance and reproducibility studies. The performance reported below serves as the demonstrated effectiveness of the device.
| Performance Metric | Acceptance Criteria (Implied by reported results) | Reported Device Performance |
|---|---|---|
| Concordance - Digital Read vs. Manual | High agreement between digital read and manual method for Ki-67 classification (≤10% neg, >10% pos). | Overall agreements: 92%, 81%, 86%, 87% across four sites. |
| Negative % Agreement (Digital Read vs. Manual) | 79% (Site 1), 100% (Site 2), 94% (Site 3), 92% (Site 4) | |
| Positive % Agreement (Digital Read vs. Manual) | 100% (Site 1), 69% (Site 2), 80% (Site 3), 81% (Site 4) | |
| Concordance - Image Analysis vs. Manual | High agreement between image analysis and manual method for Ki-67 classification. | Overall agreements: 88%, 86%, 86%, 81% across four sites. |
| Negative % Agreement (Image Analysis vs. Manual) | 79% (Site 1), 95% (Site 2), 86% (Site 3), 65% (Site 4) | |
| Positive % Agreement (Image Analysis vs. Manual) | 93% (Site 1), 81% (Site 2), 86% (Site 3), 100% (Site 4) | |
| Reproducibility - Intra-Pathologist/Inter-Day (Digital Read) | High agreement between repeated digital readings by the same pathologist on different days. | Total agreements ranged from 98% to 100% (across 3 session comparisons). |
| Reproducibility - Intra-Pathologist/Inter-Day (Image Analysis) | High agreement between repeated image analyses by the same pathologist on different days. | Total agreements ranged from 89% to 97% (across 3 session comparisons). |
| Reproducibility - Inter-Pathologist (Digital Read) | Acceptable agreement between digital readings by different pathologists. | Total agreements ranged from 71% to 85%. |
| Reproducibility - Inter-Pathologist (Image Analysis) | Acceptable agreement between image analysis interpretations by different pathologists. | Total agreements ranged from 84% to 93%. |
| Scanner Precision | High percentage agreement for image analysis fields of views across inter-site and intra-site/inter-day scans. | Approximately 90% for every comparison. |
2. Sample Size Used for the Test Set and Data Provenance
- Sample Size:
- Concordance studies:
- Site 1: n = 120
- Site 2: n = 118 (Digital Read), n = 117 (Image Analysis)
- Site 3: n = 114
- Site 4: n = 118 (Digital Read), n = 117 (Image Analysis)
- Reproducibility studies (intra-pathologist):
- The confusion matrices show cell counts of 37-38 for positive and 1-8 for negative detections in Session 1, suggesting a smaller sample size per pathologist for this specific part, but the total number of cases for reproducibility is not explicitly stated.
- Reproducibility studies (inter-pathologist): The confusion matrices show cell counts of ~30-60 for negative and positive detections per site/pathologist comparison, again implying a specific sample size for this portion of the study that is not explicitly stated as a total N.
- Concordance studies:
- Data Provenance: The document does not specify the country of origin of the data. The study was a "primary study" that "evaluated overall system performance across four sites," implying a multi-center study. It does not explicitly state if the data was retrospective or prospective. Given it's a clinical validation study, it would typically be prospective, but this is not confirmed.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Their Qualifications
- The ground truth for the concordance studies (Digital Read vs. Manual, Image Analysis vs. Manual) was established by pathologists performing a "manual method (with a traditional microscope)".
- The number of pathologists involved in establishing the "manual method" ground truth for the test set is not explicitly stated, but it implies at least one pathologist per site for the concordance studies (four sites) and three pathologists for the reproducibility studies.
- Qualifications: Referred to as "qualified pathologist[s]". Specific years of experience or sub-specialty are not provided.
4. Adjudication Method for the Test Set
The document does not describe a formal adjudication method (e.g., 2+1 or 3+1 consensus) for establishing the ground truth from the manual method. It simply states the comparison was made against the pathologist's manual results. For the reproducibility studies, it measures agreement between pathologists or within a pathologist over time, not a specific ground truth adjudication process for the test set itself.
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
- Yes, elements of an MRMC study were conducted:
- The "Agreement/Concordance" section compares the Virtuoso Digital Read and Image Analysis applications against a "Manual Method" performed by pathologists. Each pathologist's Virtuoso digital read results were compared to their manual results. This is a comparison of human performance (manual) against human-assisted performance (digital read and IA).
- The "Reproducibility" section directly assesses inter-pathologist and intra-pathologist variability with the Virtuoso system.
- Effect Size of Human Readers' Improvement with AI vs. Without AI Assistance: The document does not provide a direct "effect size" in terms of how much human readers improve with AI assistance compared to solely manual reading. Instead, it measures the agreement between the AI-assisted methods (Digital Read, Image Analysis) and the "manual method" (considered the true score). For instance, overall agreements for Digital Read vs. Manual ranged from 81-92%, and for Image Analysis vs. Manual ranged from 81-88%. This indicates a high level of agreement, but not an improvement factor in diagnostic accuracy or efficiency.
6. Standalone Algorithm Performance
- Yes, in part. The "Image Analysis" application results (e.g., in the "Virtuoso Image Analysis vs Manual Method" section) represent the performance of the algorithm itself, providing quantitative scores. However, the device description emphasizes that "The software makes no independent interpretations of the data and requires competent human intervention for all steps in the analysis process." The pathologist selects the fields of view and can accept or override the software's score. Therefore, while the algorithm provides a standalone score, its final clinical performance is inherently linked to human interaction. The "Digital Read" performance also represents an AI-assisted mode where the human reads the digital image.
7. Type of Ground Truth Used
- Expert Consensus / Pathologist Interpretation (Manual Method): The ground truth was established by pathologists performing a "manual method (with a traditional microscope)". This means the manual pathological interpretation served as the reference standard against which the digital read and image analysis results were compared.
8. Sample Size for the Training Set
The document does not provide any information regarding the sample size used for the training set of the Virtuoso system's algorithms.
9. How the Ground Truth for the Training Set Was Established
The document does not provide any information on how the ground truth for the training set was established. It only describes the validation studies for the device.
§ 864.1860 Immunohistochemistry reagents and kits.
(a)
Identification. Immunohistochemistry test systems (IHC's) are in vitro diagnostic devices consisting of polyclonal or monoclonal antibodies labeled with directions for use and performance claims, which may be packaged with ancillary reagents in kits. Their intended use is to identify, by immunological techniques, antigens in tissues or cytologic specimens. Similar devices intended for use with flow cytometry devices are not considered IHC's.(b)
Classification of immunohistochemistry devices. (1) Class I (general controls). Except as described in paragraphs (b)(2) and (b)(3) of this section, these devices are exempt from the premarket notification requirements in part 807, subpart E of this chapter. This exemption applies to IHC's that provide the pathologist with adjunctive diagnostic information that may be incorporated into the pathologist's report, but that is not ordinarily reported to the clinician as an independent finding. These IHC's are used after the primary diagnosis of tumor (neoplasm) has been made by conventional histopathology using nonimmunologic histochemical stains, such as hematoxylin and eosin. Examples of class I IHC's are differentiation markers that are used as adjunctive tests to subclassify tumors, such as keratin.(2) Class II (special control, guidance document: “FDA Guidance for Submission of Immunohistochemistry Applications to the FDA,” Center for Devices and Radiologic Health, 1998). These IHC's are intended for the detection and/or measurement of certain target analytes in order to provide prognostic or predictive data that are not directly confirmed by routine histopathologic internal and external control specimens. These IHC's provide the pathologist with information that is ordinarily reported as independent diagnostic information to the ordering clinician, and the claims associated with these data are widely accepted and supported by valid scientific evidence. Examples of class II IHC's are those intended for semiquantitative measurement of an analyte, such as hormone receptors in breast cancer.
(3) Class III (premarket approval). IHC's intended for any use not described in paragraphs (b)(1) or (b)(2) of this section.
(c)
Date of PMA or notice of completion of a PDP is required. As of May 28, 1976, an approval under section 515 of the Federal Food, Drug, and Cosmetic Act is required for any device described in paragraph (b)(3) of this section before this device may be commercially distributed. See § 864.3.