(293 days)
The Virtuoso system provides automated digital slide creation, management, analysis, and viewing. It is intended for in vitro diagnostic use as an aid to the pathologist in the display, detection, counting, review and classification of tissues and cells of clinical interest based on particular morphology, color, intensity, size, pattern and shape.
The Virtuoso™ System for p53 (DO-7) is for digital read and image analysis applications. This particular Virtuoso system is intended for use as an aid to the pathologist in the detection and semi-quantitative measurement of p53 (DO-7) protein in formalin-fixed, paraffin-embedded normal and neoplastic tissue. This device is an accessory to the Ventana Medical Systems, Inc. CONFIRM™ anti-p53 (DO-7) Mouse Monoclonal Primary Antibody assay. The Ventana Medical Systems, Inc. CONFIRM™ anti-p53 assay is indicated for the assessment of p53 protein where mutations have been linked to tumor proliferation. When used with this assay, the Virtuoso™ System for p53 (DO-7) is indicated for use as an aid in the assessment of p53 status in breast cancer patients (but is not the sole basis for treatment).
Note: The IHC p53 (D0-7) Digital Read and Image Analysis applications are adjunctive computer-assisted methodologies for the qualified pathologist in the acquisition and measurement of images from microscope glass slides of breast cancer specimens stained for the presence of p53 protein. The pathologist should verify agreement with the Image Analysis software application score. The accuracy of the test results depends on the quality of the immunohistochemical staining. It is the responsibility of a qualified pathologist to employ appropriate morphological studies and controls as specified in the instructions for the CONFIRMTM anti-p53 (DO-7) Mouse Monoclonal Primary Antibody assay used to assure the validity of the Virtuoso System for IHC p53 Digital Read and Image Analysis scores. The actual correlation of CONFIRM™ anti-p53 (DO-7) Mouse Monoclonal Primary Antibody to clinical outcome has not been established.
The Virtuoso™ System is an instrument-plus-software system designed to assist the qualified pathologist in the consistent assessment of protein expression in immunohistochemically stained histologic sections from formalin-fixed, paraffinembedded normal and neoplastic tissues. The system consists of a slide scanner (iScan), computer, monitor, keyboard, mouse, image analysis algorithms for specific immunohistochemical markers, and software with a Windows web browser-based user interface. Virtuoso is a web-based, end-to-end, digital pathology software solution that allows pathology laboratories to acquire, manage, view, analyze, share, and report digital images of pathology specimens. Using the Virtuoso software, the pathologist can view digital images, add annotations, make measurements, perform image analysis, and generate reports.
Hardware: The iScan slide scanning device captures digital images of formalin-fixed, paraffin-embedded tissues that are suitable for storage and viewing. The device includes a digital slide scanner, racks for loading glass slides, computer, scanner software, keyboard, mouse and monitor.
Software: The Virtuoso software is designed to complement the routine workflow of a qualified pathologist in the review of immunohistochemically stained histologic slides. It allows the user to select fields of view (FOVs) in the digital image for analysis and provides quantitative data on these FOVs to assist with interpretation. The software makes no independent interpretations of the data and requires competent human intervention for all steps in the analysis process.
Here's an analysis of the acceptance criteria and study details for the Virtuoso™ System for IHC p53 (DO-7), based on the provided document:
Acceptance Criteria and Reported Device Performance
The document does not explicitly state pre-defined acceptance criteria with specific thresholds for agreement percentages. Instead, it presents the results of several agreement and reproducibility studies. For the purpose of this summary, the reported performance values themselves are considered against an implicit expectation of high agreement.
| Metric | Acceptance Criteria (Implicit from context) | Reported Device Performance (Virtuoso System) |
|---|---|---|
| Agreement: Virtuoso Digital Read vs Manual | High agreement desirable (e.g., >90%) | Site 1: 93% (CI 87-97%)Site 2: 95% (CI 89-98%)Site 3: 94% (CI 88-97%)Site 4: 82% (CI 73-88%) |
| Agreement: Virtuoso Image Analysis vs Manual | High agreement desirable (e.g., >90%) | Site 1: 92% (CI 85-95%)Site 2: 97% (CI 92-99%)Site 3: 91% (CI 84-95%)Site 4: 90% (CI 83-95%) |
| Intra-Pathologist/Inter-Day Reproducibility (Digital Read) | High agreement desirable (e.g., >90%) | Total agreements ranging from 90% to 95% |
| Intra-Pathologist/Inter-Day Reproducibility (Image Analysis) | High agreement desirable (e.g., >80-90%) | Total agreements ranging from 80% to 93% |
| Inter-Pathologist Reproducibility (Digital Read) | High agreement desirable (e.g., >90%) | Total agreements ranging from 94% to 99% |
| Inter-Pathologist Reproducibility (Image Analysis) | High agreement desirable (e.g., >90%) | Total agreements ranging from 94% to 97% |
| Scanner Precision (Image Analysis) | High agreement desirable (e.g., >90%) | Approximately 90% for every comparison across three image analysis FOVs |
Study Details
The evaluation involved two studies: a primary study for overall system performance and a secondary study for scanner precision.
-
Sample size used for the test set and the data provenance:
- Agreement Studies (Virtuoso vs Manual): The sample sizes for the primary agreement study were 119 cases for Site 1, 119 cases for Site 2, 117 cases for Site 3, and 114 cases for Site 4 for the Digital Read analysis. For Image Analysis, Site 4 had 105 cases.
- Reproducibility Studies: The sample sizes for reproducibility studies are not explicitly stated as total number of cases. However, the confusion matrices for intra-pathologist reproducibility (pages 5 & 6) suggest working with sample sizes in the range of 30-40 cases across sessions (e.g., for Digital Read: 27 Neg, 13 Pos in Session 1, implying around 40 cases). For inter-pathologist reproducibility (pages 6 & 7), the sample sizes range from 119 to ~120 (e.g., Site 1 vs Site 2 had counts summing to 119).
- Scanner Precision Study: A subset of 40 clinical cases from the primary study was used.
- Data Provenance: Not explicitly stated, but the mention of "four sites" suggests data was collected from multiple clinical institutions. It is a clinical validation study and, given the nature of regulatory submissions, is almost certainly retrospective analysis of previously collected and stained slides. No country of origin is specified, but the submission is to the US FDA, implying US (and potentially international) clinical data.
-
Number of experts used to establish the ground truth for the test set and the qualifications of those experts:
- Number of Experts: For the primary study, at least three pathologists were involved for inter-pathologist reproducibility, and each of them performed manual readings for comparison. For the agreement with the manual method, "Each pathologist's Virtuoso digital read results were compared to their manual results." This implies that the manual reading by the same pathologist served as a direct comparison for their digital (DR or IA) reading.
- Qualifications of Experts: The document states the device is "designed to assist the qualified pathologist" and requires "competent human intervention." While specific years of experience are not mentioned, it is implicit that these were qualified pathologists using traditional diagnostic methods.
-
Adjudication method (e.g., 2+1, 3+1, none) for the test set:
- The primary study compared the Virtuoso system's readings (Digital Read and Image Analysis) against the manual method findings of the same pathologist. This means the pathologist's own manual interpretation served as the reference for their digital assessment, rather than an independent adjudicated ground truth from multiple experts.
- The inter-pathologist agreement section compares readings among three pathologists but does not describe an explicit adjudication process (like 2+1 majority vote) to establish a single "ground truth" across them for the reproducibility analysis itself. Instead, it measures the agreement between them.
-
If a multi-reader multi-case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance:
- This document does not describe a MRMC comparative effectiveness study that measures the improvement of human readers with AI assistance vs. without AI assistance.
- Instead, it evaluates the agreement of the AI system's outputs (Digital Read and Image Analysis) with human manual readings, and the reproducibility of human readings, both with and without the digital system (manual vs. digital read, IA vs. manual). It's more of a validation of the digital system as an equivalent tool, rather than an AI-assisted improvement study.
-
If a standalone (i.e. algorithm only, without human-in-the-loop performance) was done:
- The "Image Analysis" portion of the Virtuoso system represents the algorithm's standalone performance, albeit within a workflow where a pathologist selects the fields of view. The software "produces a quantitative score for the FOV and an aggregate score over all the FOVs for the whole slide," and "the pathologist has the choice of accepting the result or overriding with his/her own score."
- The agreement of "Virtuoso Image Analysis vs Manual Method" (page 5) effectively serves as an evaluation of the algorithm's standalone performance (after FOV selection), compared to a human's manual assessment.
-
The type of ground truth used (expert consensus, pathology, outcomes data, etc.):
- The ground truth for comparison was established by the manual interpretation of qualified pathologists using traditional microscopy. This is termed the "reference manual method."
- The p53 classifications were "less than or equal to 10% to describe negative, and greater than 10% to describe positive." This threshold-based binary classification was applied by the pathologists.
-
The sample size for the training set:
- The document does not provide any information about the sample size used for the training set of the Virtuoso system's image analysis algorithms.
-
How the ground truth for the training set was established:
- The document does not provide any information on how the ground truth for the training set was established.
§ 864.1860 Immunohistochemistry reagents and kits.
(a)
Identification. Immunohistochemistry test systems (IHC's) are in vitro diagnostic devices consisting of polyclonal or monoclonal antibodies labeled with directions for use and performance claims, which may be packaged with ancillary reagents in kits. Their intended use is to identify, by immunological techniques, antigens in tissues or cytologic specimens. Similar devices intended for use with flow cytometry devices are not considered IHC's.(b)
Classification of immunohistochemistry devices. (1) Class I (general controls). Except as described in paragraphs (b)(2) and (b)(3) of this section, these devices are exempt from the premarket notification requirements in part 807, subpart E of this chapter. This exemption applies to IHC's that provide the pathologist with adjunctive diagnostic information that may be incorporated into the pathologist's report, but that is not ordinarily reported to the clinician as an independent finding. These IHC's are used after the primary diagnosis of tumor (neoplasm) has been made by conventional histopathology using nonimmunologic histochemical stains, such as hematoxylin and eosin. Examples of class I IHC's are differentiation markers that are used as adjunctive tests to subclassify tumors, such as keratin.(2) Class II (special control, guidance document: “FDA Guidance for Submission of Immunohistochemistry Applications to the FDA,” Center for Devices and Radiologic Health, 1998). These IHC's are intended for the detection and/or measurement of certain target analytes in order to provide prognostic or predictive data that are not directly confirmed by routine histopathologic internal and external control specimens. These IHC's provide the pathologist with information that is ordinarily reported as independent diagnostic information to the ordering clinician, and the claims associated with these data are widely accepted and supported by valid scientific evidence. Examples of class II IHC's are those intended for semiquantitative measurement of an analyte, such as hormone receptors in breast cancer.
(3) Class III (premarket approval). IHC's intended for any use not described in paragraphs (b)(1) or (b)(2) of this section.
(c)
Date of PMA or notice of completion of a PDP is required. As of May 28, 1976, an approval under section 515 of the Federal Food, Drug, and Cosmetic Act is required for any device described in paragraph (b)(3) of this section before this device may be commercially distributed. See § 864.3.