(61 days)
The Vysis® AutoVysion™ System is an automated scanning microscope and image analysis system. It is intended for in vitro diagnostic use with the Vysis® PathVysion® HER-2 DNA Probe Kit to aid in the detection and enumeration of FISH signals in interphase nuclei. and to determine the LSI® HER-2 to CEP® 17 signal ratio of the HER-2/neu gene via FISH in formalinfixed, paraffin-embedded human breast cancer tissue specimens. The AutoVysion System is intended to reduce overall hands-on time by performing automated enumeration. For a small percentage of samples (less than 7%) manual enumeration may be required.
The Vysis® AutoVysion™ System is an adjunctive computer-assisted methodology to assist in the acquisition and measurement of images from microscope slides of formalin-fixed, paraffin-embedded breast cancer tissue sections for the presence of amplified HER-2/neu gene. The Vysis® AutoVysion™ System is intended for use as aid in determining HER-2/neu amplification status. in coniunction with optional manual visualization directly through the fluorescence microscope.
When used with the Vysis® PathVysion® HER-2 DNA Probe Kit, the Vysis® AutoVysion™ System is indicated for use as
a) an adjunct to existing clinical and pathologic information currently used as prognostic factors in stage II, node-positive breast cancer patients
b) an aid to predict disease-free and overall survival in patients with stage II. node positive breast cancer treated with adjuvant cyclophosphamide, doxorubicin and 5-fluorouracil (CAF) chemotherapy; and,
c) aid in the assessment of patients for whom HERCEPTIN® (Trastuzumab) treatment is being considered (see HERCEPTIN package insert).
The Vysis® AutoVysion™ System consists of an automated fluorescence microscope with motorized scanning stage, a large-format monochrome CCD camera, computer and scanning and assay specific analysis software. The microscope is equipped with a mercury arc lamp for fluorescence epi-illumination; three single-pass fluorescence filter sets for DAPI, SpectrumGreen™ (SG) and SpectrumOrange™ (SO) and a triple-pass fluorescence filter set for DAPI/SG/SQ, all mounted in a motorized filter turret; 10x and 40x objectives in a motorized objective turret; 10x eyepieces; a CCD camera; and a motorized scanning stage that holds up to 8 slides. Images of single fluorescence colors are captured by the CCD camera and transferred to the computer. All functions are controlled by the System software.
Here is a summary of the acceptance criteria and the study proving the device meets them, based on the provided text:
1. Table of Acceptance Criteria and Reported Device Performance
| Acceptance Criteria Category | Specific Metric | Acceptance Criteria (Implicit) | Reported Device Performance |
|---|---|---|---|
| Reproducibility | Day-to-day LSI HER-2/neu to CEP 17 ratio variability | No statistically significant differences in variance across study days (p-value > 0.05 for Levene's test). | All p-values for Levene's test (testing homogeneity of day-to-day variances) were > 0.05, indicating no statistically significant differences. CVs ranged from 5.12% to 29.34%. (Note: For one data point, SD and %CV were "NR" - Not Reported). |
| Reproducibility | Inter-site LSI HER-2/neu to CEP 17 ratio variability | No statistically significant differences in variance across study sites. | All p-values for inter-site reproducibility were > 0.05, indicating no statistically significant differences. CVs ranged from 1.02% to 31.23%. (Note: For one data point, SD and %CV were "NR" - Not Reported). |
| Method Comparison | Overall Concordance with Manual Enumeration | High agreement with manual enumeration. (Specific numerical threshold not explicitly stated but implied by "correctly classified"). | 92.5% (196/212) overall agreement for all informative results. If equivocal range (1.5-3.0) excluded, total agreement was 98.8% (169/171). |
| Method Comparison | Positive Agreement with Manual Enumeration | High positive agreement. | 96.0% positive agreement for all informative results. If equivocal range excluded, 100% positive agreement. |
| Method Comparison | Negative Agreement with Manual Enumeration | High negative agreement. | 89.2% negative agreement for all informative results. If equivocal range excluded, 97.5% negative agreement. |
| Method Comparison | Bias relative to Manual Enumeration (1.18-4.49 ratio) | Average bias within ±15% of the average manual enumeration ratio in this range (as per NCCLS guideline EP9-A2). | Average bias was 0.472 (SD = 1.24), which is 11.7% of the average manual enumeration ratio (4.05). This met the acceptable error of ±15%. |
| Re-tests of Discrepancies | Consistency for initial "false positives" | Repeat tests should demonstrate consistency, indicating the initial discrepancy might be due to variability rather than systematic error. | Two "false positive" results by AutoVysion™ were re-tested six times manually and by the scanner. Both methods (manual and scanner) gave positive results in five of the six repeats for these two samples, suggesting consistency on re-testing. |
2. Sample Size Used for the Test Set and the Data Provenance
- Reproducibility Test Set: 36 specimen slides prepared from four human breast tissue specimens (one normal, one borderline, one moderate, one high amplification). Each site received three of each specimen randomized over three days.
- Method Comparison Test Set: 234 clinical slides from 39 tumors with varying levels of HER-2/neu copy number.
- Data Provenance: Not explicitly stated, but it conducted at "3 clinical sites" and refers to "clinical slides from 39 tumors," implying a source from a clinical setting. It is a retrospective analysis in the sense that the slides were prepared "prior to shipment to the study sites" and then retrospectively analyzed using both methods. However, the exact country of origin or broader demographic information is not provided.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and the Qualifications of Those Experts
- For the reproducibility and method comparison studies, the "ground truth" or reference method was conventional manual microscopy performed in accordance with Vysis® PathVysion® HER-2 DNA Probe Kit.
- The study states: "A qualified user visually inspects the tumor regions...identifies areas of tumor invasion...and records the coordinates." and "Final review and reporting of sample results is performed by a qualified user."
- It also states that the predicate method was "manual microscopy performed in accordance with Vysis® PathVysion® HER-2 DNA Probe Kit." This implies that the standard manual reading by trained personnel using the established kit constitutes the reference.
- The number of experts/manual readers for the comparison is not explicitly given, but it implies standard clinical practice where at least one "qualified user" would perform the manual enumeration. For the "two false positive results," the "manual" repetition implies multiple manual reads.
- Qualifications of experts: Described as "qualified user" or standard manual enumeration. Specific qualifications (e.g., "radiologist with 10 years of experience") are not provided.
4. Adjudication Method for the Test Set
- No formal adjudication method (e.g., 2+1, 3+1) is explicitly described for the initial ground truth establishment. The "ground truth" is defined by the results of "conventional manual microscopy performed in accordance with Vysis® PathVysion® HER-2 DNA Probe Kit." This suggests a single or standard manual read serves as the reference.
- For the two "false positive" discrepancies, a re-testing protocol was used where samples were "repeated six times manually and by the scanner," which could be considered a form of informal adjudication/re-evaluation for specific discrepant cases.
5. If a Multi Reader Multi Case (MRMC) Comparative Effectiveness Study was Done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance
- No, a MRMC comparative effectiveness study was not performed in the context of human readers improving with AI vs without AI assistance.
- This study is a standalone performance study that compares the automated system's performance directly against the manual enumeration (the gold standard at the time).
- The device is described as an "adjunctive computer-assisted methodology to assist in the acquisition and measurement of images...in conjunction with optional manual visualization directly through the fluorescence microscope." However, the study presented compares the automated system's result to manual enumeration rather than measuring how human performance changes when using the system as an aid.
6. If a Standalone (i.e. algorithm only without human-in-the loop performance) was done
- Yes, a standalone performance study was done. The "Method comparison" section directly compares the "Vysis® AutoVysion™ System produced HER-2 to CEP 17 signal ratio" to "manually enumerated... HER-2 to CEP 17 signal ratio."
- The system operates "fully automatically" once target areas are identified by a user, then "hybridization signals in each area are detected and enumerated automatically." Final review and reporting are performed by a qualified user, but the core enumeration is automated. The performance metrics reported (concordance, agreement, bias) are for this standalone automated enumeration process compared to manual.
7. The Type of Ground Truth Used (expert consensus, pathology, outcomes data, etc.)
- The ground truth used was expert manual enumeration (or "conventional manual microscopy") following the Vysis® PathVysion® HER-2 DNA Probe Kit protocol. This is considered the established clinical method for determining HER-2/neu amplification status at the time.
8. The Sample Size for the Training Set
- The document does not specify the sample size for the training set for the AutoVysion™ System. It only describes the test sets used for analytical performance evaluations.
9. How the Ground Truth for the Training Set Was Established
- The document does not describe how the ground truth for any potential training set was established. It focuses solely on the validation/performance evaluation of the already developed system.
§ 866.4700 Automated fluorescence
in situ hybridization (FISH) enumeration systems.(a)
Identification. An automated FISH enumeration system is a device that consists of an automated scanning microscope, image analysis system, and customized software applications for FISH assays. This device is intended for in vitro diagnostic use with FISH assays as an aid in the detection, counting and classification of cells based on recognition of cellular color, size, and shape, and in the detection and enumeration of FISH signals in interphase nuclei of formalin-fixed, paraffin-embedded human tissue specimens.(b)
Classification. Class II (special controls). The special control is FDA's guidance document entitled “Class II Special Controls Guidance Document: Automated Fluorescencein situ Hybridization (FISH) Enumeration Systems.” See § 866.1(e) for the availability of this guidance document.