(279 days)
Genius AI Detection is a computer-aided detection and diagnosis (CADe/CADx) software device intended to be used with compatible digital breast tomosynthesis (DBT) systems to identify and mark regions of interest including soft tissue densities (masses, architectural distortions and asymmetries) and calcifications in DBT exams from compatible DBT systems and provide confidence scores that offer assessment for Certainty of Findings and a Case Score.
The device intends to aid in the interpretation of digital breast tomosynthesis exams in a concurrent fashion, where the interpreting physician confirms or dismisses the findings during the reading of the exam.
Genius AI Detection 2.0 is a software device intended to identify potential abnormalities in breast tomosynthesis images. Genius AI Detection 2.0 analyzes each standard mammographic view in a digital breast tomosynthesis examination using deep learning networks. For each detected lesion, Genius AI Detection 2.0 produces CAD results that include:
- the location of the lesion;
- an outline of the lesion;
- a confidence score for the lesion
- Genius AI Detection 2.0 also produces a case score for the entire breast tomosynthesis exam.
Genius AI Detection 2.0 packages all CAD findings derived from the corresponding analysis of a tomosynthesis exam into a DICOM Mammography CAD SR object and distributes it for display on DICOM compliant review workstations. The interpreting physician will have access to the CAD findings concurrently to the reading of the tomosynthesis exam. In addition, a combination of peripheral information such as number of marks and case scores may be used on the review workstation to enhance the interpreting physician's workflow by offering a better organization of the patient worklist.
Here's a breakdown of the acceptance criteria and study details for Genius AI Detection 2.0, based on the provided FDA 510(k) clearance letter:
Acceptance Criteria and Device Performance for Genius AI Detection 2.0
1. Table of Acceptance Criteria and Reported Device Performance
The provided document describes a non-inferiority study to demonstrate that the performance of Genius AI Detection 2.0 on Envision (ENV) images is equivalent to its performance on the predicate's Standard of Care (SOC) images (Hologic's Selenia Dimensions systems). The primary acceptance criterion was non-inferiority of the Area Under the Curve (AUC) of the ROC curve, with a 5% margin. Secondary metrics included sensitivity, specificity, and false marker rate per view.
| Acceptance Criteria Category | Specific Metric | Predicate Device Performance (SOC Images) | Subject Device Performance (ENV Images) | Acceptance Criteria Met? |
|---|---|---|---|---|
| Primary Endpoint (Non-Inferiority) | AUC of ROC Curve (ENV-SOC) | N/A (Comparison study) | -0.0017 (95% CI -0.023 - 0.020) | Yes (p-value for difference = 0.87, indicating no significant difference, and within 5% non-inferiority margin) |
| Secondary Metrics | Sensitivity | N/A (Comparison study) | No significant difference reported between modalities | Yes |
| Specificity | N/A (Comparison study) | No significant difference reported between modalities | Yes | |
| False Marker Rate per View | N/A (Comparison study) | No significant difference reported between modalities | Yes | |
| CC-MLO Correlation | Accuracy on Malignant Lesions | N/A | 90% | Yes (Considered accurate) |
| Accuracy on Negative Cases (Correlated pairs) | N/A | 73% | Yes (Considered accurate) | |
| Implant Cases | Location-specific cancer detection sensitivity | N/A | 76% (CI 68%~84%) | Yes (Considered acceptable based on confidence intervals) |
| Specificity | N/A | 67% (CI 62%~72%) | Yes (Considered acceptable based on confidence intervals) |
(Note: The document focuses on demonstrating equivalence to the predicate's performance on a new platform rather than absolute performance against a fixed threshold for all metrics, except for the implant case where specific CIs are given and deemed acceptable.)
2. Sample Size Used for the Test Set and Data Provenance
- Sample Size (Main Comparison Study): 1475 subjects
- 200 biopsy-proven cancer subjects
- 275 biopsy-proven benign subjects
- 78 BI-RADS 3 subjects (considered BI-RADS 1 or 2 upon diagnostic workup)
- 922 BI-RADS 1 and 2 subjects (at screening)
- Implant Case Test Set: 480 subjects
- 132 biopsy-proven cancer subjects
- 348 negative subjects (119 biopsy-proven benign, 229 screening negative)
- Data Provenance:
- Country of Origin: Not explicitly stated, but collected from a "national multi-center breast imaging network" within the U.S., implying U.S. origin.
- Retrospective or Prospective: The main comparison study data was collected for evaluating the safety and effectiveness of the Envision platform, with an IRB approved protocol. This suggests a retrospective study design, where existing images were gathered for evaluation. The implant cases were collected between 2015 and 2022, also indicating a retrospective approach.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications
- Number of Experts: Two
- Qualifications: Both were MQSA-certified radiologists with over 20 years of experience.
4. Adjudication Method for the Test Set
The document explicitly states that the "ground truthing to evaluate performance metrics including the locations of cancer lesions was done by two MQSA-certified radiologists with over 20 years of experience."
- Adjudication Method: It does not specify a particular adjudication method (e.g., 2+1, 3+1). It simply states that ground truthing was done by two experts. This implies either consensus was reached between the two, or potentially an unstated arbitration method if they disagreed, or that their individual findings were used for analysis. Given the phrasing, expert consensus is the most likely implied method, but not explicitly detailed.
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was done
- No, an MRMC comparative effectiveness study was NOT done. The study described is a standalone performance comparison of the AI algorithm on images from different modalities (Envision vs. Standard of Care), not a study involving human readers with and without AI assistance to measure effect size.
6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) was done
- Yes, a standalone study WAS done. The document explicitly states, "A standalone study was conducted to compare the detection performance of FDA cleared Genius AI Detection 2.0 (K221449) using Standard of Care (SOC) images acquired on the Dimensions systems against images acquired on the FDA approved Envision Mammography Platform (P080003/S009)." This study evaluated the algorithm's performance (fROC, ROC, sensitivity, specificity, false marker rate) directly against the ground truth without human intervention.
7. The Type of Ground Truth Used
- Ground Truth Type: A combination of biopsy-proven cancer and biopsy-proven benign cases, along with BI-RADS diagnostic outcomes (for negative cases). For the cancer cases, the "locations of cancer lesions" were part of the ground truth.
8. The Sample Size for the Training Set
- Not provided. The document states that the test dataset was "sequestered from any training datasets by isolating it on a secured server with controlled access permissions" and that the data for implant cases was "sequestered from the training datasets for Genius AI Detection." However, the actual sample size of the training set is not mentioned.
9. How the Ground Truth for the Training Set Was Established
- Not provided. Since the training set sample size and details are not disclosed, the method for establishing its ground truth is also not mentioned in this document. It is generally assumed that similar rigorous methods (e.g., biopsy-proven truth, expert review) would have been used for training data, but this specific filing does not detail it.
§ 892.2090 Radiological computer-assisted detection and diagnosis software.
(a)
Identification. A radiological computer-assisted detection and diagnostic software is an image processing device intended to aid in the detection, localization, and characterization of fracture, lesions, or other disease-specific findings on acquired medical images (e.g., radiography, magnetic resonance, computed tomography). The device detects, identifies, and characterizes findings based on features or information extracted from images, and provides information about the presence, location, and characteristics of the findings to the user. The analysis is intended to inform the primary diagnostic and patient management decisions that are made by the clinical user. The device is not intended as a replacement for a complete clinician's review or their clinical judgment that takes into account other relevant information from the image or patient history.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Design verification and validation must include:
(i) A detailed description of the image analysis algorithm, including a description of the algorithm inputs and outputs, each major component or block, how the algorithm and output affects or relates to clinical practice or patient care, and any algorithm limitations.
(ii) A detailed description of pre-specified performance testing protocols and dataset(s) used to assess whether the device will provide improved assisted-read detection and diagnostic performance as intended in the indicated user population(s), and to characterize the standalone device performance for labeling. Performance testing includes standalone test(s), side-by-side comparison(s), and/or a reader study, as applicable.
(iii) Results from standalone performance testing used to characterize the independent performance of the device separate from aided user performance. The performance assessment must be based on appropriate diagnostic accuracy measures (
e.g., receiver operator characteristic plot, sensitivity, specificity, positive and negative predictive values, and diagnostic likelihood ratio). Devices with localization output must include localization accuracy testing as a component of standalone testing. The test dataset must be representative of the typical patient population with enrichment made only to ensure that the test dataset contains a sufficient number of cases from important cohorts (e.g., subsets defined by clinically relevant confounders, effect modifiers, concomitant disease, and subsets defined by image acquisition characteristics) such that the performance estimates and confidence intervals of the device for these individual subsets can be characterized for the intended use population and imaging equipment.(iv) Results from performance testing that demonstrate that the device provides improved assisted-read detection and/or diagnostic performance as intended in the indicated user population(s) when used in accordance with the instructions for use. The reader population must be comprised of the intended user population in terms of clinical training, certification, and years of experience. The performance assessment must be based on appropriate diagnostic accuracy measures (
e.g., receiver operator characteristic plot, sensitivity, specificity, positive and negative predictive values, and diagnostic likelihood ratio). Test datasets must meet the requirements described in paragraph (b)(1)(iii) of this section.(v) Appropriate software documentation, including device hazard analysis, software requirements specification document, software design specification document, traceability analysis, system level test protocol, pass/fail criteria, testing results, and cybersecurity measures.
(2) Labeling must include the following:
(i) A detailed description of the patient population for which the device is indicated for use.
(ii) A detailed description of the device instructions for use, including the intended reading protocol and how the user should interpret the device output.
(iii) A detailed description of the intended user, and any user training materials or programs that address appropriate reading protocols for the device, to ensure that the end user is fully aware of how to interpret and apply the device output.
(iv) A detailed description of the device inputs and outputs.
(v) A detailed description of compatible imaging hardware and imaging protocols.
(vi) Warnings, precautions, and limitations must include situations in which the device may fail or may not operate at its expected performance level (
e.g., poor image quality or for certain subpopulations), as applicable.(vii) A detailed summary of the performance testing, including test methods, dataset characteristics, results, and a summary of sub-analyses on case distributions stratified by relevant confounders, such as anatomical characteristics, patient demographics and medical history, user experience, and imaging equipment.