(103 days)
QuantX is a computer-aided diagnosis (CADx) software device used to assist radiologists in the assessment and characterization of breast abnormalities using MR image data. The software automatically registers images, and segments and analyzes user-selected regions of interest (ROI). QuantX extracts image data from the ROI to provide volumetric analysis and computer analytics based on morphological and enhancement characteristics. These imaging (or radiomic) features are then synthesized by an artificial intelligence algorithm into a single value, the QI score, which is analyzed relative to a database of reference abnormalities with known ground truth.
QuantX is indicated for evaluation of patients presenting for high-risk screening, diagnostic imaging workup, or evaluation of extent of known disease. Extent of known disease refers to both the assessment of the boundary of a particular abnormality as well as the assessment of the total disease burden in a particular patient. In cases where multiple abnormalities are present, QuantX can be used to assess each abnormality independently.
This device provides information that may be useful in the characterization of breast abnormalities during image interpretation. For the QI score and component radiomic features, the QuantX device provides comparative analysis to lesions with known outcomes using an image atlas and histogram display format.
QuantX may also be used as an image viewer of multi-modality digital images, including ultrasound and mammography. The software also includes tools that allow users to measure and document images, and output in a structured report.
Limitations: QuantX is not intended for primary interpretation of digital mammography images.
The device is a software-only post-processing system for patient breast images that includes analysis of MR images, and viewing ultrasound and mammographic images.
MR images are acquired from a third-party acquisition device. The images can be loaded into the QuantX device manually or automatically if connected to a DICOMcompatible device. Users select and load the patient case to use the QuantX software tools in the examination of the images. Different types of MR sequences (T1, DCE, T2, DWI, etc.) can be viewed at the same time as mammography or ultrasound images from the same patient.
QuantX includes image registration, and automated segmentation and analysis functions, based on a seed point indicated by the user. Users can select a ROI manually from the MR image, or use the automatic segmentation tool to obtain and accept a ROI. for input to the QuantX analytics. The QuantX analytics display the QI Most Enhancing Curve, the Average Enhancing Curve, and volume of the specified region.
QuantX provides users the QI Score, based on the morphological and enhancement characteristics of the region of interest. The QuantX package provides comparative analysis for the QI score and its component element features to lesions with known ground truth (either biopsy- proven diagnosis or minimum one year follow-up negative scan for non-biopsied lesions) using an image atlas and histogram display format.
A user experienced with the significance of such data will be able to view and interpret this additional information during the diagnosis of breast lesions.
Users may select from a variety of information sources to make the diagnosis. The key features of the device are related categorization of lesions include the display of similar cases and the histogram of known lesions for various analytic features (included the Ql score). The QI Score is not a "probability of malignancy," but is intended for the organization of an online atlas (reference database) provided to the user as the Similar Case Database. The QI score is based on a machine learning algorithm, trained on a subset of features calculated on a segmented lesions.
Here's a summary of the acceptance criteria and the study that proves the device meets them, based on the provided text:
Acceptance Criteria and Device Performance for QuantX
1. Table of Acceptance Criteria and Reported Device Performance
The primary acceptance criteria for QuantX, as a Class II device with Special Controls, revolve around its ability to improve reader performance in diagnosing breast cancer when used as an aid compared to without it.
| Acceptance Criteria (from Special Control 1.iii) | Reported Device Performance (from MRMC Study Results) |
|---|---|
| Improvement in Reader Performance: Demonstruate that the device improves reader performance in the intended use population when used in accordance with the instructions for use, based on appropriate diagnostic accuracy measures (e.g., receiver operator characteristic plot, sensitivity, specificity, predictive value, and diagnostic likelihood ratio). | Primary Endpoint (Improvement in AUC): |
| Proper-binormal method: |
- AUC (FIRST READ, without QuantX): 0.7055
- AUC (SECOND READ, with QuantX): 0.7575
- ΔAUC (SECOND READ - FIRST READ): 0.0520 (95% CI: [0.0022, 0.1018], p-value: 0.0408)
Trapezoidal method: - AUC (FIRST READ, without QuantX): 0.7090
- AUC (SECOND READ, with QuantX): 0.7577
- ΔAUC (SECOND READ - FIRST READ): 0.0487 (95% CI: [-0.0011, 0.0985], p-value: 0.0550)
Conclusion: The study "marginally met the primary endpoint" with the proper-binormal method demonstrating a statistically significant improvement in AUC. |
| No Unintended Reduction in Sensitivity or Specificity (Secondary Analysis): Ensure there is not an unintended reduction in either sensitivity or specificity. | Secondary Analyses (Sensitivity & Specificity, descriptive):
BI-RADS cut-point of 3 (≥3 indicates positive): - Sensitivity Difference: 3.8% (95% CI: [0.8, 7.4])
- Specificity Difference: -1.0% (95% CI: [-6.5, 4.3])
BI-RADS cut-point of 4a (≥4a indicates positive): - Sensitivity Difference: 5.1% (95% CI: [-0.9, 10.9])
- Specificity Difference: -0.5% (95% CI: [-7.3, 6.0])
Conclusion: Secondary analyses "suggest improved sensitivity based on BI-RADS 3 as the cut-point without decreased specificity for some readers" (although not pre-specified for formal hypothesis testing). |
| Standalone Performance: Standalone performance testing protocols and results of the device. | Standalone Performance: - On Similar Case Database (i.e., training data): AUC = 0.86 ± 0.02 (mean ± standard error). (Note: This is not considered independent validation, but contains cases from important cohorts.)
- On Reader Study Test Database (automated segmentation): AUC = 0.75 ± 0.05 (mean ± standard error).
- On Reader Study Test Database (clinical reader study segmentation variability): AUC = 0.71 ± 0.05 (mean ± standard error). |
2. Sample Size Used for the Test Set and Data Provenance
- Test Set Sample Size: The reader study test database included 111 breast MRI cases, with one lesion per case used for ROC analysis. Of these, 54 were cancerous and 57 were non-cancerous.
- Data Provenance: The data was retrospectively collected from three different institutions: an academic breast imaging center, a dedicated cancer imaging center, and a community-based imaging center. The cases were collected from the US (e.g., University of Chicago Medical Center, Memorial Sloan Kettering Cancer Center, and X-Ray Associates of New Mexico).
- Date Range of Data Collection:
- Philips 1.5T: 02/2009 - 01/2014
- Philips 3T: 05/2010 - 12/2013
- Siemens 1.5T: 10/2013 - 12/2014
- GE 1.5T: 03/2011
- GE 3T: 01/2009 - 09/2013
- Date Range of Data Collection:
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications
- Number of Experts: The ground truth for the non-biopsied non-cancers was determined by multidisciplinary review. The text does not specify the exact number of experts involved in this review for the test set, but it states that "lesion type had been determined by multidisciplinary review" for cases in the Similar Case Database (which shared case inclusion criteria with the standalone testing test set), and for the reader study test set, non-biopsied non-cancers required "clinical and radiology reports and a negative follow-up MRI study at a minimum of 12 months."
- Qualifications of Those Experts: For biopsied cases, ground truth was directly from pathology reports, implying pathologists were the experts. For non-biopsied non-cancers, the ground truth was established by "clinical and radiology reports and a negative follow-up MRI study," suggesting radiologists and clinicians were involved. The description of reader qualifications for the MRMC study (radiologists with at least 1 year of breast MRI interpretation experience, fellowship-trained in breast imaging or 2 years' breast imaging experience, MQSA qualified) gives an indication of the expertise typically expected in such a setting.
4. Adjudication Method for the Test Set
The ground truth was established based on biopsy-proven diagnosis (pathology reports) for cancers and biopsied non-cancers. For non-biopsied benign lesions, it was based on clinical and radiology reports and a negative follow-up MRI study at a minimum of 12 months, along with multidisciplinary review. This implies a form of consensus or adjudicated ground truth for the non-biopsied benign cases. The document mentions concordance was defined as a tissue biopsy result being compatible with the abnormal pre-biopsy imaging appearance, indicating an adjudicating process for these specific cases.
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study Was Done, and Effect Size
- Yes, an MRMC comparative effectiveness study was done.
- Effect Size of Human Readers' Improvement with AI vs. Without AI Assistance:
- Using the proper-binormal method (primary analysis): The average AUC across all readers improved by 0.0520 (from 0.7055 to 0.7575) when using QuantX.
- Using the trapezoidal method (secondary analysis): The average AUC across all readers improved by 0.0487 (from 0.7090 to 0.7577).
6. If a Standalone (Algorithm Only Without Human-in-the-Loop Performance) Was Done
- Yes, standalone performance testing was done.
- Results:
- On the Similar Case Database (training data): AUC = 0.86 ± 0.02.
- On the Reader Study Test Database (with automated segmentation, no manual correction): AUC = 0.75 ± 0.05.
- On the Reader Study Test Database (with segmentation allowing for clinical reader study variability in seed point locations): AUC = 0.71 ± 0.05.
7. The Type of Ground Truth Used (Expert Consensus, Pathology, Outcomes Data, Etc.)
The ground truth for both the training and test sets was a combination of:
- Pathology: Biopsy-proven diagnosis for cancerous lesions and biopsied non-cancerous lesions.
- Outcomes Data/Follow-up: Negative follow-up MRI study at a minimum of 12 months for non-biopsied benign lesions.
- Expert Consensus/Multidisciplinary Review: For non-biopsied benign lesions, "clinical and radiology reports" were considered, and for cases in the Similar Case Database, lesion type "had been determined by multidisciplinary review."
8. The Sample Size for the Training Set
The text refers to the "Similar Case Database" as the training database (or the data on which the QI Score's machine learning algorithm was trained).
- Training Set Sample Size: This database included a total of 652 lesions (314 benign and 338 malignant), collected across different MR system manufacturers and field strengths between 2008 and 2014. More detailed breakdown: Philips (429 lesions), GE (48 lesions), Siemens (66 lesions).
9. How the Ground Truth for the Training Set Was Established
The ground truth for the training set (Similar Case Database) was established using the same criteria as the test set:
- Biopsy-proven truth: For lesions with pathology.
- Clinical and radiology reports + negative follow-up: For non-biopsied benign lesions (minimum 12 months).
- Multidisciplinary review: For all cases, ensuring lesion type had been determined by a multidisciplinary review.
§ 892.2060 Radiological computer-assisted diagnostic software for lesions suspicious of cancer.
(a)
Identification. A radiological computer-assisted diagnostic software for lesions suspicious of cancer is an image processing prescription device intended to aid in the characterization of lesions as suspicious for cancer identified on acquired medical images such as magnetic resonance, mammography, radiography, or computed tomography. The device characterizes lesions based on features or information extracted from the images and provides information about the lesion(s) to the user. Diagnostic and patient management decisions are made by the clinical user.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Design verification and validation must include:
(i) A detailed description of the image analysis algorithms including, but not limited to, a detailed description of the algorithm inputs and outputs, each major component or block, and algorithm limitations.
(ii) A detailed description of pre-specified performance testing protocols and dataset(s) used to assess whether the device will improve reader performance as intended.
(iii) Results from performance testing protocols that demonstrate that the device improves reader performance in the intended use population when used in accordance with the instructions for use. The performance assessment must be based on appropriate diagnostic accuracy measures (
e.g., receiver operator characteristic plot, sensitivity, specificity, predictive value, and diagnostic likelihood ratio). The test dataset must contain sufficient numbers of cases from important cohorts (e.g., subsets defined by clinically relevant confounders, effect modifiers, concomitant diseases, and subsets defined by image acquisition characteristics) such that the performance estimates and confidence intervals of the device for these individual subsets can be characterized for the intended use population and imaging equipment.(iv) Standalone performance testing protocols and results of the device.
(v) Appropriate software documentation (
e.g., device hazard analysis; software requirements specification document; software design specification document; traceability analysis; and description of verification and validation activities including system level test protocol, pass/fail criteria, results, and cybersecurity).(2) Labeling must include:
(i) A detailed description of the patient population for which the device is indicated for use.
(ii) A detailed description of the intended reading protocol.
(iii) A detailed description of the intended user and recommended user training.
(iv) A detailed description of the device inputs and outputs.
(v) A detailed description of compatible imaging hardware and imaging protocols.
(vi) Warnings, precautions, and limitations, including situations in which the device may fail or may not operate at its expected performance level (
e.g., poor image quality or for certain subpopulations), as applicable.(vii) Detailed instructions for use.
(viii) A detailed summary of the performance testing, including: Test methods, dataset characteristics, results, and a summary of sub-analyses on case distributions stratified by relevant confounders (
e.g., lesion and organ characteristics, disease stages, and imaging equipment).