(250 days)
TumorSight Viz is intended to be used in the visualization and analysis of breast magnetic resonance imaging (MRI) studies for patients with biopsy proven early-stage or locally advanced breast cancer. TumorSight Viz supports evaluation of dynamic MR data acquired from breast studies during contrast administration. TumorSight Viz performs processing functions (such as image registration, subtractions, measurements, 3D renderings, and reformats).
TumorSight Viz also includes user-configurable features for visualizing and analyzing findings in breast MRI studies. Patient management decisions should not be made based solely on the results of TumorSight Viz.
TumorSight Viz is an image processing system designed to assist in the visualization and analysis of breast DCE-MRI studies.
TumorSight reads DICOM magnetic resonance images. TumorSight processes and displays the results on the TumorSight web application.
Available features support:
- Visualization (standard image viewing tools, MIPs, and reformats)
- Analysis (registration, subtractions, kinetic curves, parametric image maps, segmentation and 3D volume rendering)
- Communication and storage (DICOM import, retrieval, and study storage)
The TumorSight system consists of proprietary software developed by SimBioSys, Inc. hosted on a cloud-based platform and accessed on an off-the-shelf computer.
Here's a summary of the acceptance criteria and study details for TumorSight Viz, based on the provided text:
1. Acceptance Criteria and Reported Device Performance
The acceptance criteria are implied by demonstrating that the device's performance (Mean Absolute Error and Dice Coefficients) is comparable to inter-radiologist variability and the predicate device, CADstream, and that "all tests met the acceptance criteria".
| Measurement Description | Units | Acceptance Criteria (Implied) | Validation Testing (Mean Abs. Error ± Std. Dev.) |
|---|---|---|---|
| Tumor Volume (n=157) | cubic centimeters (cc) | Comparable to inter-radiologist variability | 6.48 ± 12.67 |
| Tumor-to-breast volume ratio (n=157) | % | Comparable to inter-radiologist variability | 0.56 ± 0.93 |
| Tumor longest dimension (n=163) | centimeters (cm) | Comparable to inter-radiologist variability | 1.48 ± 1.46 |
| Tumor-to-nipple distance (n=161) | centimeters (cm) | Comparable to inter-radiologist variability | 1.00 ± 1.03 |
| Tumor-to-skin distance (n=163) | centimeters (cm) | Comparable to inter-radiologist variability | 0.63 ± 0.60 |
| Tumor-to-chest distance (n=163) | centimeters (cm) | Comparable to inter-radiologist variability | 0.94 ± 1.34 |
| Tumor center of mass (n=157) | centimeters (cm) | Comparable to inter-radiologist variability | 0.735 ± 1.26 |
| Performance Measurement | Metric | Acceptance Criteria (Implied) | Validation Testing (Mean ± Std. Dev.) |
|---|---|---|---|
| Tumor segmentation (n=157) | Volume Dice | Sufficient for indicating location, volume, surface agreement | 0.676 ± 0.289 |
| Surface Dice | Sufficient for indicating location, volume, surface agreement | 0.873 ± 0.264 |
Additionally, for the direct comparison with the CADstream predicate device and ground truth:
| Performance Measurement | Metric | TumorSight Viz / Ground Truth (Mean Abs. Error ± Std. Dev.) | CADStream / Ground Truth (Mean Abs. Error ± Std. Dev.) | Inter-radiologist Variability (Mean Abs. Error ± Std. Dev.) |
|---|---|---|---|---|
| Longest Dimension (n=136) | Abs. Distance Error | 1.40 cm ± 1.43 cm | 1.11 cm ± 1.52 cm | 1.17 cm ± 1.38 cm |
| Tumor to Skin (n=136) | Abs. Distance Error | 0.61 cm ± 0.46 cm | 0.49 cm ± 0.56 cm | 0.49 cm ± 0.54 cm |
| Tumor to Chest (n=136) | Abs. Distance Error | 0.77 cm ± 0.90 cm | 1.37 cm ± 1.01 cm | 0.79 cm ± 1.01 cm |
| Tumor to Nipple (n=134) | Abs. Distance Error | 0.98 cm ± 1.06 cm | 0.80 cm ± 0.86 cm | 0.82 cm ± 0.98 cm |
| Tumor Volume (n=134) | Abs. Distance Error | 6.69 cc ± 13.53 cc | 8.09 cc ± 17.42 cc | N/A (not provided for inter-radiologist variability) |
The document states: "The mean absolute error and variability between the automated measurements (Validation Testing) and ground truth for tumor volume (measured in cc) and landmark distances (measured in cm) was similar to the variability between device-to-radiologist measurements and inter-radiologist variability. This demonstrates that the error in measurements is consistent to the variability between expert readers." It also notes: "We found that all tests met the acceptance criteria, demonstrating adequate performance for our intended use." And for the comparison to the predicate: "The differences in error between the mean absolute errors (MAE) for the predicate and subject device are clinically acceptable because they are on the order of one to two voxels for the mean voxel size in the dataset. These differences are clinically insignificant."
2. Sample Size Used for the Test Set and Data Provenance
- Sample Size for Validation (Test Set): 161 patients, corresponding to 163 samples (accounting for bilateral disease).
- Data Provenance: Obtained from six (6) clinical sites in the U.S. All patients had pathologically confirmed invasive, early stage or locally advanced breast cancer. The data was collected to ensure adequate coverage of MRI manufacturer and field strength and similarity with the broader U.S. population for patient age, breast cancer subtype, T stage, histologic subtype, and race/ethnicity. This data is retrospective.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications of Those Experts
- Number of Experts: Seven (7) U.S. Board Certified radiologists.
- Qualifications of Experts: U.S. Board Certified radiologists. Specific experience level (e.g., years of experience) is not explicitly stated beyond "expert readers."
4. Adjudication Method for the Test Set
- Adjudication Method: For each case, two radiologists independently measured various characteristics. If the two radiologists did not agree on whether the candidate segmentation was appropriate, a third radiologist provided an additional opinion and established a ground truth by majority consensus (2+1 adjudication).
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was Done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance
- The document describes a performance comparison between TumorSight Viz, CADstream (predicate), and ground truth, as well as inter-radiologist variability. However, it does not describe an MRMC comparative effectiveness study directly measuring how much human readers improve with AI vs. without AI assistance. The study focuses on the standalone performance of TumorSight Viz and its comparability to a predicate device and human variability.
6. If a Standalone (i.e. algorithm only without human-in-the-loop performance) was Done
- Yes, a standalone study was done. The reported performance metrics (Mean Absolute Error, Dice Coefficients) for TumorSight Viz against a radiologist-established ground truth represent the standalone performance of the algorithm. The document explicitly states: "The measurements generated from the device result directly from the segmentation methodology and are an inferred reflection of the performance of the deep learning algorithm."
7. The type of ground truth used (expert consensus, pathology, outcomes data, etc.)
- Type of Ground Truth: Pathologically confirmed breast cancer cases (for patient inclusion) combined with expert consensus of U.S. Board Certified radiologists for specific image measurements and segmentation appropriateness. The ground truth was established by two radiologists measuring characteristics, with a third radiologist adjudicating disagreements by majority consensus.
8. The sample size for the training set
- Training Set Sample Size: 390 samples (from 736 patients mentioned for training and tuning).
- Tuning Set Sample Size: 376 samples (from 736 patients mentioned for training and tuning).
9. How the ground truth for the training set was established
- The document states that 736 patients (766 samples) were used for "training and tuning the device." It explicitly mentions that for the validation set, "Seven (7) U.S. Board Certified radiologists reviewed 163 validation samples to establish the ground truth for the dataset..."
- The method for establishing ground truth for the training set is not explicitly detailed in the provided text. It is generally implied that such ground truth would also be established by experts, but the specifics are not given for the training portion.
§ 892.2050 Medical image management and processing system.
(a)
Identification. A medical image management and processing system is a device that provides one or more capabilities relating to the review and digital processing of medical images for the purposes of interpretation by a trained practitioner of disease detection, diagnosis, or patient management. The software components may provide advanced or complex image processing functions for image manipulation, enhancement, or quantification that are intended for use in the interpretation and analysis of medical images. Advanced image manipulation functions may include image segmentation, multimodality image registration, or 3D visualization. Complex quantitative functions may include semi-automated measurements or time-series measurements.(b)
Classification. Class II (special controls; voluntary standards—Digital Imaging and Communications in Medicine (DICOM) Std., Joint Photographic Experts Group (JPEG) Std., Society of Motion Picture and Television Engineers (SMPTE) Test Pattern).