(266 days)
The iCAS -LV is intended to receive multi-phase volume datasets of reconstructed studies from PACS devices, to process them and to transfer the processing output to the PACS in DICOM format. It is a PC-based, self-contained, noninvasive image analysis software application. The device provides tools for visualization, measurements, segmentation, annotation, images registration, processing, and reporting.
The device is intended for use by trained physicians. Further, the iCAS-LV is indicated to support the physicians in visualization of CT reconstructed images and evaluation of physician-identified liver lesions. The combination of the visualization, interactive segmentation, measurements, automatic registration, and volumetric analysis, supports the physician in evaluation of the lesions in terms of size, shape, position and changes over time. The iCAS should not be used in isolation for diagnosis and making patient management decisions.
The iCAS-LV (iCAS hereafter) provides tools for interactive segmentation of radiologist-identified liver lesions, automatic lesion length (diameter) and lesion volume computation, supervised automatic liver registration of the prior and current contrast-enhanced CT (ceCT scans, henceforth CT scans), semiautomatic lesions matching between two scans, and automatic lesion length (diameter) and volume change over time computation. These software tools will enable the user to easily assess the individual lesions volume and length (diameter), the total lesions burden volume and their evolution over time. This may help save radiologists and clinicians significant time and effort and improve the comprehensiveness and reliability of their reporting. The processing of the CT scans by iCAS does not rely on nor use any CT scanner-specific data. The device is compatible with CT scanners of vendors and models that conform to DICOM requirements as specified in the device Labeling.
The output is a quantitative analysis of the liver lesions volumes and their locations in the prior and current CT scans and a quantitative analysis of the volumetric changes of these lesions over time. The pipeline process consists then of two validation/manual steps and four automatic steps as follows:
-
- Generate liver ROI of both Prior and Current scans.
-
- Registration of liver ROls using deformable registration.
-
- Lesion segmentation using 3D U-Net models. Segmentations are not displayed to the user until lesions are identified in Step 4
-
- Designation of the liver lesions by the radiologist in both prior and current scans, with the selection and validation of their computed segmentations, including, when needed, manual correction of these computed lesion segmentations.
-
- Semi-automatic lesion matching of the identified lesions in the Prior and Current CT scans, including Labeling the lesions as existing, new or disappeared.
-
- Lesions and lesions change quantification (volume and diameter), which can be extended to a complete longitudinal CT study analysis. The technique computes various quantitative lesion change measures and identifies key slices in each scan.
The provided text describes the acceptance criteria and the study that proves the device meets the acceptance criteria for the iCAS-LV device.
Acceptance Criteria and Device Performance Study for iCAS-LV
The iCAS-LV device is an image analysis software application intended to support physicians in the visualization and evaluation of physician-identified liver lesions based on CT reconstructed images. The studies presented demonstrate the device's performance in segmenting and quantifying liver lesions.
1. Table of Acceptance Criteria and Reported Device Performance
The document does not explicitly state quantitative acceptance criteria in a pass/fail format with specific thresholds. However, it describes performance testing conclusions, which can be interpreted as demonstrating meeting implicit criteria for agreement with expert assessments for clinical utility.
| Metric/Testing Type | Acceptance Criteria (Implicit) | Reported Device Performance |
|---|---|---|
| Phantom Testing | Algorithm estimates for sphere volume and diameter should perform well against ground truth and expert readers. | - Volume (cc) Estimation: The algorithm performed well against the ground truth for volume estimation and in relation to expert readers. |
- Sphere Diameter (mm) Estimation: Algorithm estimates were slightly better than reader estimates in most cases, with readers similarly overestimating diameter.
- RECIST (mm) Assessments: Also collected as an additional assessment.
- Changes Across Locations: Analysis on pairs of phantoms indicated the scanner was in line with reader performance for changes in location, volumes, and invariance due to positioning.
- Multi-Scanner Performance: Performance confirmed across four scanner manufacturers (GE, Philips, Siemens, Canon) with "small differences," supporting effectiveness for both volume and diameter estimation on multiple platforms. |
| Standalone Performance Testing (Clinical Data) | Physician-assisted iCAS-LV assessments of lesion 3D volumetric data should agree with radiologist manual assessments. | - The analysis supports the utility of the iCAS algorithm based on DICE, ASSD, SHD, volume, and RECIST measurements. - The comparison results demonstrated that when used by a trained radiologist, the iCAS-LV assessment of lesions 3D volumetric data agrees with the radiologist's manual assessment. |
| Software V&V | Software should be validated for its intended use per design documentation and fulfill relevant requirements. | Demonstrated passing results on all applicable unit, integration, and requirements testing. |
2. Sample Size Used for the Test Set and Data Provenance
- Sample Size for Test Set:
- Patients: 108 patients
- CT Scans: 219 contrast-enhanced CT (ceCT) scans
- Liver Lesions: 2,127 liver lesions in total (1,942 with diameter >5mm, 1,130 with diameter >10mm).
- Pairs of Lesions/Scans (for longitudinal changes): Mean number of 4.4±5.0 pairs of lesions per pairs of scans.
- Data Provenance: Retrospective, multi-site data.
- 54 patients from Israel and Italy.
- 54 patients from the US.
- Scanner Manufacturers Represented: GE Medical Systems, Philips, Siemens, Toshiba, Hitachi, and some classified as "unknown."
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications of those Experts
- Number of Experts: Three experienced radiologists.
- Qualifications:
- Two experienced radiologists: Independently identified and delineated liver metastases.
- One of these two is US board-certified.
- A third senior radiologist: Reviewed and compared the findings of the first two.
4. Adjudication Method for the Test Set
The ground truthing process involved a form of adjudication:
- Two experienced radiologists independently identified and delineated lesions.
- A third senior radiologist reviewed and compared their findings.
- The final lesion delineations were validated or modified by the third radiologist, establishing the Ground Truth. This can be interpreted as a form of expert consensus with a senior reviewer for final decision.
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was Done
- The provided text does not explicitly describe a conventional MRMC comparative effectiveness study designed to measure the effect size of how much human readers improve with AI vs. without AI assistance.
- The clinical data analysis primarily focused on the agreement of physician-assisted iCAS software estimates with ground truth established by radiologists, rather than a direct comparison of human readers' performance with and without the device. The study "compared lesion volume and length (diameter), as well as changes in lesion volume and length over time, using estimates generated by physician-assisted iCAS software, in comparison to ground truth determined by three radiologists." This is a performance study of the device-assisted workflow, not a human-reader comparative study.
6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) was Done
- Yes, a "Standalone Performance Testing" section is explicitly mentioned.
- The description of this testing states: "HighRAD conducted a clinical data analysis comparing lesion volume and length (diameter), as well as changes in lesion volume and length over time, using estimates generated by physician-assisted iCAS software, in comparison to ground truth determined by three radiologists."
- While the term "physician-assisted" is used in its description, the context of "Standalone Performance Testing" and the metrics (DICE, ASSD, SHD) typically associated with segmentation accuracy of an algorithm against a ground truth, suggest an evaluation of the algorithm's output which the physician would then validate/correct (as described in the workflow, step 4: "selection and validation of their computed segmentations, including, when needed, manual correction"). The "agreement" with manual assessment implies the automated part of the system is the focus of this standalone evaluation. The initial segmentation (step 3 of workflow, "Lesion segmentation using 3D U-Net models") happens before physician designation and validation (step 4). The "Standalone" section seems to refer to the performance of these automated steps which form the basis for the physician's validation/correction.
7. The Type of Ground Truth Used
- Expert Consensus / Expert Delineation: The ground truth for the clinical data analysis was established by the consensus of three experienced radiologists, with a senior radiologist validating or modifying the final lesion delineations.
- Phantom Testing: For phantom studies, a "ground truth" was used against which the algorithm's and readers' estimates were compared; for physical phantoms, this typically refers to the known physical dimensions and volumes of the spheres within the phantom.
8. The Sample Size for the Training Set
The document does not specify the sample size for the training set. It only describes the dataset used for "testing" or "validation" of the deep learning algorithm.
9. How the Ground Truth for the Training Set was Established
The document does not provide information on how the ground truth was established for the training set, as it does not describe the training set itself.
§ 892.2050 Medical image management and processing system.
(a)
Identification. A medical image management and processing system is a device that provides one or more capabilities relating to the review and digital processing of medical images for the purposes of interpretation by a trained practitioner of disease detection, diagnosis, or patient management. The software components may provide advanced or complex image processing functions for image manipulation, enhancement, or quantification that are intended for use in the interpretation and analysis of medical images. Advanced image manipulation functions may include image segmentation, multimodality image registration, or 3D visualization. Complex quantitative functions may include semi-automated measurements or time-series measurements.(b)
Classification. Class II (special controls; voluntary standards—Digital Imaging and Communications in Medicine (DICOM) Std., Joint Photographic Experts Group (JPEG) Std., Society of Motion Picture and Television Engineers (SMPTE) Test Pattern).