(87 days)
CINA-VCF is a radiological computer aided triage and notification software indicated for use in patients aged 50 years and over undergoing non-enhanced or contrast-enhanced CT scans which include the chest and/or abdomen.
The device is intended to assist hospital networks and appropriately trained medical specialists within the standard-of-care bone health setting in workflow triage by flagging and communication of suspected positive cases of Vertebral Compression Fractures (VCF) findings.
CINA-VCF uses an artificial intelligence algorithm to analyze images and highlight cases with detected findings on a standalone application in parallel to the ongoing standard of care image interpretation. The device does not alter the original medical image, and it is not intended to be used as a diagnostic device.
The results of CINA-VCF are intended to be used in conjunction with other patient information and based on professional judgment to assist with triage/prioritization of medical images. Notified clinicians are ultimately responsible for reviewing full images per the standard of care.
CINA-VCF is a radiological computer-assisted triage and notification software device.
CINA-VCF runs on a standard "off the shelf" server/workstation and consists of VCF Image Processing Application, which can be integrated, deployed and used with the CINA Platform (cleared under K200855) or other compatible medical image communications devices. CINA-VCF receives nonenhanced or contrast-enhanced CT scans (which include the chest and/or abdomen) identified by the CINA Platform or other compatible medical image communications device, processes them using algorithmic methods involving execution of multiple computational steps to identify suspected presence of Vertebral Compression Fractures (VCF) findings and generates results files to be transferred by CINA Platform or a similar medical image communications device for output to a PACS system or workstation for worklist prioritization.
DICOM images are received, recorded and filtered before processing. The series are processed chronologically by running algorithms on each series to detect suspected positive findings of a vertebral compressions fracture (VCF).
The device uses deep learning models to detect VCF at the T1-L5 level. The models were trained endto-end on a dataset of 886 series collected from multiple centers in the USA and France satisfying the device protocol and representing a large distribution of scanner models from Siemens, Philips, GE and Canon (formerly Toshiba), acquisition protocols, spine presentation and fracture location and severity. Additional models, trained on subsets of this dataset, are used to locate the spine, identify the vertebra bodies and exclude vertebra which have been subjected to vertebroplasty or contains orthopedic material.
The Worklist Application displays all incoming suspect cases, each notified case is marked with an icon. In addition, compressed, grayscale, unannotated images that are captioned "not for diagnostic use" is displayed as a preview function. This compressed preview is meant for informational purposes only, does not contain any marking of the findings, and is not intended for diagnostic use beyond notification.
Presenting the specialist with worklist prioritization facilitates earlier triage by allowing prioritization of images in the PACS. Thus, the suspect case receives attention earlier than would have been the case in the standard of care practice alone.
The CINA Platform is an example of medical image communications platform for integrating and deploying the CINA-VCF image processing applications. It provides the necessary requirements for interoperability based on the standardized DICOM protocol and services to communicate with existing systems in the hospital radiology department such as CT modalities or other DICOM nodes (DICOM router or PACS for example). It is responsible for transferring, converting formats, notifying of suspected findings and displaying medical device data such as radiological data. The CINA Platform server includes the Worklist client application which receives notifications from the CINA-VCF Image Processing application.
Here's a breakdown of the acceptance criteria and the study proving the device meets them, based on the provided text:
1. Table of Acceptance Criteria and Reported Device Performance
| Acceptance Criterion | Reported Device Performance (CINA-VCF) |
|---|---|
| Primary Endpoint: ROC AUC | 0.974 [95% CI: 0.962 - 0.986] (Exceeded the 0.95 performance goal) |
| Sensitivity | 95.2% [95% CI: 90.7% - 97.9%] |
| Specificity | 92.9% [95% CI: 89.4% - 96.5%] |
| Accuracy (Overall Agreement) | 93.7% [95% CI: 91.1% - 95.7%] |
| Time-to-Notification (All cases, Mean ± SD) | 23.4 ± 8.4 seconds (Median: 21.0 seconds, 95% CI: [22.7 - 24.2], Min: 9.0, Max: 60.0) |
| Time-to-Notification (True Positive cases, Mean ± SD) | 21.7 ± 7.5 seconds (Median: 20.0 seconds, 95% CI: [20.5 - 22.8], Min: 9.0, Max: 45.0) |
2. Sample Size Used for the Test Set and Data Provenance
- Sample Size: 474 clinical anonymized cases.
- Data Provenance: Retrospective, multinational study. Data provided from multiple US (66.9%) and OUS (33.1%) clinical sites. The data included 180 (37.9%) positive cases (CT with VCF) and 294 (62.1%) negative cases.
- Patient Demographics: Mean age 72.1 ± 10.1 years old (MIN = 50 yo and MAX = 100 yo), 50.8% female. Data accounted for race/ethnicity in the intended US patient population.
- Image Acquisition: Acquired by 4 different scanner makers and 38 different scanner models. Various scanner parameters were considered, including slice thickness, number of detector rows, kVp ranges, contrast vs. non-contrast, imaging protocol (chest and/or abdomen), and reconstruction kernel (soft/standard).
3. Number of Experts Used to Establish Ground Truth for the Test Set and Qualifications
- Number of Experts: Three.
- Qualifications: US-board-certified expert radiologists.
4. Adjudication Method for the Test Set
- Method: Consensus of three US-board-certified expert radiologists. A case was considered positive if at least one moderate or severe vertebral compression fracture located within the thoracic or lumbar spine was identified by the experts.
5. If a Multi-reader Multi-case (MRMC) Comparative Effectiveness Study Was Done
- No, a multi-reader multi-case (MRMC) comparative effectiveness study was not conducted to evaluate human readers with and without AI assistance for effect size. The study focused on the standalone performance of the AI device and compared its time-to-notification to a predicate device, not directly to human reader performance with or without the AI.
6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) Was Done
- Yes, a standalone performance testing was performed. The study describes "Avicenna.Al conducted a retrospective, multinational and blinded study with the CINA-VCF application... to evaluate the software's performance."
7. The Type of Ground Truth Used
- Type of Ground Truth: Expert consensus. Specifically, "ground truth established by consensus of three US-board-certified expert radiologists."
8. The Sample Size for the Training Set
- Sample Size: 886 series.
9. How the Ground Truth for the Training Set Was Established
- The device uses deep learning models that "were trained end-to-end on a dataset of 886 series collected from multiple centers in the USA and France satisfying the device protocol and representing a large distribution of scanner models... and fracture location and severity." While the text describes the dataset, it does not explicitly state how the ground truth for this training set was established. It implies that the "device protocol" guided the selection of data, likely with some form of expert labeling or pre-existing clinical reports classifying the fractures, similar to the test set ground truth, but this is not directly stated.
§ 892.2080 Radiological computer aided triage and notification software.
(a)
Identification. Radiological computer aided triage and notification software is an image processing prescription device intended to aid in prioritization and triage of radiological medical images. The device notifies a designated list of clinicians of the availability of time sensitive radiological medical images for review based on computer aided image analysis of those images performed by the device. The device does not mark, highlight, or direct users' attention to a specific location in the original image. The device does not remove cases from a reading queue. The device operates in parallel with the standard of care, which remains the default option for all cases.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Design verification and validation must include:
(i) A detailed description of the notification and triage algorithms and all underlying image analysis algorithms including, but not limited to, a detailed description of the algorithm inputs and outputs, each major component or block, how the algorithm affects or relates to clinical practice or patient care, and any algorithm limitations.
(ii) A detailed description of pre-specified performance testing protocols and dataset(s) used to assess whether the device will provide effective triage (
e.g., improved time to review of prioritized images for pre-specified clinicians).(iii) Results from performance testing that demonstrate that the device will provide effective triage. The performance assessment must be based on an appropriate measure to estimate the clinical effectiveness. The test dataset must contain sufficient numbers of cases from important cohorts (
e.g., subsets defined by clinically relevant confounders, effect modifiers, associated diseases, and subsets defined by image acquisition characteristics) such that the performance estimates and confidence intervals for these individual subsets can be characterized with the device for the intended use population and imaging equipment.(iv) Stand-alone performance testing protocols and results of the device.
(v) Appropriate software documentation (
e.g., device hazard analysis; software requirements specification document; software design specification document; traceability analysis; description of verification and validation activities including system level test protocol, pass/fail criteria, and results).(2) Labeling must include the following:
(i) A detailed description of the patient population for which the device is indicated for use;
(ii) A detailed description of the intended user and user training that addresses appropriate use protocols for the device;
(iii) Discussion of warnings, precautions, and limitations must include situations in which the device may fail or may not operate at its expected performance level (
e.g., poor image quality for certain subpopulations), as applicable;(iv) A detailed description of compatible imaging hardware, imaging protocols, and requirements for input images;
(v) Device operating instructions; and
(vi) A detailed summary of the performance testing, including: test methods, dataset characteristics, triage effectiveness (
e.g., improved time to review of prioritized images for pre-specified clinicians), diagnostic accuracy of algorithms informing triage decision, and results with associated statistical uncertainty (e.g., confidence intervals), including a summary of subanalyses on case distributions stratified by relevant confounders, such as lesion and organ characteristics, disease stages, and imaging equipment.