(184 days)
V5med Lung AI is a Computer-Aided Detection (CAD) software designed to assist radiologists in detecting pulmonary nodules (with diameter of 4-30 mm) during CT examinations of the chest for asymptomatic populations. This software provides adjunctive information to alert radiologists to regions of interest with suspected lung nodules that may otherwise be overlooked. It can be used in a concurrent read mode, where the AI analysis results are displayed alongside the original CT images during either the initial review or any subsequent reviews by the radiologist. V5med Lung AI does not replace the radiologist's critical judgment or diagnostic processes and should not be used in isolation from the original CT series.
The V5med Lung AI is a software product designed to detect nodules in the lungs. The detection model is trained using a Deep Convolutional Neural Network (CNN) based algorithm, enabling automatic detection of lung nodules ranging from 4 to 30 mm in chest CT images.
The system integrates algorithm logic and database on the same server, ensuring simplicity and ease of maintenance. It accepts chest CT images from a PACS system, Radiological Information System (RIS), or directly from a CT scanner, analyzes the images, and provides output annotations regarding lung nodules.
This document describes the regulatory acceptance criteria met by the V5med Lung AI device and the study conducted to prove its performance.
1. Table of Acceptance Criteria and Reported Device Performance
The acceptance criteria for the V5med Lung AI device are implicitly set by the endpoints measured in the clinical performance evaluation, which aimed to demonstrate improved radiologist performance with the AI tool compared to unaided reads. The reported device performance directly addresses these implicit criteria.
| Metric | Acceptance Criteria (Implicit) | Reported Device Performance (Aided vs Un-aided) | Difference (95% CI) | Result |
|---|---|---|---|---|
| AUC (Localization-Specific ROC) | Significant increase in AUC with aid. | Unaided: 0.734, Aided: 0.830 | 0.0959 (0.0586, 0.1332) | Met (Significant increase, CI entirely above 0) |
| Reading Times (seconds) | Significant decrease in reading time with aid. | Unaided: 133.0, Aided: 115.9 | -17.1 (-26.7, -9.0) | Met (Significant decrease, CI entirely below 0) |
2. Sample Size Used for the Test Set and Data Provenance
- Test Set Sample Size: 340 chest CT scans.
- Data Provenance:
- Country of Origin: Not explicitly stated, but all screening cases were acquired from the NLST (National Lung Screening Trial) CT arm, implying a US-based origin.
- Retrospective or Prospective: Retrospective. The study was a "retrospective, fully crossed, multi-reader multi-case (MRMC) study."
3. Number of Experts Used to Establish Ground Truth for the Test Set and Qualifications
The document does not explicitly state the number of experts used to establish the ground truth for the test set. It mentions sixteen board-certified radiologists participated in the reader study. These radiologists were involved in reading the cases with and without the AI aid, and their performance served as the basis for evaluating the AI's effectiveness.
4. Adjudication Method for the Test Set
The document does not explicitly describe an adjudication method for establishing the ground truth for the test set. It mentions a "fully crossed, multi-reader multi-case (MRMC) study" design, where sixteen radiologists read the cases. This implies that the performance metrics (AUC, reading times) were derived from their individual interpretations, likely compared against a pre-established consensus ground truth, though the method for that consensus is not detailed.
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study Was Done, and Effect Size
- Yes, a multi-reader multi-case (MRMC) comparative effectiveness study was done.
- Effect Size of Human Readers Improvement with AI vs. Without AI Assistance:
- AUC: Radiologists using V5med Lung AI showed a 0.0959 increase in AUC (from 0.734 unaided to 0.830 aided). The 95% confidence interval for this difference was (0.0586, 0.1332), indicating a statistically significant improvement.
- Reading Times: Radiologists using V5med Lung AI showed a 17.1-second decrease in reading time (from 133.0 seconds unaided to 115.9 seconds aided). The 95% confidence interval for this difference was (-26.7, -9.0), indicating a statistically significant reduction. This translates to approximately a 13% improvement in reading time.
6. If a Standalone (Algorithm Only Without Human-in-the-Loop Performance) Was Done
- Yes, standalone performance testing was conducted.
- The document states: "Standalone performance testing, which included chest CT scans from lung cancer screening population and non-screening population, was conducted to validate detection accuracy of V5med Lung AI."
- It further notes: "Results showed that V5med Lung AI had similar nodule detection sensitivity compared to those of the predicate device."
7. The Type of Ground Truth Used
The document does not explicitly define the type of ground truth used for the standalone performance testing or the MRMC study's assessment of radiologist performance. However, given the context of lung nodule detection in oncology, common ground truth methods include:
- Expert Consensus: Multiple expert radiologists review cases and reach a consensus on the presence and characteristics of nodules. This is a very common method for CAD device validation. The prompt indicates "consensus" as a possibility.
- Pathology: Biopsy results confirming the nature of lesions, though this is often not feasible for all identified nodules, especially in large screening datasets.
- Outcomes Data: Longitudinal follow-up of patients to see if nodules grow or are confirmed to be malignant over time.
Given the NLST data source, it is highly probable that the ground truth was established through a rigorous process, likely involving expert consensus and potentially correlation with long-term follow-up from the trial, but the specific details are not provided in this excerpt.
8. The Sample Size for the Training Set
The document does not provide the sample size for the training set. It only states that the detection model was "trained using a Deep Convolutional Neural Network (CNN) based algorithm."
9. How the Ground Truth for the Training Set Was Established
The document does not describe how the ground truth for the training set was established. It only mentions the use of a Deep Convolutional Neural Network (CNN) based algorithm for training the detection model.
§ 892.2050 Medical image management and processing system.
(a)
Identification. A medical image management and processing system is a device that provides one or more capabilities relating to the review and digital processing of medical images for the purposes of interpretation by a trained practitioner of disease detection, diagnosis, or patient management. The software components may provide advanced or complex image processing functions for image manipulation, enhancement, or quantification that are intended for use in the interpretation and analysis of medical images. Advanced image manipulation functions may include image segmentation, multimodality image registration, or 3D visualization. Complex quantitative functions may include semi-automated measurements or time-series measurements.(b)
Classification. Class II (special controls; voluntary standards—Digital Imaging and Communications in Medicine (DICOM) Std., Joint Photographic Experts Group (JPEG) Std., Society of Motion Picture and Television Engineers (SMPTE) Test Pattern).