(205 days)
Not Found
Yes
The summary explicitly states that the device "uses an artificial intelligence algorithm" and "utilizes Albased image analysis algorithms," and also mentions "Deep Convolutional Neural Networks (DCNN)," which are a type of machine learning.
No
Explanation: This device is a diagnostic tool that analyzes medical images for critical findings and helps prioritize worklists. It does not directly provide therapy or treatment for any medical condition.
No
The device is described as a "prioritization-only software tool" and explicitly states that its results "are not intended to be used on a stand-alone basis for clinical decision-making". It aids in worklist prioritization by identifying suspected critical findings but does not provide a definitive diagnosis.
Yes
The device is described as "radiological computer-assisted triage and notification software" and "radiological computer-assisted prioritization software". It analyzes images and provides output within the PACS system. There is no mention of any accompanying hardware components or hardware-specific validation.
Based on the provided information, RADIFY® Triage is not an In Vitro Diagnostic (IVD) device.
Here's why:
- IVD Definition: In Vitro Diagnostic devices are used to examine specimens taken from the human body, such as blood, urine, or tissue, to provide information about a person's health.
- RADIFY® Triage Function: RADIFY® Triage analyzes medical images (chest X-rays), which are not specimens taken from the body. It processes existing radiological data.
- Intended Use: The intended use is for radiological computer-assisted triage and notification software for analyzing chest X-ray images. This is clearly focused on image analysis and workflow prioritization within a radiology setting, not on analyzing biological samples.
Therefore, RADIFY® Triage falls under the category of medical imaging software or a radiological device, not an In Vitro Diagnostic device.
No
The letter does not state that the FDA has reviewed and approved or cleared a Predetermined Change Control Plan (PCCP) for this specific device.
Intended Use / Indications for Use
RADIFY® Triage is a radiological computer-assisted triage and notification software that analyzes adult chest X-ray images for the presence of pre-specified suspected critical findings (pleural effusion and/or pneumothorax).
RADIFY® Triage uses an artificial intelligence algorithm to analyze images for features suggestive of critical findings and provides case-level output available in the PACS for worklist prioritization or triage.
As a passive notification for prioritization-only software tool within the standard of care workflow, RADIFY® Triage does not send a proactive alert directly to the appropriately trained medical specialists. The product is not intended to direct attention to specific portions of an image. Its results are not intended to be used on a stand-alone basis for clinical decision-making. The device does not remove the cases from the queue and does not flag the condition as being absent.
Product codes
QFM
Device Description
RADIFY® Triage is a radiological computer-assisted prioritization software that utilizes Albased image analysis algorithms to identify pre-specified critical findings (pleural effusion and/or pneumothorax) on frontal (AP and PA) views chest X-ray images and flag the images in the PACS to enable worklist prioritization by the appropriately trained medical specialists who are qualified to interpret chest radiographs. The software does not alter the order or remove cases from the reading queue.
The algorithm was trained on datasets from US and non-USA sources. This training dataset consisted of 93.7% of the data from South Africa, and 6.3% of the data from the USA. The input for RADIFY® Triage is a frontal chest x-ray (AP and PA view) in digital imaging and communications in medicine (DICOM) format.
Chest X-rays are sent to RADIFY® Triage via PACS (Picture Archiving and Communication System (PACS) and processed by the device for analysis. Following receipt of chest x-rays, the software device automatically analyses each image to detect features suggestive of pneumothorax and/or pleural effusion. Chest x-rays without the suspicious findings are placed in the worklist for routine review, which is the standard of care. RADIFY® Triage does not provide any proactive alerts and is not intended to direct attention to specific portions of the image. The results are not intended to be used on a standalone basis for clinical decision-making nor is it intended to rule out the target conditions or otherwise preclude clinical assessment of x-ray cases.
Mentions image processing
Yes
Mentions AI, DNN, or ML
Yes
Input Imaging Modality
Chest X-ray
Anatomical Site
Chest/Lung
Indicated Patient Age Range
Adult
Intended User / Care Setting
Appropriately trained medical specialists who are qualified to interpret chest radiographs.
Description of the training set, sample size, data source, and annotation protocol
The algorithm was trained on datasets from US and non-USA sources. This training dataset consisted of 93.7% of the data from South Africa, and 6.3% of the data from the USA.
Description of the test set, sample size, data source, and annotation protocol
The study dataset included a total of 1229 scans (pneumothorax) from various parts of the US.
The study dataset included a total 1229 scans (pleural effusion: 392 with pleural effusion and 837 without pleural effusion) from a large urban hospital in New York City and 3 different private clinics in urban and suburban areas in Texas state. A data collection protocol was set in place to ensure that there were sufficient numbers of important subgroups.
The ground truth was established by 3 board-certified ABR (USA) radiologists with a minimum of 11 years of experience.
Summary of Performance Studies (study type, sample size, AUC, MRMC, standalone performance, key results)
Clinical studies were conducted on retrospectively collected Chest X-rays to evaluate the performance of RADIFY® Triage for triaging of pneumothorax and pleural effusion. There was total independence between the training and test dataset with a new study of unseen images from an independent dataset from 2 sites in the US being used to present the performance testing results.
Pneumothorax:
Sample size: 1229 scans with pneumothorax and 959 scans without pneumothorax.
AUC: 97.43 (95% Cl: [97.12, 97.74])
Sensitivity: 94.81% (95% Cl: [93.90, 95.73])
Specificity: 97.91% (95% Cl: [97.00, 98.83])
Key results: The AUC of the device in triaging scans with findings suspicious of pneumothorax exceeded the acceptance criteria. The device's generalizability was ensured by performing subgroup analyses, finding consistent results in both genders and across a wide range of ages (22-85+ years), and across image acquisition devices. Performance accuracy of detecting Pneumothorax was maintained at AUC of 96.36% (95% Cl [95.78, 96.93]) in cases with co-existing pleural effusion.
Pleural Effusion:
Sample size: 392 scans with pleural effusion and 837 scans without pleural effusion.
AUC: 97.61 (95% Cl: [97.36, 97.86])
Sensitivity: 94.39% (95% Cl: [93.26, 95.51])
Specificity: 96.42% (95% CI: [95.29, 98.00])
Key results: The AUC of the device in triaging scans with findings suspicious of pleural effusion exceeded the acceptance criteria. The device's generalizability for triaging scans with pleural effusion was ensured by performing subgroup analyses, finding consistent results in both genders and across a wide range of ages (22-85+ years), and across image acquisition devices. Performance accuracy of detecting Pleural effusion was maintained at AUC of 98.20% (95% Cl [97.82, 98.58]) in cases with co-existing pneumothorax.
The device's performance time averaged at 3s.
Key Metrics (Sensitivity, Specificity, PPV, NPV, etc.)
Pneumothorax:
Sensitivity: 94.81% (95% CI (93.90, 95.73))
Specificity: 97.91% (95% CI (97.00, 98.83))
AUC: 97.43% (95% CI (97.12, 97.74))
Pleural Effusion:
Sensitivity: 94.39% (95% CI (93.26, 95.5))
Specificity: 97.642% (95% CI (95.29, 97.54))
AUC: 97.61% (95% CI (97.36, 97.86))
All:
Sensitivity: 94.26% (95% CI (93.53, 94.99))
Specificity: 97.27% (95% CI (96.54, 98.00))
AUC: 97.62% (95% CI (97.43, 97.81))
Predicate Device(s):
Reference Device(s):
Not Found
Predetermined Change Control Plan (PCCP) - All Relevant Information
Not Found
§ 892.2080 Radiological computer aided triage and notification software.
(a)
Identification. Radiological computer aided triage and notification software is an image processing prescription device intended to aid in prioritization and triage of radiological medical images. The device notifies a designated list of clinicians of the availability of time sensitive radiological medical images for review based on computer aided image analysis of those images performed by the device. The device does not mark, highlight, or direct users' attention to a specific location in the original image. The device does not remove cases from a reading queue. The device operates in parallel with the standard of care, which remains the default option for all cases.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Design verification and validation must include:
(i) A detailed description of the notification and triage algorithms and all underlying image analysis algorithms including, but not limited to, a detailed description of the algorithm inputs and outputs, each major component or block, how the algorithm affects or relates to clinical practice or patient care, and any algorithm limitations.
(ii) A detailed description of pre-specified performance testing protocols and dataset(s) used to assess whether the device will provide effective triage (
e.g., improved time to review of prioritized images for pre-specified clinicians).(iii) Results from performance testing that demonstrate that the device will provide effective triage. The performance assessment must be based on an appropriate measure to estimate the clinical effectiveness. The test dataset must contain sufficient numbers of cases from important cohorts (
e.g., subsets defined by clinically relevant confounders, effect modifiers, associated diseases, and subsets defined by image acquisition characteristics) such that the performance estimates and confidence intervals for these individual subsets can be characterized with the device for the intended use population and imaging equipment.(iv) Stand-alone performance testing protocols and results of the device.
(v) Appropriate software documentation (
e.g., device hazard analysis; software requirements specification document; software design specification document; traceability analysis; description of verification and validation activities including system level test protocol, pass/fail criteria, and results).(2) Labeling must include the following:
(i) A detailed description of the patient population for which the device is indicated for use;
(ii) A detailed description of the intended user and user training that addresses appropriate use protocols for the device;
(iii) Discussion of warnings, precautions, and limitations must include situations in which the device may fail or may not operate at its expected performance level (
e.g., poor image quality for certain subpopulations), as applicable;(iv) A detailed description of compatible imaging hardware, imaging protocols, and requirements for input images;
(v) Device operating instructions; and
(vi) A detailed summary of the performance testing, including: test methods, dataset characteristics, triage effectiveness (
e.g., improved time to review of prioritized images for pre-specified clinicians), diagnostic accuracy of algorithms informing triage decision, and results with associated statistical uncertainty (e.g., confidence intervals), including a summary of subanalyses on case distributions stratified by relevant confounders, such as lesion and organ characteristics, disease stages, and imaging equipment.
0
Image /page/0/Picture/0 description: The image shows the logo of the U.S. Food and Drug Administration (FDA). On the left is the Department of Health & Human Services logo. To the right of that is the FDA logo, which is a blue square with the letters "FDA" in white. To the right of the blue square is the text "U.S. FOOD & DRUG ADMINISTRATION" in blue.
Envisionit DeepAI Ltd % Michael Pogose Director of Quality Assurance and Regulatory Affairs Hardian Ltd t/a Hardian Health c/o Galloways, 3rd Floor 21 Perrymount Road Haywards Heath, West Sussex RH16 3TP United Kingdom
Re: K231871
January 17, 2024
Trade/Device Name: Radify® Triage Regulation Number: 21 CFR 892.2080 Regulation Name: Radiological computer aided triage and notification software Regulatory Class: Class II Product Code: QFM Dated: December 19, 2023 Received: December 19, 2023
Dear Michael Pogose:
We have reviewed your section 510(k) premarket notification of intent to market the device referenced above and have determined the device is substantially equivalent (for the indications for use stated in the enclosure) to legally marketed predicate devices marketed in interstate commerce prior to May 28, 1976, the enactment date of the Medical Device Amendments, or to devices that have been reclassified in accordance with the provisions of the Federal Food, Drug, and Cosmetic Act (the Act) that do not require approval of a premarket approval application (PMA). You may, therefore, market the device, subject to the general controls provisions of the Act. Although this letter refers to your product as a device, please be aware that some cleared products may instead be combination products. The 510(k) Premarket Notification Database available at https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm identifies combination product submissions. The general controls provisions of the Act include requirements for annual registration, listing of devices, good manufacturing practice, labeling, and prohibitions against misbranding and adulteration. Please note: CDRH does not evaluate information related to contract liability warranties. We remind you, however, that device labeling must be truthful and not misleading.
If your device is classified (see above) into either class II (Special Controls) or class III (PMA), it may be subject to additional controls. Existing major regulations affecting your device can be found in the Code of Federal Regulations, Title 21, Parts 800 to 898. In addition, FDA may publish further announcements concerning your device in the Federal Register.
Additional information about changes that may require a new premarket notification are provided in the FDA guidance documents entitled "Deciding When to Submit a 510(k) for a Change to an Existing Device" (https://www.fda.gov/media/99812/download) and "Deciding When to Submit a 510(k) for a Software Change to an Existing Device" (https://www.fda.gov/media/99785/download).
1
Your device is also subject to, among other requirements, the Quality System (QS) regulation (21 CFR Part 820), which includes, but is not limited to, 21 CFR 820.30, Design controls; 21 CFR 820.90, Nonconforming product; and 21 CFR 820.100, Corrective and preventive action. Please note that regardless of whether a change requires premarket review, the QS regulation requires device manufacturers to review and approve changes to device design and production (21 CFR 820.30 and 21 CFR 820.70) and document changes and approvals in the device master record (21 CFR 820.181).
Please be advised that FDA's issuance of a substantial equivalence determination does not mean that FDA has made a determination that your device complies with other requirements of the Act or any Federal statutes and regulations administered by other Federal agencies. You must comply with all the Act's requirements, including, but not limited to: registration and listing (21 CFR Part 807); labeling (21 CFR Part 801); medical device reporting of medical device-related adverse events) (21 CFR Part 803) for devices or postmarketing safety reporting (21 CFR Part 4, Subpart B) for combination products (see https://www.fda.gov/combination-products/guidance-regulatory-information/postmarketing-safety-reportingcombination-products); good manufacturing practice requirements as set forth in the quality systems (QS) regulation (21 CFR Part 820) for devices or current good manufacturing practices (21 CFR Part 4, Subpart A) for combination products; and, if applicable, the electronic product radiation control provisions (Sections 531-542 of the Act); 21 CFR Parts 1000-1050.
Also, please note the regulation entitled, "Misbranding by reference to premarket notification" (21 CFR 807.97). For questions regarding the reporting of adverse events under the MDR regulation (21 CFR Part 803), please go to https://www.fda.gov/medical-device-safety/medical-device-reportingmdr-how-report-medical-device-problems.
For comprehensive regulatory information about medical devices and radiation-emitting products, including information about labeling regulations, please see Device Advice (https://www.fda.gov/medicaldevices/device-advice-comprehensive-regulatory-assistance) and CDRH Learn (https://www.fda.gov/training-and-continuing-education/cdrh-learn). Additionally, you may contact the Division of Industry and Consumer Education (DICE) to ask a question about a specific regulatory topic. See the DICE website (https://www.fda.gov/medical-device-advice-comprehensive-regulatoryassistance/contact-us-division-industry-and-consumer-education-dice) for more information or contact DICE by email (DICE@fda.hhs.gov) or phone (1-800-638-2041 or 301-796-7100).
Sincerely,
Jessica Lamb
Jessica Lamb. Ph.D. Assistant Director Imaging Software Team DHT8B: Division of Radiological Imaging Devices and Electronic Products OHT8: Office of Radiological Health Office of Product Evaluation and Quality Center for Devices and Radiological Health
2
Indications for Use
Submission Number (if known)
Device Name
RADIFY® Triage
Indications for Use (Describe)
RADIFY® Triage is a radiological computer-assisted triage and notification software that analyzes adult chest X-ray images for the presence of pre-specified suspected critical findings (pleural effusion and/or pneumothorax).
RADIFY® Triage uses an artificial intelligence algorithm to analyze images for features suggestive of critical findings and provides case-level output available in the PACS for worklist prioritization or triage.
As a passive notification for prioritization-only software tool within the standard of care workflow, RADIFY® Triage does not send a proactive alert directly to the appropriately trained medical specialists. The product is not intended to direct attention to specific portions of an image. Its results are not intended to be used on a stand-alone basis for clinical decision-making. The device does not remove the cases from the queue and does not flag the condition as being absent.
Type of Use (Select one or both, as applicable)
Prescription Use (Part 21 CFR 801 Subpart D)
ver-The-Counter Use (21 CFR 801 Subpart C)
CONTINUE ON A SEPARATE PAGE IF NEEDED.
This section applies only to requirements of the Paperwork Reduction Act of 1995.
DO NOT SEND YOUR COMPLETED FORM TO THE PRA STAFF EMAIL ADDRESS BELOW.
The burden time for this collection of information is estimated to average 79 hours per response, including the time to review instructions, search existing data sources, gather and maintain the data needed and complete and review the collection of information. Send comments regarding this burden estimate or any other aspect of this information collection, including suggestions for reducing this burden, to:
Department of Health and Human Services Food and Drug Administration Office of Chief Information Officer Paperwork Reduction Act (PRA) Staff PRAStaff@fda.hhs.gov
"An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB number."
3
K231871
510(k) Summary K231871 13th January 2024
1. SUBMITTER INFORMATION
Company address:
Envisionit Deep AI Ltd Coveham House, Downside Bridge Road, Cobham, KT11 3 EP, United Kingdom
Contact: Dr Jaishree Naidoo Phone: +447440058274 info@envisionit.ai
2. DEVICE
Name of Device: RADIFY® Triage Common or Usual name: RADIFY® Triage Classification Name: Radiological Computer-Assisted Prioritization Software For Lesions Classification Regulation: 21 CFR 892.2080 Regulatory Class: Class II Product Code: QFM
3. PREDICATE DEVICE
The Envisionit Deep AI RADIFY® Triage is substantially equivalent to the following devices:
| Primary Predicate: | |
|---|---|
| Manufacturer Name | Lunit Inc |
| Device Trade Name | Lunit INSIGHT CXR Triage |
| 510(k) Number | K211733 |
4. INTENDED USE/INDICATIONS FOR USE
RADIFY® Triage is a radiological computer-assisted triage and notification software that analyzes adult chest X-ray images for the presence of pre-specified suspected critical findings (pleural effusion and/or pneumothorax).
RADIFY® Triage uses an artificial intelligence algorithm to analyze images for features suggestive of critical findings and provides case-level output available in the PACS for worklist prioritization or triage.
4
As a passive notification for prioritization-only software tool within the standard of care workflow, RADIFY® Triage does not send a proactive alert directly to the appropriately trained medical specialists. The product is not intended to direct attention to specific portions of an image. Its results are not intended to be used on a stand-alone basis for clinical decision-making. The device does not remove the cases from the queue and does not flag the condition as being absent.
5. DEVICE DESCRIPTION
RADIFY® Triage is a radiological computer-assisted prioritization software that utilizes Albased image analysis algorithms to identify pre-specified critical findings (pleural effusion and/or pneumothorax) on frontal (AP and PA) views chest X-ray images and flag the images in the PACS to enable worklist prioritization by the appropriately trained medical specialists who are qualified to interpret chest radiographs. The software does not alter the order or remove cases from the reading queue.
The algorithm was trained on datasets from US and non-USA sources. This training dataset consisted of 93.7% of the data from South Africa, and 6.3% of the data from the USA. The input for RADIFY® Triage is a frontal chest x-ray (AP and PA view) in digital imaging and communications in medicine (DICOM) format.
Chest X-rays are sent to RADIFY® Triage via PACS (Picture Archiving and Communication System (PACS) and processed by the device for analysis. Following receipt of chest x-rays, the software device automatically analyses each image to detect features suggestive of pneumothorax and/or pleural effusion. Chest x-rays without the suspicious findings are placed in the worklist for routine review, which is the standard of care. RADIFY® Triage does not provide any proactive alerts and is not intended to direct attention to specific portions of the image. The results are not intended to be used on a standalone basis for clinical decision-making nor is it intended to rule out the target conditions or otherwise preclude clinical assessment of x-ray cases.
6. COMPARISON OF PREDICATE DEVICES
RADIFY® Triage and the identified predicate device(s) are software-only devices that use Artificial intelligence (AI) algorithms and are intended to aid in triage and prioritization of radiological images.
Both RADIFY® Triage and Lunit Insight (Primary Predicate) have the same principles of operation and underlying technological characteristics:
-
- Artificial Intelligence Algorithm(s) Deep Convolutional Neural Networks (DCNN)
-
- Triage and notification software
There are no notable technological differences between the subject and the primary predicate device. Both Envisionit Deep Al and Lunit INSIGHT CXR Triage analyze chest x-rays for the presence of pre-specified target conditions - pleural effusion and pneumothorax.
5
Both devices identify time-sensitive findings and provide passive notification for suspected pneumothorax and pleural effusion.
In terms of establishing substantial equivalence, the subject and predicate device have the same intended use, as an image processing tool that triages images for features suggestive of critical findings and produces case-level output. The indications for use proposed for the subject device are similar to those of the predicate device.
Comparison of the key features of the subject, and predicate devices is provided in Table 1.
6
| Predicate Device | Subject Device | |
|---|---|---|
| Lunit INSIGHT CXR Triage | RADIFY® Triage | |
| Device Name | Lunit INSIGHT CXR Triage | RADIFY® Triage |
| 510 (k) Number | K211733 | K231871 |
| Regulation | 21 CFR 892.2080 | 21 CFR 892.2080 |
| Regulation Description | Radiological computer aided | |
| triage and notification | ||
| software | Radiological computer aided | |
| triage and notification | ||
| software | ||
| Product Code | QFM | QFM |
| Device Type | Radiological Computer- | |
| Assisted Prioritization | ||
| Software For Lesions | Radiological Computer- | |
| Assisted Prioritization | ||
| Software For Lesions | ||
| Manufacturer | Lunit Inc. | Envisionit Deep Al |
| Intended Use/Indications for | ||
| Use | Lunit Insight CXR Triage is a | |
| radiological computer- | ||
| assisted triage and | ||
| notification software that | ||
| analyzes adult chest X-ray | ||
| images for the presence of | ||
| pre-specified suspected | ||
| critical findings (pleural | ||
| effusion and/or | ||
| pneumothorax). Lunit | ||
| INSIGHT CXR Triage uses an | ||
| artificial intelligence | ||
| algorithm to | ||
| analyze images for features | ||
| suggestive of critical findings | ||
| and | ||
| provides case-level output | ||
| available | ||
| in the PACS/workstation for | ||
| worklist prioritization or | ||
| triage. | ||
| As a passive notification for | ||
| prioritization-only software | RADIFY® Triage is a | |
| radiological computer- | ||
| assisted triage and | ||
| notification software that | ||
| analyzes adult chest X-ray | ||
| images for the presence of | ||
| pre-specified suspected | ||
| critical findings (pleural | ||
| effusion and/or | ||
| pneumothorax). | ||
| RADIFY® Triage uses an | ||
| artificial intelligence | ||
| algorithm to analyze images | ||
| for features suggestive of | ||
| critical findings and provides | ||
| case-level output available | ||
| in the PACS for worklist | ||
| prioritization or triage. | ||
| As a passive notification for | ||
| prioritization-only software | ||
| tool within the standard of | ||
| care workflow, RADIFY® | ||
| Predicate Device | Subject Device | |
| Lunit INSIGHT CXR Triage | RADIFY® Triage | |
| tool within standard of care | ||
| workflow, Lunit INSIGHT | ||
| CXR Triage does not send a | ||
| proactive alert directly to | ||
| the appropriately trained | ||
| medical specialists. Lunit | ||
| INSIGHT CXR Triage is not | ||
| intended to direct attention | ||
| to specific portions of an | ||
| image or to anomalies other | ||
| than pleural effusion and/or | ||
| pneumothorax. Its results | ||
| are not intended to be used | ||
| on a stand-alone basis for | ||
| clinical decision-making. | Triage does not send a | |
| proactive alert directly to | ||
| the appropriately trained | ||
| medical specialists. The | ||
| product is not intended to | ||
| direct attention to specific | ||
| portions of an image. Its | ||
| results are not intended to | ||
| be used on a stand-alone | ||
| basis for clinical decision- | ||
| making. The device does not | ||
| remove the cases from the | ||
| queue and does not flag the | ||
| condition as being absent. | ||
| Intended User | Appropriately trained | |
| medical specialists who are | ||
| qualified to interpret chest | ||
| radiographs. | Appropriately trained | |
| medical specialists who are | ||
| qualified to interpret chest | ||
| radiographs. | ||
| Modality | Chest X-ray | Chest X-ray |
| Target Clinical Conditions | Pleural effusion, | |
| Pneumothorax on | ||
| Chest/Lung Frontal Chest X- | ||
| ray | Pleural effusion, | |
| Pneumothorax on | ||
| Chest/Lung Frontal Chest X- | ||
| ray | ||
| Algorithm for pre-specified | ||
| critical findings detection | Al algorithm designed to | |
| detect pleural effusion and | ||
| pneumothorax in chest X- | ||
| ray images. Lunit INSIGHT | ||
| CXR Triage uses a vendor | ||
| agnostic algorithm | ||
| compatible with DICOM | ||
| chest Xray images | Al algorithm designed to | |
| detect pleural effusion and | ||
| pneumothorax in chest X- | ||
| ray images. RADIFY® Triage | ||
| uses a vendor agnostic | ||
| algorithm compatible with | ||
| DICOM chest Xray images | ||
| Notification only/Parallel | ||
| workflow | Yes | Yes |
| Input format | DICOM | DICOM |
| Device output in case of | ||
| positive detection | When deployed on other | |
| radiological imaging | When deployed on other | |
| radiological imaging | ||
| Predicate Device | Subject Device | |
| Lunit INSIGHT CXR Triage | RADIFY® Triage | |
| equipment, Lunit INSIGHT CXR Triage automatically runs after image acquisition and prioritizes and displays the analysis result through the worklist interface of PACS/workstation. No markup on original image. Secondary capture of the finding. Upon image acquisition from other radiological imaging equipment (e.g. X-ray systems), an on-device, technologist notification indicating which cases were flagged by Lunit INSIGHT CXR Triage in PACS, is generated. The on device notification is contextual and does not provide any diagnostic information. It is not intended to inform any clinical decision, prioritization, or action to the technologist | equipment, RADIFY® Triage automatically runs after image acquisition and prioritizes and displays the analysis result through the worklist interface of PACS. No markup on original image. Secondary capture of the finding. | |
| Notification (i.e., recipient, timing and means of notification) | Passive notification. Images with suspicion of pleural effusion and/or pneumothorax are flagged in PACS/workstation. | Passive notification. Images with suspicion of pleural effusion and/or pneumothorax are flagged in PACS/workstation. |
| Where generated results (i.e., DICOM files) are stored | PACS/Workstation | PACS/Workstation |
| Performance level – Timing of notification | The average time taken for the notification to travel from the Lunit INSIGHT CXR Triage to the point at which the result is displayed in the destination PACS/RIS/EPR worklist is 14.66 seconds. | The average time taken for the notification to travel from the RADIFY® Triage to the point at which the result is displayed in the destination PACS/RIS/EPR worklist is 3 seconds. |
| Predicate Device | Subject Device | |
| Lunit INSIGHT CXR Triage | RADIFY® Triage | |
| Performance level – | ||
| accuracy of classification | Pleural Effusion | Pleural Effusion |
| ROC AUC > 0.95 | ROC AUC > 0.95 | |
| AUC: 0.9686 (95% CI: | ||
| [0.9547, 0.9824]) | AUC: 97.61 (95% CI: [97.36, | |
| 97.86]) | ||
| Sensitivity 89.86% (95% CI: | ||
| [86.72, 93.00]) | Sensitivity 94.39% (95% CI: | |
| [93.26, 95.51]) | ||
| Specificity 93.48% (95% CI: | ||
| [91.06, 95.91]) | Specificity 96.42% (95% CI: | |
| [95.29, 98.00]) | ||
| Pneumothorax | Pneumothorax | |
| ROC AUC > 0.95 | ROC AUC > 0.95 | |
| AUC: 0.9630 (95% CI: | ||
| [0.9521, 0.9739]) | AUC: 97.43 (95% CI: [97.12, | |
| 97.74]) | ||
| Sensitivity 88.92% (95% CI: | ||
| [85.60, 92.24]) | Sensitivity 94.81% (95% CI: | |
| [93.90, 95.73]) | ||
| Specificity 90.51% (95% CI: | ||
| [88.18, 92.83]) | Specificity 97.91% (95% CI: | |
| [97.00, 98.83]) |
Table 1. Summary of Substantial Equivalence to Predicate Device
7
8
9
7. TESTING
Software
Software verification and validation testing were conducted, and documentation was provided as recommended by FDA's Guidance for Industry and FDA Staff, "Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices." The software documentation level for this device is Basic Documentation.
Performance Testing - Clinical
The performance of RADIFY® Triage was validated by clinical tests. All the safety parameters of the device were verified in accordance with the software specifications and applicable performance standards and met the acceptance criteria (Device shows > 95% AUC) (passed), demonstrating that the software fulfils all its requirement specifications.
Clinical studies were conducted on retrospectively collected Chest X-rays to evaluate the performance of RADIFY® Triage for triaging of pneumothorax and pleural effusion. There was total independence between the training and test dataset with a new study of unseen images from an independent dataset from 2 sites in the US being used to present the performance testing results as follows.
Pneumothorax
10
The study dataset included a total of 1229 scans with pneumothorax and 959 scans without pneumothorax) from various parts of the US.
The dataset was obtained from three hospitals across the US in order to generate evidence on the device function in various subgroups. A data collection protocol was set in place to ensure that there were sufficient numbers of important subgroups. The protocol specifies the inclusion and exclusion criteria and the expected numbers of cases and controls to be included.
The dataset consisted of 670 males and 559 females. The cases were aged from 22 years to above 85 years. To assess safety and performance of RADIFY® Triage device, a total of 1229 images were obtained from a large urban hospital in New York City and 3 different private clinics in urban and suburban areas in Texas state, in order to generate evidence on the device function in various subgroups. The dataset consisted of clinical confounders that included cardiomegaly, consolidation, infiltrates, opacities, and nodules. The dataset was also obtained from various X-ray device manufacturers to ensure consistent performance.
The algorithm was trained on training data from across the world, consisting of 93.9% of the data from South Africa and 6.1% of the data from the USA.
The external validation test set was obtained from sites that were different from the training data sites and therefore this ensured the independence of the test data from training data.
The ground truth was established by 3 board-certified ABR (USA) radiologists with a minimum of 11 years of experience.
The AUC of the device in triaging scans with findings suspicious of pneumothorax exceeded the acceptance criteria with AUC: 97.43 (95% Cl: [97.12, 97.74]), Sensitivity 94.81% (95% Cl: [93.90, 95.73]) and Specificity 97.91% (95% Cl: [97.00, 98.83]).
The predicate Lunit INSIGHT CXR Triage's performance was ROC AUC 96.30 (95% Cl: 95.21 -97.39), Sensitivity 88.92% (95% Cl: 85.60 - 92.24) and Specificity 90.51% (95% Cl: 88.18 -92.83).
The device's generalizability was ensured by performing subgroup analyses. The results for pneumothorax were consistent in both genders and the AUC was 97.23 (96.83, 97.64), Sensitivity 95.45, 95% (CI 94.17, 96.74) and Specificity 97.57, 95% CI (96.28, 98.86) in male, and AUC 97.52 (97.03, 98.02), Sensitivity 93.62, 95% Cl (92.32, 94.91), and Specificity 98.28, 95% CI (96.98, 99.57) in female.
The device's results were also found to be consistent in a wide range of ages: with AUC of 97.95 95% Cl (97.26, 98.64) in the 22-44 age group, 98.70 95% Cl (98.30, 99.10) in the 45-64 age group, 96.72 95% Cl (96.18, 97.26) in the 65-84 age group and 95.64 95% Cl (94.41, 96.87) in the 85 or greater age group. The device's results were also consistent across image acquisition devices.
In the 88 scans that had both pleural effusion and pneumothorax co-existing, performance accuracy of detecting Pneumothorax was maintained at: AUC of 96.36% (95% Cl [95.78, 96.93]).
11
Pleural Effusion
The study dataset included a total 1229 scans (392 scans with pleural effusion and 837 scans without pleural effusion) from a large urban hospital in New York City and 3 different private clinics in urban and suburban areas in Texas state. A data collection protocol was set in place to ensure that there were sufficient numbers of important subgroups. There were 559 scans from females and 670 from males. The ages ranged from 22 years to greater than 85 years.
The ground truth was established by 3 board-certified ABR (USA) radiologists with a minimum of 11 years of experience.
The AUC of the device in triaging scans with findings suspicious of pleural effusion exceeded the acceptance criteria with AUC: 97.61 (95% Cl: [97.36, 97.86]), Sensitivity 94.39% (95% Cl: [93.26, 95.51]) and Specificity 96.42% (95% CI: [95.29, 98.00]).
The predicate, Lunit INSIGHT CXR Triage, reported a performance of ROC AUC 0.9686 (95% Cl: 0.9547 - 0.9824), sensitivity 89.86% (95% Cl: 86.72 - 93.00) and specificity 93.48% (95% CI: 91.06 - 95.91).
The device's generalizability for triaging scans with pleural effusion was ensured by performing subgroup analyses.
The results for pleural effusion were consistent in both genders and the AUC was 97.62, 95% Cl (97.24,97.99), Sensitivity 92.44, 95% Cl (90.83, 94.05) and Specificity 97.67, 95% Cl (96.06, 99.29) in female, and AUC 97.61, 95% Cl (97.28, 97.94), Sensitivity 95.91, 95% Cl (94.34, 97.48) and Specificity 95.33, 95% CI (93.77, 96.90) in male.
The device's results were also found to be consistent in a wide range of ages: with AUC of 97.36,95% Cl (96.61, 98.11) in the 22-44 age group, 96.32 95% Cl (95.76, 96.88) in the 45-64 age group, 97.94 95% Cl (97.57, 98.32) in the 65-84 age group and 98.73, 95% Cl (98.33, 99.14) in the 85 or greater age group. The device's results were also consistent across image acquisition devices.
In the 88 scans that had both pleural effusion and pneumothorax co-existing, performance accuracy of detecting Pleural effusion was maintained at: AUC of 98.20% (95% Cl [97.82, 98.58]).
The device's performance time was also assessed for RADIFY® Triage. This is the time to analyze the study and send the notification to the worklist. The performance time averaged at 3s. This is comparable to the performance of the predicate performance of 14.66s. Table 2 below shows overall performance results.
12
| Sensitivity (%) | Specificity (%) | AUC (%) (95% CI) | |
|---|---|---|---|
| Pneumothorax | 94.81 | ||
| 95% CI (93.90, 95.73) | 97.91 | ||
| 95% CI (97.00, 98.83) | 97.43 | ||
| 95% CI (97.12, 97.74) | |||
| Pleural Effusion | 94.39 | ||
| 95% CI (93.26, 95.5) | 97.642 | ||
| 95% CI(95.29, 97.54) | 97.61 | ||
| 95% CI (97.36, 97.86) | |||
| All | 94.26 | ||
| 95% CI (93.53, 94.99) | 97.27 | ||
| 95% CI (96.54, 98.00) | 97.62 | ||
| 95% CI (97.43, 97.81) |
Table 2 Overall results of Accuracy testing for RADIFY® Triage
8. CONCLUSION
The comparison of devices in Table 1 and the software and performance testing presented above demonstrate that the RADIFY® Triage device is substantially equivalent to the predicate device. The RADIFY® Triage is a software-only device, similar to the predicate (Lunit INSIGHT CXR Triage). It is as safe and effective as the predicate device. It has the same intended users and similar indications, technological characteristics, and principles of operation as the predicate device.
There are no differences in the indications, there is no risk of its safety and effectiveness being affected when used as labelled. Both devices operate in parallel to the standard of care workflow. The performance testing demonstrates that the RADIFY® Triage performs as intended and is therefore substantially equivalent to the predicate. Software and Clinical testing support that the device performs in accordance with the device requirements.