RADIFY® Triage is a radiological computer-assisted triage and notification software that analyzes adult chest X-ray images for the presence of pre-specified suspected critical findings (pleural effusion and/or pneumothorax).

RADIFY® Triage uses an artificial intelligence algorithm to analyze images for features suggestive of critical findings and provides case-level output available in the PACS for worklist prioritization or triage.

As a passive notification for prioritization-only software tool within the standard of care workflow, RADIFY® Triage does not send a proactive alert directly to the appropriately trained medical specialists. The product is not intended to direct attention to specific portions of an image. Its results are not intended to be used on a stand-alone basis for clinical decision-making. The device does not remove the cases from the queue and does not flag the condition as being absent.

Device Description

RADIFY® Triage is a radiological computer-assisted prioritization software that utilizes Albased image analysis algorithms to identify pre-specified critical findings (pleural effusion and/or pneumothorax) on frontal (AP and PA) views chest X-ray images and flag the images in the PACS to enable worklist prioritization by the appropriately trained medical specialists who are qualified to interpret chest radiographs. The software does not alter the order or remove cases from the reading queue.

The algorithm was trained on datasets from US and non-USA sources. This training dataset consisted of 93.7% of the data from South Africa, and 6.3% of the data from the USA. The input for RADIFY® Triage is a frontal chest x-ray (AP and PA view) in digital imaging and communications in medicine (DICOM) format.

Chest X-rays are sent to RADIFY® Triage via PACS (Picture Archiving and Communication System (PACS) and processed by the device for analysis. Following receipt of chest x-rays, the software device automatically analyses each image to detect features suggestive of pneumothorax and/or pleural effusion. Chest x-rays without the suspicious findings are placed in the worklist for routine review, which is the standard of care. RADIFY® Triage does not provide any proactive alerts and is not intended to direct attention to specific portions of the image. The results are not intended to be used on a standalone basis for clinical decision-making nor is it intended to rule out the target conditions or otherwise preclude clinical assessment of x-ray cases.

AI/ML Overview

Here's a breakdown of the acceptance criteria and the study details for the Radify® Triage device, based on the provided document:

1. Acceptance Criteria and Reported Device Performance

Condition	Acceptance Criteria (ROC AUC)	Reported Device Performance (ROC AUC)	Reported Device Sensitivity	Reported Device Specificity
Pleural Effusion	> 0.95	0.9761 (95% CI: [0.9736, 0.9786])	94.39% (95% CI: [93.26, 95.51])	96.42% (95% CI: [95.29, 98.00])
Pneumothorax	> 0.95	0.9743 (95% CI: [0.9712, 0.9774])	94.81% (95% CI: [93.90, 95.73])	97.91% (95% CI: [97.00, 98.83])
Overall	N/A (implied by individual)	0.9762 (95% CI: [0.9743, 0.9781])	94.26% (95% CI: [93.53, 94.99])	97.27% (95% CI: [96.54, 98.00])
Notification Time	(Implicitly comparable to predicate)	Average of 3 seconds	N/A	N/A

Note: The document explicitly states the acceptance criteria for performance as "Device shows > 95% AUC".

2. Sample Size and Data Provenance for the Test Set

Test Set Sample Size:
- Pneumothorax: 2188 scans (1229 with pneumothorax + 959 without pneumothorax).
- Pleural Effusion: 1229 scans (392 with pleural effusion + 837 without pleural effusion).
- Shared Cases: 88 scans had both pleural effusion and pneumothorax co-existing.
Data Provenance: Retrospective, obtained from three hospitals across the US: one large urban hospital in New York City and three different private clinics in urban and suburban areas in Texas state.

3. Number, Qualifications, and Adjudication Method of Experts for Test Set Ground Truth

Number of Experts: 3
Qualifications of Experts: Board-certified ABR (USA) radiologists with a minimum of 11 years of experience.
Adjudication Method: Not explicitly stated, but the phrase "The ground truth was established by 3 board-certified ABR (USA) radiologists" implies a consensus-based approach, likely a majority vote or discussion to reach agreement. It does not specify 2+1 or 3+1, but suggests a similar process.

4. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study

Was an MRMC study done? No, the document describes a standalone (algorithm only) performance evaluation against a radiologist-established ground truth. It does not mention a study to compare human reader performance with and without AI assistance.
Effect size of human readers improving with AI vs. without AI assistance: Not applicable as an MRMC comparative effectiveness study was not performed or reported.

5. Standalone Performance Study

Was a standalone study done? Yes, the document details the performance of the RADIFY® Triage algorithm alone, analyzing chest X-ray images for pneumothorax and pleural effusion. The reported metrics (AUC, sensitivity, specificity) are for the algorithm's performance in detecting these conditions compared to the established ground truth.

6. Type of Ground Truth Used

Type of Ground Truth: Expert consensus, established by 3 board-certified ABR (USA) radiologists.

7. Sample Size for the Training Set

Training Set Sample Size: Not explicitly stated as a total number of images, but the composition is given: "The algorithm was trained on datasets from US and non-USA sources. This training dataset consisted of 93.7% of the data from South Africa, and 6.3% of the data from the USA."

8. How Ground Truth for the Training Set Was Established

How Ground Truth Was Established: Not explicitly detailed for the training set. The document only states that the algorithm was trained on datasets and then evaluated on a separate, independent test set where the ground truth was established by the 3 expert radiologists. It's common practice for training data ground truth to be established through similar expert review processes, but this specific detail is not provided for the training data in the given text.

Summary

{0}------------------------------------------------

Image /page/0/Picture/0 description: The image shows the logo of the U.S. Food and Drug Administration (FDA). On the left is the Department of Health & Human Services logo. To the right of that is the FDA logo, which is a blue square with the letters "FDA" in white. To the right of the blue square is the text "U.S. FOOD & DRUG ADMINISTRATION" in blue.

Envisionit DeepAI Ltd % Michael Pogose Director of Quality Assurance and Regulatory Affairs Hardian Ltd t/a Hardian Health c/o Galloways, 3rd Floor 21 Perrymount Road Haywards Heath, West Sussex RH16 3TP United Kingdom

Re: K231871

January 17, 2024

Trade/Device Name: Radify® Triage Regulation Number: 21 CFR 892.2080 Regulation Name: Radiological computer aided triage and notification software Regulatory Class: Class II Product Code: QFM Dated: December 19, 2023 Received: December 19, 2023

Dear Michael Pogose:

We have reviewed your section 510(k) premarket notification of intent to market the device referenced above and have determined the device is substantially equivalent (for the indications for use stated in the enclosure) to legally marketed predicate devices marketed in interstate commerce prior to May 28, 1976, the enactment date of the Medical Device Amendments, or to devices that have been reclassified in accordance with the provisions of the Federal Food, Drug, and Cosmetic Act (the Act) that do not require approval of a premarket approval application (PMA). You may, therefore, market the device, subject to the general controls provisions of the Act. Although this letter refers to your product as a device, please be aware that some cleared products may instead be combination products. The 510(k) Premarket Notification Database available at https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm identifies combination product submissions. The general controls provisions of the Act include requirements for annual registration, listing of devices, good manufacturing practice, labeling, and prohibitions against misbranding and adulteration. Please note: CDRH does not evaluate information related to contract liability warranties. We remind you, however, that device labeling must be truthful and not misleading.

If your device is classified (see above) into either class II (Special Controls) or class III (PMA), it may be subject to additional controls. Existing major regulations affecting your device can be found in the Code of Federal Regulations, Title 21, Parts 800 to 898. In addition, FDA may publish further announcements concerning your device in the Federal Register.

Additional information about changes that may require a new premarket notification are provided in the FDA guidance documents entitled "Deciding When to Submit a 510(k) for a Change to an Existing Device" (https://www.fda.gov/media/99812/download) and "Deciding When to Submit a 510(k) for a Software Change to an Existing Device" (https://www.fda.gov/media/99785/download).

{1}------------------------------------------------

Your device is also subject to, among other requirements, the Quality System (QS) regulation (21 CFR Part 820), which includes, but is not limited to, 21 CFR 820.30, Design controls; 21 CFR 820.90, Nonconforming product; and 21 CFR 820.100, Corrective and preventive action. Please note that regardless of whether a change requires premarket review, the QS regulation requires device manufacturers to review and approve changes to device design and production (21 CFR 820.30 and 21 CFR 820.70) and document changes and approvals in the device master record (21 CFR 820.181).

Please be advised that FDA's issuance of a substantial equivalence determination does not mean that FDA has made a determination that your device complies with other requirements of the Act or any Federal statutes and regulations administered by other Federal agencies. You must comply with all the Act's requirements, including, but not limited to: registration and listing (21 CFR Part 807); labeling (21 CFR Part 801); medical device reporting of medical device-related adverse events) (21 CFR Part 803) for devices or postmarketing safety reporting (21 CFR Part 4, Subpart B) for combination products (see https://www.fda.gov/combination-products/guidance-regulatory-information/postmarketing-safety-reportingcombination-products); good manufacturing practice requirements as set forth in the quality systems (QS) regulation (21 CFR Part 820) for devices or current good manufacturing practices (21 CFR Part 4, Subpart A) for combination products; and, if applicable, the electronic product radiation control provisions (Sections 531-542 of the Act); 21 CFR Parts 1000-1050.

Also, please note the regulation entitled, "Misbranding by reference to premarket notification" (21 CFR 807.97). For questions regarding the reporting of adverse events under the MDR regulation (21 CFR Part 803), please go to https://www.fda.gov/medical-device-safety/medical-device-reportingmdr-how-report-medical-device-problems.

For comprehensive regulatory information about medical devices and radiation-emitting products, including information about labeling regulations, please see Device Advice (https://www.fda.gov/medicaldevices/device-advice-comprehensive-regulatory-assistance) and CDRH Learn (https://www.fda.gov/training-and-continuing-education/cdrh-learn). Additionally, you may contact the Division of Industry and Consumer Education (DICE) to ask a question about a specific regulatory topic. See the DICE website (https://www.fda.gov/medical-device-advice-comprehensive-regulatoryassistance/contact-us-division-industry-and-consumer-education-dice) for more information or contact DICE by email (DICE@fda.hhs.gov) or phone (1-800-638-2041 or 301-796-7100).

Sincerely,

Jessica Lamb

Jessica Lamb. Ph.D. Assistant Director Imaging Software Team DHT8B: Division of Radiological Imaging Devices and Electronic Products OHT8: Office of Radiological Health Office of Product Evaluation and Quality Center for Devices and Radiological Health

{2}------------------------------------------------

Indications for Use

Submission Number (if known)

K231871

Device Name

RADIFY® Triage

Indications for Use (Describe)

Type of Use (Select one or both, as applicable)

Prescription Use (Part 21 CFR 801 Subpart D)

ver-The-Counter Use (21 CFR 801 Subpart C)

CONTINUE ON A SEPARATE PAGE IF NEEDED.

This section applies only to requirements of the Paperwork Reduction Act of 1995.

DO NOT SEND YOUR COMPLETED FORM TO THE PRA STAFF EMAIL ADDRESS BELOW.

The burden time for this collection of information is estimated to average 79 hours per response, including the time to review instructions, search existing data sources, gather and maintain the data needed and complete and review the collection of information. Send comments regarding this burden estimate or any other aspect of this information collection, including suggestions for reducing this burden, to:

Department of Health and Human Services Food and Drug Administration Office of Chief Information Officer Paperwork Reduction Act (PRA) Staff PRAStaff@fda.hhs.gov

"An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB number."

{3}------------------------------------------------

K231871

510(k) Summary K231871 13th January 2024

1. SUBMITTER INFORMATION

Company address:

Envisionit Deep AI Ltd Coveham House, Downside Bridge Road, Cobham, KT11 3 EP, United Kingdom

Contact: Dr Jaishree Naidoo Phone: +447440058274 info@envisionit.ai

2. DEVICE

Name of Device: RADIFY® Triage Common or Usual name: RADIFY® Triage Classification Name: Radiological Computer-Assisted Prioritization Software For Lesions Classification Regulation: 21 CFR 892.2080 Regulatory Class: Class II Product Code: QFM

3. PREDICATE DEVICE

The Envisionit Deep AI RADIFY® Triage is substantially equivalent to the following devices:

Primary Predicate:
Manufacturer Name	Lunit Inc
Device Trade Name	Lunit INSIGHT CXR Triage
510(k) Number	K211733

4. INTENDED USE/INDICATIONS FOR USE

{4}------------------------------------------------

5. DEVICE DESCRIPTION

6. COMPARISON OF PREDICATE DEVICES

RADIFY® Triage and the identified predicate device(s) are software-only devices that use Artificial intelligence (AI) algorithms and are intended to aid in triage and prioritization of radiological images.

Both RADIFY® Triage and Lunit Insight (Primary Predicate) have the same principles of operation and underlying technological characteristics:

1. Artificial Intelligence Algorithm(s) Deep Convolutional Neural Networks (DCNN)
1. Triage and notification software

There are no notable technological differences between the subject and the primary predicate device. Both Envisionit Deep Al and Lunit INSIGHT CXR Triage analyze chest x-rays for the presence of pre-specified target conditions - pleural effusion and pneumothorax.

{5}------------------------------------------------

Both devices identify time-sensitive findings and provide passive notification for suspected pneumothorax and pleural effusion.

In terms of establishing substantial equivalence, the subject and predicate device have the same intended use, as an image processing tool that triages images for features suggestive of critical findings and produces case-level output. The indications for use proposed for the subject device are similar to those of the predicate device.

Comparison of the key features of the subject, and predicate devices is provided in Table 1.

{6}------------------------------------------------

	Predicate Device	Subject Device
	Lunit INSIGHT CXR Triage	RADIFY® Triage
Device Name	Lunit INSIGHT CXR Triage	RADIFY® Triage
510 (k) Number	K211733	K231871
Regulation	21 CFR 892.2080	21 CFR 892.2080
Regulation Description	Radiological computer aidedtriage and notificationsoftware	Radiological computer aidedtriage and notificationsoftware
Product Code	QFM	QFM
Device Type	Radiological Computer-Assisted PrioritizationSoftware For Lesions	Radiological Computer-Assisted PrioritizationSoftware For Lesions
Manufacturer	Lunit Inc.	Envisionit Deep Al
Intended Use/Indications forUse	Lunit Insight CXR Triage is aradiological computer-assisted triage andnotification software thatanalyzes adult chest X-rayimages for the presence ofpre-specified suspectedcritical findings (pleuraleffusion and/orpneumothorax). LunitINSIGHT CXR Triage uses anartificial intelligencealgorithm toanalyze images for featuressuggestive of critical findingsandprovides case-level outputavailablein the PACS/workstation forworklist prioritization ortriage.As a passive notification forprioritization-only software	RADIFY® Triage is aradiological computer-assisted triage andnotification software thatanalyzes adult chest X-rayimages for the presence ofpre-specified suspectedcritical findings (pleuraleffusion and/orpneumothorax).RADIFY® Triage uses anartificial intelligencealgorithm to analyze imagesfor features suggestive ofcritical findings and providescase-level output availablein the PACS for worklistprioritization or triage.As a passive notification forprioritization-only softwaretool within the standard ofcare workflow, RADIFY®
	Predicate Device	Subject Device
	Lunit INSIGHT CXR Triage	RADIFY® Triage
	tool within standard of careworkflow, Lunit INSIGHTCXR Triage does not send aproactive alert directly tothe appropriately trainedmedical specialists. LunitINSIGHT CXR Triage is notintended to direct attentionto specific portions of animage or to anomalies otherthan pleural effusion and/orpneumothorax. Its resultsare not intended to be usedon a stand-alone basis forclinical decision-making.	Triage does not send aproactive alert directly tothe appropriately trainedmedical specialists. Theproduct is not intended todirect attention to specificportions of an image. Itsresults are not intended tobe used on a stand-alonebasis for clinical decision-making. The device does notremove the cases from thequeue and does not flag thecondition as being absent.
Intended User	Appropriately trainedmedical specialists who arequalified to interpret chestradiographs.	Appropriately trainedmedical specialists who arequalified to interpret chestradiographs.
Modality	Chest X-ray	Chest X-ray
Target Clinical Conditions	Pleural effusion,Pneumothorax onChest/Lung Frontal Chest X-ray	Pleural effusion,Pneumothorax onChest/Lung Frontal Chest X-ray
Algorithm for pre-specifiedcritical findings detection	Al algorithm designed todetect pleural effusion andpneumothorax in chest X-ray images. Lunit INSIGHTCXR Triage uses a vendoragnostic algorithmcompatible with DICOMchest Xray images	Al algorithm designed todetect pleural effusion andpneumothorax in chest X-ray images. RADIFY® Triageuses a vendor agnosticalgorithm compatible withDICOM chest Xray images
Notification only/Parallelworkflow	Yes	Yes
Input format	DICOM	DICOM
Device output in case ofpositive detection	When deployed on otherradiological imaging	When deployed on otherradiological imaging
	Predicate Device	Subject Device
	Lunit INSIGHT CXR Triage	RADIFY® Triage
	equipment, Lunit INSIGHT CXR Triage automatically runs after image acquisition and prioritizes and displays the analysis result through the worklist interface of PACS/workstation. No markup on original image. Secondary capture of the finding. Upon image acquisition from other radiological imaging equipment (e.g. X-ray systems), an on-device, technologist notification indicating which cases were flagged by Lunit INSIGHT CXR Triage in PACS, is generated. The on device notification is contextual and does not provide any diagnostic information. It is not intended to inform any clinical decision, prioritization, or action to the technologist	equipment, RADIFY® Triage automatically runs after image acquisition and prioritizes and displays the analysis result through the worklist interface of PACS. No markup on original image. Secondary capture of the finding.
Notification (i.e., recipient, timing and means of notification)	Passive notification. Images with suspicion of pleural effusion and/or pneumothorax are flagged in PACS/workstation.	Passive notification. Images with suspicion of pleural effusion and/or pneumothorax are flagged in PACS/workstation.
Where generated results (i.e., DICOM files) are stored	PACS/Workstation	PACS/Workstation
Performance level – Timing of notification	The average time taken for the notification to travel from the Lunit INSIGHT CXR Triage to the point at which the result is displayed in the destination PACS/RIS/EPR worklist is 14.66 seconds.	The average time taken for the notification to travel from the RADIFY® Triage to the point at which the result is displayed in the destination PACS/RIS/EPR worklist is 3 seconds.
	Predicate Device	Subject Device
	Lunit INSIGHT CXR Triage	RADIFY® Triage
Performance level –accuracy of classification	Pleural Effusion	Pleural Effusion
	ROC AUC > 0.95	ROC AUC > 0.95
	AUC: 0.9686 (95% CI:[0.9547, 0.9824])	AUC: 97.61 (95% CI: [97.36,97.86])
	Sensitivity 89.86% (95% CI:[86.72, 93.00])	Sensitivity 94.39% (95% CI:[93.26, 95.51])
	Specificity 93.48% (95% CI:[91.06, 95.91])	Specificity 96.42% (95% CI:[95.29, 98.00])
	Pneumothorax	Pneumothorax
	ROC AUC > 0.95	ROC AUC > 0.95
	AUC: 0.9630 (95% CI:[0.9521, 0.9739])	AUC: 97.43 (95% CI: [97.12,97.74])
	Sensitivity 88.92% (95% CI:[85.60, 92.24])	Sensitivity 94.81% (95% CI:[93.90, 95.73])
	Specificity 90.51% (95% CI:[88.18, 92.83])	Specificity 97.91% (95% CI:[97.00, 98.83])

Table 1. Summary of Substantial Equivalence to Predicate Device

{7}------------------------------------------------

{8}------------------------------------------------

{9}------------------------------------------------

7. TESTING

Software

Software verification and validation testing were conducted, and documentation was provided as recommended by FDA's Guidance for Industry and FDA Staff, "Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices." The software documentation level for this device is Basic Documentation.

Performance Testing - Clinical

The performance of RADIFY® Triage was validated by clinical tests. All the safety parameters of the device were verified in accordance with the software specifications and applicable performance standards and met the acceptance criteria (Device shows > 95% AUC) (passed), demonstrating that the software fulfils all its requirement specifications.

Clinical studies were conducted on retrospectively collected Chest X-rays to evaluate the performance of RADIFY® Triage for triaging of pneumothorax and pleural effusion. There was total independence between the training and test dataset with a new study of unseen images from an independent dataset from 2 sites in the US being used to present the performance testing results as follows.

Pneumothorax

{10}------------------------------------------------

The study dataset included a total of 1229 scans with pneumothorax and 959 scans without pneumothorax) from various parts of the US.

The dataset was obtained from three hospitals across the US in order to generate evidence on the device function in various subgroups. A data collection protocol was set in place to ensure that there were sufficient numbers of important subgroups. The protocol specifies the inclusion and exclusion criteria and the expected numbers of cases and controls to be included.

The dataset consisted of 670 males and 559 females. The cases were aged from 22 years to above 85 years. To assess safety and performance of RADIFY® Triage device, a total of 1229 images were obtained from a large urban hospital in New York City and 3 different private clinics in urban and suburban areas in Texas state, in order to generate evidence on the device function in various subgroups. The dataset consisted of clinical confounders that included cardiomegaly, consolidation, infiltrates, opacities, and nodules. The dataset was also obtained from various X-ray device manufacturers to ensure consistent performance.

The algorithm was trained on training data from across the world, consisting of 93.9% of the data from South Africa and 6.1% of the data from the USA.

The external validation test set was obtained from sites that were different from the training data sites and therefore this ensured the independence of the test data from training data.

The ground truth was established by 3 board-certified ABR (USA) radiologists with a minimum of 11 years of experience.

The AUC of the device in triaging scans with findings suspicious of pneumothorax exceeded the acceptance criteria with AUC: 97.43 (95% Cl: [97.12, 97.74]), Sensitivity 94.81% (95% Cl: [93.90, 95.73]) and Specificity 97.91% (95% Cl: [97.00, 98.83]).

The predicate Lunit INSIGHT CXR Triage's performance was ROC AUC 96.30 (95% Cl: 95.21 -97.39), Sensitivity 88.92% (95% Cl: 85.60 - 92.24) and Specificity 90.51% (95% Cl: 88.18 -92.83).

The device's generalizability was ensured by performing subgroup analyses. The results for pneumothorax were consistent in both genders and the AUC was 97.23 (96.83, 97.64), Sensitivity 95.45, 95% (CI 94.17, 96.74) and Specificity 97.57, 95% CI (96.28, 98.86) in male, and AUC 97.52 (97.03, 98.02), Sensitivity 93.62, 95% Cl (92.32, 94.91), and Specificity 98.28, 95% CI (96.98, 99.57) in female.

The device's results were also found to be consistent in a wide range of ages: with AUC of 97.95 95% Cl (97.26, 98.64) in the 22-44 age group, 98.70 95% Cl (98.30, 99.10) in the 45-64 age group, 96.72 95% Cl (96.18, 97.26) in the 65-84 age group and 95.64 95% Cl (94.41, 96.87) in the 85 or greater age group. The device's results were also consistent across image acquisition devices.

In the 88 scans that had both pleural effusion and pneumothorax co-existing, performance accuracy of detecting Pneumothorax was maintained at: AUC of 96.36% (95% Cl [95.78, 96.93]).

{11}------------------------------------------------

Pleural Effusion

The study dataset included a total 1229 scans (392 scans with pleural effusion and 837 scans without pleural effusion) from a large urban hospital in New York City and 3 different private clinics in urban and suburban areas in Texas state. A data collection protocol was set in place to ensure that there were sufficient numbers of important subgroups. There were 559 scans from females and 670 from males. The ages ranged from 22 years to greater than 85 years.

The ground truth was established by 3 board-certified ABR (USA) radiologists with a minimum of 11 years of experience.

The AUC of the device in triaging scans with findings suspicious of pleural effusion exceeded the acceptance criteria with AUC: 97.61 (95% Cl: [97.36, 97.86]), Sensitivity 94.39% (95% Cl: [93.26, 95.51]) and Specificity 96.42% (95% CI: [95.29, 98.00]).

The predicate, Lunit INSIGHT CXR Triage, reported a performance of ROC AUC 0.9686 (95% Cl: 0.9547 - 0.9824), sensitivity 89.86% (95% Cl: 86.72 - 93.00) and specificity 93.48% (95% CI: 91.06 - 95.91).

The device's generalizability for triaging scans with pleural effusion was ensured by performing subgroup analyses.

The results for pleural effusion were consistent in both genders and the AUC was 97.62, 95% Cl (97.24,97.99), Sensitivity 92.44, 95% Cl (90.83, 94.05) and Specificity 97.67, 95% Cl (96.06, 99.29) in female, and AUC 97.61, 95% Cl (97.28, 97.94), Sensitivity 95.91, 95% Cl (94.34, 97.48) and Specificity 95.33, 95% CI (93.77, 96.90) in male.

The device's results were also found to be consistent in a wide range of ages: with AUC of 97.36,95% Cl (96.61, 98.11) in the 22-44 age group, 96.32 95% Cl (95.76, 96.88) in the 45-64 age group, 97.94 95% Cl (97.57, 98.32) in the 65-84 age group and 98.73, 95% Cl (98.33, 99.14) in the 85 or greater age group. The device's results were also consistent across image acquisition devices.

In the 88 scans that had both pleural effusion and pneumothorax co-existing, performance accuracy of detecting Pleural effusion was maintained at: AUC of 98.20% (95% Cl [97.82, 98.58]).

The device's performance time was also assessed for RADIFY® Triage. This is the time to analyze the study and send the notification to the worklist. The performance time averaged at 3s. This is comparable to the performance of the predicate performance of 14.66s. Table 2 below shows overall performance results.

{12}------------------------------------------------

	Sensitivity (%)	Specificity (%)	AUC (%) (95% CI)
Pneumothorax	94.8195% CI (93.90, 95.73)	97.9195% CI (97.00, 98.83)	97.4395% CI (97.12, 97.74)
Pleural Effusion	94.3995% CI (93.26, 95.5)	97.64295% CI(95.29, 97.54)	97.6195% CI (97.36, 97.86)
All	94.2695% CI (93.53, 94.99)	97.2795% CI (96.54, 98.00)	97.6295% CI (97.43, 97.81)

Table 2 Overall results of Accuracy testing for RADIFY® Triage

8. CONCLUSION

The comparison of devices in Table 1 and the software and performance testing presented above demonstrate that the RADIFY® Triage device is substantially equivalent to the predicate device. The RADIFY® Triage is a software-only device, similar to the predicate (Lunit INSIGHT CXR Triage). It is as safe and effective as the predicate device. It has the same intended users and similar indications, technological characteristics, and principles of operation as the predicate device.

There are no differences in the indications, there is no risk of its safety and effectiveness being affected when used as labelled. Both devices operate in parallel to the standard of care workflow. The performance testing demonstrates that the RADIFY® Triage performs as intended and is therefore substantially equivalent to the predicate. Software and Clinical testing support that the device performs in accordance with the device requirements.

Regulation Number and Section

§ 892.2080 Radiological computer aided triage and notification software.

(a)
Identification. Radiological computer aided triage and notification software is an image processing prescription device intended to aid in prioritization and triage of radiological medical images. The device notifies a designated list of clinicians of the availability of time sensitive radiological medical images for review based on computer aided image analysis of those images performed by the device. The device does not mark, highlight, or direct users' attention to a specific location in the original image. The device does not remove cases from a reading queue. The device operates in parallel with the standard of care, which remains the default option for all cases.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Design verification and validation must include:
(i) A detailed description of the notification and triage algorithms and all underlying image analysis algorithms including, but not limited to, a detailed description of the algorithm inputs and outputs, each major component or block, how the algorithm affects or relates to clinical practice or patient care, and any algorithm limitations.
(ii) A detailed description of pre-specified performance testing protocols and dataset(s) used to assess whether the device will provide effective triage (
e.g., improved time to review of prioritized images for pre-specified clinicians).(iii) Results from performance testing that demonstrate that the device will provide effective triage. The performance assessment must be based on an appropriate measure to estimate the clinical effectiveness. The test dataset must contain sufficient numbers of cases from important cohorts (
e.g., subsets defined by clinically relevant confounders, effect modifiers, associated diseases, and subsets defined by image acquisition characteristics) such that the performance estimates and confidence intervals for these individual subsets can be characterized with the device for the intended use population and imaging equipment.(iv) Stand-alone performance testing protocols and results of the device.
(v) Appropriate software documentation (
e.g., device hazard analysis; software requirements specification document; software design specification document; traceability analysis; description of verification and validation activities including system level test protocol, pass/fail criteria, and results).(2) Labeling must include the following:
(i) A detailed description of the patient population for which the device is indicated for use;
(ii) A detailed description of the intended user and user training that addresses appropriate use protocols for the device;
(iii) Discussion of warnings, precautions, and limitations must include situations in which the device may fail or may not operate at its expected performance level (
e.g., poor image quality for certain subpopulations), as applicable;(iv) A detailed description of compatible imaging hardware, imaging protocols, and requirements for input images;
(v) Device operating instructions; and
(vi) A detailed summary of the performance testing, including: test methods, dataset characteristics, triage effectiveness (
e.g., improved time to review of prioritized images for pre-specified clinicians), diagnostic accuracy of algorithms informing triage decision, and results with associated statistical uncertainty (e.g., confidence intervals), including a summary of subanalyses on case distributions stratified by relevant confounders, such as lesion and organ characteristics, disease stages, and imaging equipment.