HALO is a notification only cloud-based image processing software artificial intelligence algorithms to analyze patient imaging data in parallel to the standard of care imaging interpretation. Its intended use is to identify suggestive imaging patterns of a pre-specified condition and to directly notify an appropriate medical specialist.

HALO's indication is to facilitate the evaluation of the brain vasculature on patients suspected of stroke by processing and analyzing CT angiograms of the brain acquired in an acute setting. After completion of the data analysis, HALO sends a notification if a pattern suggestive for a suspected intracranial Large Vessel Occlusion (LVO) of the anterior circulation (ICA, M1 or M2) has been identified in an image.

The intended users of HALO are defined as medical specialists or a team of specialists that are involved in the diagnosis and care of stroke patients at emergency department where stroke patients are administered. The include physicians such as neurologists, radiologists, and/or other emergency department physicians.

HALO's output should not be used for primary diagnosis or clinical decisions; the final diagnosis is always decided upon by the medical specialist. HALO is indicated for CT scanners from GE Healthcare and Philips.

Device Description

HALO is a notification only, cloud-based clinical support tool which identifies image features and communicates the analysis results to a specialist in parallel to the standard of care workflow.

HALO is designed to process CT angiograms of the brain and facilitate evaluation of these images using artificial intelligence to detect patterns suggestive of an intracranial large vessel occlusion (LVO) of the anterior circulation.

A copy of the original CTA images is sent to HALO cloud servers for automatic image processing. After analyzing the images, HALO sends a notification regarding a suspected finding to a specialist, recommending review of these images. The specialist can review the results remotely in a compatible DICOM web viewer.

AI/ML Overview

Here's a detailed breakdown of the acceptance criteria and study proving the device meets them, based on the provided FDA 510(k) summary for HALO:

1. Table of Acceptance Criteria and Reported Device Performance

Acceptance Criteria	Reported Device Performance
Primary Endpoints:
LVO Detection Sensitivity	91.3% (95% CI, 86.6%-94.8%)
LVO Detection Specificity	85.9% (95% CI, 80.6%-90.2%)
Area Under the Curve (AUC) for LVO Detection	0.97
Secondary Endpoints:
Median Notification Time for Detected LVOs	4 minutes 29 seconds (minimum 3:47, maximum 7:12)

The document states that "The HALO performance with regard to sensitivity and specificity, and the notification time are both equivalent to that of the selected predicate device." This implies that the reported performance metrics met or exceeded the established criteria for substantial equivalence to the predicate.

2. Sample Size and Data Provenance

Test Set Sample Size: 427 patients after exclusions (originally 434 CTA scans).
Data Provenance: Retrospective, multi-center clinical study. Patients were admitted to US comprehensive stroke centers.

3. Number and Qualifications of Experts for Ground Truth

Number of Experts: 3 neuro radiologists.
Qualifications: "Expert panel consisting of 3 neuro radiologists." Specific details on years of experience or board certification are not provided in this document.

4. Adjudication Method for the Test Set

The document states: "Ground truth was established by an expert panel consisting of 3 neuro radiologists." While it doesn't explicitly detail the adjudication method (e.g., 2+1, 3+1, consensus discussion), the wording suggests a consensus-based approach among the three experts. "Established by" implies a final, agreed-upon determination, not individual readings.

5. Multi Reader Multi Case (MRMC) Comparative Effectiveness Study

No MRMC comparative effectiveness study involving human readers with vs. without AI assistance is mentioned in the provided text for this specific device clearance. The study described focuses on the standalone performance of the AI algorithm.

6. Standalone (Algorithm Only) Performance

Yes, a standalone performance study was done. The reported sensitivity, specificity, and AUC are all metrics of the algorithm's performance without human intervention in the diagnosis and notification process. The intended use of HALO is to "directly notify an appropriate medical specialist" if a suspected finding is identified, running "in parallel to the standard of care imaging interpretation." This means its function is to flag cases for specialist review, not to replace it.

7. Type of Ground Truth Used

The ground truth used was expert consensus among three neuro radiologists, based on their interpretation of the CTA scans.

8. Sample Size for the Training Set

The document does not specify the sample size used for the training set. It only mentions the test set of 427 patients. It alludes to the algorithm using "a database of images" for its AI model but provides no numbers for this database's size or composition regarding training.

9. How Ground Truth for the Training Set Was Established

The document does not explicitly state how the ground truth for the training set was established. It only details the ground truth establishment for the test set. It is common practice for training data ground truth to be established through expert labeling or other robust methods, but this information is not provided here.

Summary

{0}------------------------------------------------

July 8, 2021

Image /page/0/Picture/1 description: The image shows the logo of the U.S. Food and Drug Administration (FDA). The logo consists of two parts: on the left, there is a seal with an eagle emblem, and on the right, there is the text "FDA U.S. FOOD & DRUG ADMINISTRATION" in blue. The text is arranged in three lines, with "FDA" in a larger font size and a blue square behind it.

Nico.Lab B.V. % Ruojuan Zhang RA Consultant MD Squared B.V. High Tech Campus 29 Eindhoven. Noord Brabant 5656 AE THE NETHERLANDS

Re: K211788

Trade/Device Name: HALO Regulation Number: 21 CFR 892.2080 Regulation Name: Radiological computer aided triage and notification software Regulatory Class: Class II Product Code: QAS Dated: June 5, 2021 Received: June 9, 2021

Dear Ruojuan Zhang:

We have reviewed your Section 510(k) premarket notification of intent to market the device referenced above and have determined the device is substantially equivalent (for the indications for use stated in the enclosure) to legally marketed predicate devices marketed in interstate commerce prior to May 28, 1976, the enactment date of the Medical Device Amendments, or to devices that have been reclassified in accordance with the provisions of the Federal Food, Drug, and Cosmetic Act (Act) that do not require approval of a premarket approval application (PMA). You may, therefore, market the device, subject to the general controls provisions of the Act. Although this letter refers to your product as a device, please be aware that some cleared products may instead be combination products. The 510(k) Premarket Notification Database located at https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm identifies combination product submissions. The general controls provisions of the Act include requirements for annual registration, listing of devices, good manufacturing practice, labeling, and prohibitions against misbranding and adulteration. Please note: CDRH does not evaluate information related to contract liability warranties. We remind you, however, that device labeling must be truthful and not misleading.

If your device is classified (see above) into either class II (Special Controls) or class III (PMA), it may be subject to additional controls. Existing major regulations affecting your device can be found in the Code of Federal Regulations, Title 21, Parts 800 to 898. In addition, FDA may publish further announcements concerning your device in the Federal Register.

Please be advised that FDA's issuance of a substantial equivalence determination does not mean that FDA has made a determination that your device complies with other requirements of the Act or any Federal statutes and regulations administered by other Federal agencies. You must comply with all the Act's requirements, including, but not limited to: registration and listing (21 CFR Part 807); labeling (21 CFR Part

{1}------------------------------------------------

801); medical device reporting of medical device-related adverse events) (21 CFR 803) for devices or postmarketing safety reporting (21 CFR 4, Subpart B) for combination products (see https://www.fda.gov/combination-products/guidance-regulatory-information/postmarketing-safety-reportingcombination-products); good manufacturing practice requirements as set forth in the quality systems (QS) regulation (21 CFR Part 820) for devices or current good manufacturing practices (21 CFR 4. Subpart A) for combination products; and, if applicable, the electronic product radiation control provisions (Sections 531-542 of the Act); 21 CFR 1000-1050.

Also, please note the regulation entitled, "Misbranding by reference to premarket notification" (21 CFR Part 807.97). For questions regarding the reporting of adverse events under the MDR regulation (21 CFR Part 803), please go to https://www.fda.gov/medical-device-safety/medical-device-reportingmdr-how-report-medical-device-problems.

For comprehensive regulatory information about mediation-emitting products, including information about labeling regulations, please see Device Advice (https://www.fda.gov/medicaldevices/device-advice-comprehensive-regulatory-assistance) and CDRH Learn (https://www.fda.gov/training-and-continuing-education/cdrh-learn). Additionally, you may contact the Division of Industry and Consumer Education (DICE) to ask a question about a specific regulatory topic. See the DICE website (https://www.fda.gov/medical-device-advice-comprehensive-regulatoryassistance/contact-us-division-industry-and-consumer-education-dice) for more information or contact DICE by email (DICE@fda.hhs.gov) or phone (1-800-638-2041 or 301-796-7100).

Sincerely,

For

Thalia T. Mills, Ph.D. Director Division of Radiological Health OHT7: Office of In Vitro Diagnostics and Radiological Health Office of Product Evaluation and Quality Center for Devices and Radiological Health

Enclosure

{2}------------------------------------------------

DEPARTMENT OF HEALTH AND HUMAN SERVICES Food and Drug Administration

Indications for Use

510(k) Number (if known)

K211788

Device Name HALO

Indications for Use (Describe)

Type of Use (Select one or both, as applicable)
✓ Prescription Use (Part 21 CFR 801 Subpart D)	Over-The-Counter Use (21 CFR 801 Subpart C)
CONTINUE ON A SEPARATE PAGE IF NEEDED.
This section applies only to requirements of the Paperwork Reduction Act of 1995.
DO NOT SEND YOUR COMPLETED FORM TO THE PRA STAFF EMAIL ADDRESS BELOW.
The burden time for this collection of information is estimated to average 79 hours per response, including the time to review instructions, search existing data sources, gather and maintain the data needed and complete and review the collection of information. Send comments regarding this burden estimate or any other aspect of this information collection, including suggestions for reducing this burden, to:
Department of Health and Human ServicesFood and Drug AdministrationOffice of Chief Information OfficerPaperwork Reduction Act (PRA) StaffPRAStaff@fda.hhs.gov
"An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB number."

{3}------------------------------------------------

510(k) Summary

This 510(k) summary of safety and effectiveness information is prepared in accordance with 21 CFR §807.92.

Submission Number:	K211788
Date Prepared:	June 4, 2021
Manufacturer:	NICO.LAB B.V.Paasheuvelweg 25, 1105 BP AmsterdamThe Netherlands
Primary ContactPerson:	Merel BoersCEOPhone: +31 6 22177913E-mail: mboers@nico-lab.com
Device:
	Trade Name:	HALO
	Classification Name:	Radiological Computer-Assissted Triage andNotification Software
	Classification Regulation:	21CFR §892.2080
	Classification Panel:	Radiology
	Device Class:	Class II
	Primary Product Code:	QAS
Primary PredicateDevice:
	Trade Name:	HALO
	Manufacturer:	NICO.LAB B.V.
	510(k) Clearance:	K200873 (November 20, 2020)
	Classification Name:	Radiological Computer-Assissted Triage andNotification Software
	Classification Regulation:	21CFR §892.2080
	Classification Panel:	Radiology
	Device Class:	Class II
	Product Code:	QAS
Device description:	HALO is a notification only, cloud-based clinical support tool whichidentifies image features and communicates the analysis results to aspecialist in parallel to the standard of care workflow.HALO is designed to process CT angiograms of the brain and facilitateevaluation of these images using artificial intelligence to detect patternssuggestive of an intracranial large vessel occlusion (LVO) of the anteriorcirculation.A copy of the original CTA images is sent to HALO cloud servers forautomatic image processing. After analyzing the images, HALO sends anotification regarding a suspected finding to a specialist, recommendingreview of these images. The specialist can review the results remotely ina compatible DICOM web viewer.
Indications for Use:	HALO is a notification only cloud-based image processing softwareapplication using artificial intelligence algorithms to analyze patientimaging data in parallel to the standard of care imaging interpretation. Itsintended use is to identify suggestive imaging patterns of a pre-specifiedclinical condition and to directly notify an appropriate medical specialist.HALO's indication is to facilitate the evaluation of the brain vasculatureon patients suspected of stroke by processing and analyzing CTangiograms of the brain acquired in an acute setting. After completion ofthe data analysis, HALO sends a notification if a pattern suggestive for asuspected intracranial Large Vessel Occlusion (LVO) of the anteriorcirculation (ICA, M1 or M2) has been identified in an image.The intended users of HALO are defined as medical specialists or a teamof specialists that are involved in the diagnosis and care of stroke patientsat emergency departments or other departments where stroke patients areadministered. They include physicians such as neurologists, radiologists,and/or other emergency department physicians.HALO's output should not be used for primary diagnosis or clinicaldecisions; the final diagnosis is always decided upon by the medicalspecialist. HALO is indicated for CT scanners from GE Healthcare andPhilips.
Technologicalcharacteristics:	The subject device, HALO, is substantially equivalent to the predicatedevice, the previously cleared version of the HALO (K200873). Incomparing the technological characteristics, both the subject andpredicate devices process CT angiograms of the brain and facilitateevaluation of these images using artificial intelligence to detect patternssuggestive of an intracranial large vessel occlusion (LVO) of the anterior

{4}------------------------------------------------

{5}------------------------------------------------

circulation. After analyzing the images, the device sends a notification regarding a suspected finding to a specialist, recommending review of these images. Where the subject and predicate only differ is that the subject device not only process images from GE Healthcare, but also Philips scanners.

The technological characteristic of HALO and how it is comparable to the predicate device are summarized below.

	PredicateHALO	SubjectHALO
Clinical condition	Large vesselocclusion	Large vesselocclusion
Anatomical regionof interest	Head	Head
Data acquisitionprotocol	CT angiogram imagesof the brain	CT angiogram imagesof the brain
Segmentation ofregion of interest	No; device does notmark, highlight, ordirect users' attentionto a specific locationin the original image.	No; device does notmark, highlight, ordirect users' attentionto a specific locationin the original image.
Core Algorithm	Artificial intelligencealgorithm withdatabase of images	Artificial intelligencealgorithm withdatabase of images
Device Output/Notification	The software sends anotification email tothe specialistidentifying the studyof interest.Additionally, thedevice provides userwith a link to DICOMWeb viewer allowingusers review theimages.	The software sends anotification email tothe specialistidentifying the studyof interest.Additionally, thedevice provides userwith a link to DICOMWeb viewer allowingusers review theimages.
Triage effectiveness	Notification time isdefined as the timebetween upload ofCTA (CTA imagesare received in thecloud) to theavailability of AIresults (user receivesa notification to view	Notification time isdefined as the timebetween upload ofCTA (CTA imagesare received in thecloud) to theavailability of AIresults (user receivesa notification to view
	the suspectedfinding).	the suspectedfinding).
Independent ofstandard of careworkflow	No cases are removedfrom worklist	No cases are removedfrom worklist

{6}------------------------------------------------

Performance Data:

Summary of Clinical In a multi-center clinical study, the performance of the HALO clinical decision support algorithm for LVO detection was retrospectively evaluated in a consecutive patient cohort admitted to US comprehensive stroke centers.

434 CTA scans of the brain were collected. After exclusion, 427 patients were included for further analyses. Ground truth was established by an expert panel consisting of 3 neuro radiologists. For the primary endpoint: calculation of the performance of the HALO algorithm showed a sensitivity and specificity for LVO detection of respectively 91.3% (95% CI, 86.6%-94.8%) and 85.9% (95% CI, 80.6%-90.2%). The area under the curve (AUC) is 0.97.

For the secondary endpoints the median notification time for the detected LVO cases was 4 minutes 29 seconds, with a minimum of 3:47 and maximal 7:12.

The HALO performance with regard to sensitivity and specificity, and the notification time are both equivalent to that of the selected predicate device. Therefore, the HALO algorithm fulfils the requirement of a suitable screening tool to support diagnosis of LVOs by flagging these scans as requiring urgent radiologist review.

Substantial The clinical performance tests provided in this 510(k) premarket Equivalence notification demonstrate HALO is substantial equivalence to its predicate Conclusion: device, the previously cleared HALO (K200873). HALO has the same intended use and similar indications, technological characteristics, and principles of operation as its predicate device. The minor differences in indications do not alter the triage use of the device and do not affect its safety and effectiveness.

Regulation Number and Section

§ 892.2080 Radiological computer aided triage and notification software.

(a)
Identification. Radiological computer aided triage and notification software is an image processing prescription device intended to aid in prioritization and triage of radiological medical images. The device notifies a designated list of clinicians of the availability of time sensitive radiological medical images for review based on computer aided image analysis of those images performed by the device. The device does not mark, highlight, or direct users' attention to a specific location in the original image. The device does not remove cases from a reading queue. The device operates in parallel with the standard of care, which remains the default option for all cases.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Design verification and validation must include:
(i) A detailed description of the notification and triage algorithms and all underlying image analysis algorithms including, but not limited to, a detailed description of the algorithm inputs and outputs, each major component or block, how the algorithm affects or relates to clinical practice or patient care, and any algorithm limitations.
(ii) A detailed description of pre-specified performance testing protocols and dataset(s) used to assess whether the device will provide effective triage (
e.g., improved time to review of prioritized images for pre-specified clinicians).(iii) Results from performance testing that demonstrate that the device will provide effective triage. The performance assessment must be based on an appropriate measure to estimate the clinical effectiveness. The test dataset must contain sufficient numbers of cases from important cohorts (
e.g., subsets defined by clinically relevant confounders, effect modifiers, associated diseases, and subsets defined by image acquisition characteristics) such that the performance estimates and confidence intervals for these individual subsets can be characterized with the device for the intended use population and imaging equipment.(iv) Stand-alone performance testing protocols and results of the device.
(v) Appropriate software documentation (
e.g., device hazard analysis; software requirements specification document; software design specification document; traceability analysis; description of verification and validation activities including system level test protocol, pass/fail criteria, and results).(2) Labeling must include the following:
(i) A detailed description of the patient population for which the device is indicated for use;
(ii) A detailed description of the intended user and user training that addresses appropriate use protocols for the device;
(iii) Discussion of warnings, precautions, and limitations must include situations in which the device may fail or may not operate at its expected performance level (
e.g., poor image quality for certain subpopulations), as applicable;(iv) A detailed description of compatible imaging hardware, imaging protocols, and requirements for input images;
(v) Device operating instructions; and
(vi) A detailed summary of the performance testing, including: test methods, dataset characteristics, triage effectiveness (
e.g., improved time to review of prioritized images for pre-specified clinicians), diagnostic accuracy of algorithms informing triage decision, and results with associated statistical uncertainty (e.g., confidence intervals), including a summary of subanalyses on case distributions stratified by relevant confounders, such as lesion and organ characteristics, disease stages, and imaging equipment.