The UltraSight AI Guidance is intended to assist medical professionals (not including expert sonographers) in acquiring cardiac ultrasound images. UltraSight Al Guidance is an accessory to compatible general-purpose diagnostic ultrasound systems. UltraSight Al Guidance is indicated for use in two-dimensional transthoracic echocardiography (2D- TTE) for adult patients, specifically in the acquisition of the following standard views: Parasternal Long-Axis (PLAX), Parasternal Short-Axis at the Aortic Valve (PSAX-AV), Parasternal Short-Axis at the Mitral Valve (PSAX-MV), Parasternal Short-Axis at the Papillary Muscle (PSAX-PM), Apical 4-Chamber (AP4), Apical 5-Chamber (AP5), Apical 2-Chamber (AP2), Apical 3-Chamber (AP3), Subcostal 4-Chamber (SubC4), and Subcostal Inferior Vena Cava (SC-IVC).

Device Description

UltraSight Al Guidance is a mobile application based on machine learning that uses artificial intelligence (AI) to provide dynamic real-time guidance on the position and orientation of the transducer to help non-expert users acquire diagnostic-quality tomographic views of the system provides guidance for ten standard cardiac views.

AI/ML Overview

Here's a detailed breakdown of the acceptance criteria and the studies performed for the UltraSight AI Guidance device, based on the provided text:

1. Table of Acceptance Criteria and Reported Device Performance

For Standalone AI Performance (Algorithm Only):

Performance Metric	Acceptance Criteria	Reported Device Performance (Mean)	95% Confidence Interval	Additional Notes
Quality Bar	AUC > 0.8	0.86	[0.85, 0.87]	Shows good classification performance.
	PPV > 0.75	0.93	[0.92, 0.94]	Shows good classification performance.
View Detection	AUC > 0.8	0.988	(0.985, 0.990)	Good classification performance for "Hold position" vs. "Navigate" and "Hold position" vs. "No heart". Stratified analysis also met this.
Probe Guidance	AUC > 0.8	0.821	[0.813, 0.827]	Good classification performance for guiding probe movements. Stratified tests showed acceptable individual classifiers.

For Clinical Performance (Human-in-the-loop with AI guidance):

Performance Metric (Visual Quality for Visual Assessment - Majority Agreement)	Acceptance Criteria (Implicit: demonstrate non-experts can acquire diagnostic quality)	Reported Device Performance (Non-expert users with AI Guidance)	Additional Notes
LV size	N/A (Comparative to sonographer performance implicitly desired)	93-100% of cases (Pivotal Study)	Sufficient visual quality for assessment.
LV function	N/A	93-100% of cases (Pivotal Study)	Sufficient visual quality for assessment.
RV size	N/A	93-100% of cases (Pivotal Study)	Sufficient visual quality for assessment.
Non-trivial pericardial effusion	N/A	93-100% of cases (Pivotal Study)	Sufficient visual quality for assessment.
MV structure	N/A	98% of cases (Pivotal Study)	Sufficient visual quality for assessment.
RV function	N/A	94% of cases (Pivotal Study)	Sufficient visual quality for assessment.
Left atrium size	N/A	94% of cases (Pivotal Study)	Sufficient visual quality for assessment.
AV structure	N/A	89% of cases (Pivotal Study)	Sufficient visual quality for assessment.
TV structure	N/A	74% of cases (Pivotal Study)	Sufficient visual quality for assessment.
IVC size	N/A	67% of cases (Pivotal Study)	Sufficient visual quality for assessment.
Diagnostic Quality Score >= 3 (ACEP scale) for specific views	N/A	Pilot Study: Range 78.6-97.9% of clips	Most clips taken by non-expert users met this.

Note: The clinical study explicitly states the goal was to evaluate whether non-expert users could acquire diagnostic quality images with the AI guidance. The results demonstrate this effectively, comparing favorably to the predicate. The "acceptance criteria" for clinical performance are implicitly met by successful completion of these clinical endpoints at high percentages.

2. Sample Size Used for the Test Set and Data Provenance

For Standalone AI Performance Testing:

Quality Bar Test Set: 312 clips
View Detection & Guidance Tests Test Set: 75 subjects, totaling 2.3 million frames of ultrasound images.
Data Provenance: The data used for performance testing (test set) was collected at different sites, geographically separated, from the sites used for algorithm development. The data was collected from a population representative of the intended population. It is implied to be retrospective, as it's a pre-collected "test set". The country of origin is not explicitly stated.

For Clinical Performance (Pivotal Study):

Test Set (Subjects): 240 subjects.
Data Provenance: Prospective, multi-center study. Country of origin not explicitly stated, but the submission is to the US FDA, suggesting the study likely included US sites or data relevant to the US population. The comparison group was scans by cardiac sonographers without AI guidance.

3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications

For Standalone AI Performance Testing (Quality Bar):

Number of Experts: 3.
Qualifications: Cardiologists. No further details on their years of experience are provided.

For Clinical Performance (Pilot & Pivotal Studies):

Number of Experts: 5.
Qualifications: Expert cardiologists. No further details on their years of experience are provided.

4. Adjudication Method for the Test Set

For Standalone AI Performance Testing:

Quality Bar: Each clip was annotated with a "diagnosable / non-diagnosable" label by three cardiologists. The final ground truth for "diagnosable" vs. "non-diagnosable" is based on their inputs, but the exact adjudication method (e.g., simple majority, weighted majority, or if a consensus meeting occurred) is not explicitly detailed. However, the use of "majority agreement" for clinical parameters in the clinical study (below) suggests a similar approach might have been used implicitly for the standalone ground truth if not explicitly stated.
View Detection: Ground truth labels were defined on the frame level using annotation of expert sonographers. No specific number of experts or adjudication method is described beyond "expert sonographers."
Probe Guidance: Similar to view detection, ground truth for guidance cues was established by experts, but the exact method or number of experts for adjudication is not detailed.

For Clinical Performance (Pilot & Pivotal Studies):

Adjudication Method: Cardiologists reviewed clips and their assessments were based on majority agreement for visual quality of cardiac parameters. They were blinded to whether the clip was acquired by a non-expert user or a sonographer and to each other's evaluations. Cohen's kappa coefficient was used to assess intra-cardiologist variability.

5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study Was Done, and the Effect Size of How Much Human Readers Improve with AI vs. Without AI Assistance

Yes, a prospective multi-center, multi-case (MRMC) pivotal clinical study was conducted.

Comparison: Non-expert users with UltraSight AI Guidance vs. Cardiac sonographers without AI guidance (using the same hardware).
Effect Size (Improvement with AI for non-expert users): The study demonstrated that non-expert users, with the AI guidance, achieved diagnostic quality scans comparable to those performed by sonographers. For the four co-primary endpoints (LV size, LV function, RV size, and non-trivial pericardial effusion), non-expert users with AI guidance acquired scans deemed to have adequate visual quality in 93-100% of cases (based on majority agreement of expert cardiologists). This indicates a significant improvement for non-expert users, enabling them to produce scans previously only achievable by expert sonographers. While a direct "without AI" performance for non-experts wasn't explicitly tested in this specific pivotal comparative arm (they compared with experts without AI), the entire premise is that without AI guidance, these non-experts would not be able to achieve such diagnostic quality, implying a very large effect size of the AI in bringing non-experts to near-expert performance.

The Pilot Study provides further context: "The exams performed by the non-expert users had sufficient visual quality in 100% of cases based on majority agreement to assess LV size and function, RV size, and pericardial effusion." This further reinforces the high effectiveness.

6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) Was Done

Yes, standalone performance testing of the AI algorithms was conducted. The results for the Quality Bar, View Detection, and Probe Guidance features, with their respective AUC and PPV scores against defined acceptance criteria, are presented under "Non-Clinical Standalone Performance Testing of AI algorithms."

7. The Type of Ground Truth Used

For Standalone AI Performance (Quality Bar): Expert cardiologist annotation ("diagnosable / non-diagnosable") based on ACEP guidelines.
For Standalone AI Performance (View Detection & Probe Guidance): Expert sonographer annotations.
For Clinical Performance (Pilot & Pivotal Studies): Expert cardiologist consensus (majority agreement) on the visual quality for assessing various cardiac parameters. This is effectively expert consensus.

8. The Sample Size for the Training Set

The document notes "Algorithm development" and lists a "number of subjects" and "number of samples." While not explicitly called "training set," these numbers represent the data used for the algorithm's development.

Number of Subjects: 580
Number of Samples: 5 million frames of ultrasound images

9. How the Ground Truth for the Training Set Was Established

The document states that the data used for performance testing (test set) was collected at "different sites, geographically separated, from the sites used for collection of the algorithm development data." This implies that the ground truth for the algorithm development data (training set) would have also been established through similar expert annotations or a process that led to the labels required for training the deep learning models. However, the specific methods for establishing ground truth for the training set are not explicitly described in the provided text. It can be inferred that it involved expert labeling, similar to the test set, but no details on the number of experts, their qualifications, or adjudication methods for the training data are given.

Summary

{0}------------------------------------------------

July 24, 2023

Image /page/0/Picture/1 description: The image contains the logos of the Department of Health and Human Services and the U.S. Food and Drug Administration (FDA). The Department of Health and Human Services logo is on the left, featuring a stylized human figure. To the right is the FDA logo, with the letters "FDA" in a blue square, followed by the words "U.S. FOOD & DRUG ADMINISTRATION" in blue text.

UltraSight Ltd. % Janice Hogan Partner Hogan Lovells US LLP 1735 Market Street, Floor 23 PHILADELPHIA PA 19103

Re: K223347

Trade/Device Name: UltraSight AI Guidance Regulation Number: 21 CFR 892.2100 Regulation Name: Radiological acquisition and/or optimization guidance system Regulatory Class: Class II Product Code: QJU Dated: June 23, 2023 Received: June 23, 2023

Dear Janice Hogan:

We have reviewed your Section 510(k) premarket notification of intent to market the device referenced above and have determined the device is substantially equivalent (for the indications for use stated in the enclosure) to legally marketed predicate devices marketed in interstate commerce prior to May 28, 1976, the enactment date of the Medical Device Amendments, or to devices that have been reclassified in accordance with the provisions of the Federal Food, Drug, and Cosmetic Act (Act) that do not require approval of a premarket approval application (PMA). You may, therefore, market the device, subject to the general controls provisions of the Act. Although this letter refers to your product as a device, please be aware that some cleared products may instead be combination products. The 510(k) Premarket Notification Database located at https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm identifies combination product submissions. The general controls provisions of the Act include requirements for annual registration, listing of devices, good manufacturing practice, labeling, and prohibitions against misbranding and adulteration. Please note: CDRH does not evaluate information related to contract liability warranties. We remind you, however, that device labeling must be truthful and not misleading.

If your device is classified (see above) into either class II (Special Controls) or class III (PMA), it may be subject to additional controls. Existing major regulations affecting your device can be found in the Code of Federal Regulations, Title 21, Parts 800 to 898. In addition, FDA may publish further announcements concerning your device in the Federal Register.

Please be advised that FDA's issuance of a substantial equivalence determination does not mean that FDA has made a determination that your device complies with other requirements of the Act or any Federal statutes and regulations administered by other Federal agencies. You must comply with all the Act's

{1}------------------------------------------------

requirements, including, but not limited to: registration and listing (21 CFR Part 807); labeling (21 CFR Part 801); medical device reporting of medical device-related adverse events) (21 CFR 803) for devices or postmarketing safety reporting (21 CFR 4, Subpart B) for combination products (see https://www.fda.gov/combination-products/guidance-regulatory-information/postmarketing-safety-reportingcombination-products); good manufacturing practice requirements as set forth in the quality systems (OS) regulation (21 CFR Part 820) for devices or current good manufacturing practices (21 CFR 4, Subpart A) for combination products; and, if applicable, the electronic product radiation control provisions (Sections 531-542 of the Act); 21 CFR 1000-1050.

Also, please note the regulation entitled, "Misbranding by reference to premarket notification" (21 CFR Part 807.97). For questions regarding the reporting of adverse events under the MDR regulation (21 CFR Part 803), please go to https://www.fda.gov/medical-device-safety/medical-device-reportingmdr-how-report-medical-device-problems.

For comprehensive regulatory information about mediation-emitting products, including information about labeling regulations, please see Device Advice (https://www.fda.gov/medicaldevices/device-advice-comprehensive-regulatory-assistance) and CDRH Learn (https://www.fda.gov/training-and-continuing-education/cdrh-learn). Additionally, you may contact the Division of Industry and Consumer Education (DICE) to ask a question about a specific regulatory topic. See the DICE website (https://www.fda.gov/medical-device-advice-comprehensive-regulatoryassistance/contact-us-division-industry-and-consumer-education-dice) for more information or contact DICE by email (DICE@fda.hhs.gov) or phone (1-800-638-2041 or 301-796-7100).

Sincerely,

Yanna S. Kang -S

Yanna Kang, Ph.D. Assistant Director Mammography and Ultrasound Team DHT8C: Division of Radiological Imaging and Radiation Therapy Devices OHT8: Office of Radiological Health Office of Product Evaluation and Quality Center for Devices and Radiological Health

Enclosure

{2}------------------------------------------------

Indications for Use

510(k) Number (if known)

K223347

Device Name UltraSight AI Guidance

Indications for Use (Describe)

The UltraSight AI Guidance is intended to assist medical professionals (not including expert sonographers) in acquiring cardiac ultrasound images. UltraSight AI Guidance is an accessory to compatible general-purpose diagnostic ultrasound systems. UltraSight Al Guidance is indicated for use in two-dimensional transthoracic echocardiography (2D- TTE) for adult patients, specifically in the acquisition of the following standard views: Parasternal Long-Axis (PLAX), Parasternal Short-Axis at the Aortic Valve (PSAX-AV), Parasternal Short-Axis at the Mitral Valve (PSAX-MV), Parasternal Short-Axis at the Papillary Muscle (PSAX-PM), Apical 4-Chamber (AP4), Apical 5-Chamber (AP2), Apical 2-Chamber (AP2), Apical 3-Chamber (AP3), Subcostal 4-Chamber (SubC4), and Subcostal Inferior Vena Cava (SC-IVC).

X Prescription Use (Part 21 CFR 801 Subpart D)

Over-The-Counter Use (21 CFR 801 Subpart C)

CONTINUE ON A SEPARATE PAGE IF NEEDED.

This section applies only to requirements of the Paperwork Reduction Act of 1995.

DO NOT SEND YOUR COMPLETED FORM TO THE PRA STAFF EMAIL ADDRESS BELOW.

The burden time for this collection of information is estimated to average 79 hours per response, including the time to review instructions, search existing data sources, gather and maintain the data needed and complete and review the collection of information. Send comments regarding this burden estimate or any other aspect of this information collection, including suggestions for reducing this burden, to:

Department of Health and Human Services Food and Drug Administration Office of Chief Information Officer Paperwork Reduction Act (PRA) Staff PRAStaff(@fda.hhs.gov

"An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB number."

{3}------------------------------------------------

510(k) SUMMARY UltraSight's UltraSight Al Guidance

K223347

Applicant name: UltraSight Ltd.

8 Pinhas Sapir St. Ness Ziona 7403631 Israel Phone: +1-724-648-3514 Email: info@ultrasight.com Contact Person: Davidi Vortman Date Prepared: July 19, 2023

Name of Device: UltraSight AI Guidance

Classification Name: Image Acquisition And/Or Optimization Guided By Artificial Intelligence

Classification Code: QJU

Device class: II

Regulation number: 892.2100

Panel: Radiology

Predicate Devices: Caption Health's Caption Guidance (DEN190040)

Intended Use / Indications for Use

The UltraSight Al Guidance is intended to assist medical professionals (not including expert sonographers) in acquiring cardiac ultrasound images. UltraSight Al Guidance is an accessory to compatible general-purpose diagnostic ultrasound systems. UltraSight Al Guidance is indicated for use in two-dimensional transthoracic echocardiography (2D- TTE) for adult patients, specifically in the acquisition of the following standard views: Parasternal Long-Axis (PLAX), Parasternal Short-Axis at the Aortic Valve (PSAX-AV), Parasternal Short-Axis at the Mitral Valve (PSAX-MV), Parasternal Short-Axis at the Papillary Muscle (PSAX-PM), Apical 4-Chamber (AP4), Apical 5-Chamber (AP5), Apical 2-Chamber (AP2), Apical 3-Chamber (AP3), Subcostal 4-Chamber (SubC4), and Subcostal Inferior Vena Cava (SC-IVC).

Device Description

{4}------------------------------------------------

Main features:

Quality Bar: The system displays an image quality bar that is continuously updated while the user scans the subject, while attempting to find the maximal quality bar is a score for image diagnosability. It represents the classification between high and low quality images, where high quality images are defined as grade 3 or more based on ACEP guidelines (Rachel B. Liu et al., "Emergency Ultrasound Standard Reporting Guidelines", 2018, American College of Emergency Physicians).
· View Detection: The system differentiates between three states, and the display adjusts accordingly:
- No heart heart is not visible, the user is expected to first find the acoustic window 1. where some heart tissue is visible and then follow the navigation instructions.
- Navigate the quality bar is below 8 bars, the heart is visible, the user should watch the 2. guidance cues and move the probe accordingly.
- 1. Hold position the quality bar has reached 8 bars (or more), the tail of the probe turns green, and the user should record a clip. Saving a clip functionality is not part of the UltraSight Al system; the user saves the clip manually using the Lumify interface.
· Probe Guidance: The probe guidance feature provides graphic on-screen instructions for the user to emulate how a sonographer would manipulate the transducer to acquire the target cardiac view. The five possible guidance cues, rotation, tilt, rock, and slides in the lateralmedial and up/down (with respect to the subjects' head) directions.

The guidance user interface is composed of a 3D probe display that shows orientation guidance cues (rotations, tilts and rocks) and a cross that shows slide guidance cues (slides in x and y directions) The user infers the maneuver they should perform from viewing the 3D probe display and the slides cross. No explicit text messages appear on the screen regarding guidance cues.

Technological Characteristics

The UltraSight AI Guidance software application is similar in its technological features to its predicate device, the Caption Health's Caption Guidance (DEN190040). Both systems are intended as an assistive tool to aid medical professionals in the acquisition of cardiac ultrasound images. Specifically, both systems are indicated to aid in the acquisition of the following standard views: Parasternal Long-Axis (PLAX), Parasternal Short-Axis at the Aortic Valve (PSAX-AV), Parasternal Short-Axis at the Mitral Valve (PSAX-MV), Parasternal Short-Axis at the Papillary Muscle (PSAX-PM), Apical 4-Chamber (AP4), Apical 5-Chamber (AP5), Apical 2-Chamber (AP2), Apical 3-Chamber (AP3), Subcostal 4-Chamber (SubC4), and Subcostal Inferior Vena Cava (SC-IVC). Both systems are software applications that apply deep learning-based algorithm to emulate the expertise of a sonographer by providing real-time guidance on how to position and manipulate the transducer on a patient's body. Both systems are installed on a third-party, previously cleared ultrasound device. There are minor technological differences between the subject and the predicate device, mainly the ultrasound device with which the software interfaces and few features offered by the predicate device and not by the subject device (auto capture and save best clip). However, as explained in more detail

{5}------------------------------------------------

below, these differences do not raise new or different questions of safety or effectiveness since the principal technology remains very similar and, in both instances, the key question is whether the device enables the intended users to acquire limited (10-view) 2D transthoracic echocardiography (2D-TTE) in diagnostic quality. The clinical and human factors validation demonstrated that the use of the UltraSight Al Guidance, by users with no prior scanning experience, enabled users to acquire limited (10-view) 2D transthoracic echocardiography (2D-TTE) with similar performance as the predicate device.

A table comparing the key features of the subject and the predicate devices is provided below:

Parameters	Caption GuidanceDEN190040	Proposed UltraSight AlGuidance	Conclusion
Classificationname	Image Acquisition And/ OrOptimization Guided ByArtificial Intelligence	Image Acquisition And/ OrOptimization Guided ByArtificial Intelligence	Identical
Product Code	QJU	QJU	Identical
Intended Use	The Caption Guidancesoftware is intended toassist medicalprofessionals in theacquisition of cardiacultrasound images.Caption Guidancesoftware is an accessoryto compatible generalpurpose diagnosticultrasound systems	The UltraSight AlGuidance software isintended to assist medicalprofessionals in theacquisition of cardiacultrasound images.UltraSight Al Guidancesoftware is an accessoryto compatible generalpurpose diagnosticultrasound systems	Identical
Indications for	Caption Guidance	UltraSight Al Guidance	Similar
use	software is indicated foruse in two-dimensionaltransthoracicechocardiography (2D-TTE) for adult patients,specifically in theacquisition of the followingstandard views:Parasternal Long-Axis(PLAX), ParasternalShort-Axis at the AorticValve (PSAX-AV),Parasternal Short-Axis atthe Mitral Valve (PSAX-MV), Parasternal Short-Axis at the PapillaryMuscle (PSAX-PM),Apical 4-Chamber (AP4)	software is indicated foruse in two-dimensionaltransthoracicechocardiography (2D-TTE) for adult patients,specifically in theacquisition of the followingstandard views:Parasternal Long-Axis(PLAX), ParasternalShort-Axis at the AorticValve (PSAX-AV),Parasternal Short-Axis atthe Mitral Valve (PSAX-MV), Parasternal Short-Axis at the PapillaryMuscle (PSAX-PM),Apical 4-Chamber (AP4)	Since the use of theUltraSight Al Guidanceand its impact to expertsonographers has notbeen evaluated, thisusers' population isexcluded from the device'sintended users.
Parameters	Caption GuidanceDEN190040	Proposed UltraSight AlGuidance	Conclusion
	Apical 2-Chamber (AP2),Apical 3-Chamber (AP3),Subcostal 4-Chamber(SubC4), and SubcostalInferior Vena Cava (SC-IVC).	Apical 2-Chamber (AP2),Apical 3-Chamber (AP3),Subcostal 4-Chamber(SubC4), and SubcostalInferior Vena Cava (SC-IVC).The UltraSight AlGuidance is not intendedto be used bySonographers.
Intended user	Medical professionals(including expertsonographers)	Medical professionals (notincluding expertsonographers)	Substantially Equivalent:It should be noted thatthe intended users'population of theUltraSight Al Guidance isslightly narrower than thepredicate and does notinclude sonographers.However, the intendeduser population of theUltraSight Al Guidance isa subset of the broaderuser population for thepredicate; thus, theintended use andindications are fullyencompassed within thepredicate indications.Therefore, the firstcriterion for a finding ofsubstantial equivalenceis satisfied.
CompatibleUltrasoundsystem andprobe	The uSmart 3200t Plusultrasound system with the3200t-compatible Terason4V2A linear phased arrayprobe	Philips Lumify With S4-1probe and Samsung TabS6 and S7 tablets	Substantially Equivalent:Both the predicate deviceand the proposed devicefunction as a softwareaccessory to a cleared tomarket ultrasound device.Both ultrasound devicesare cleared for thediagnostic ultrasoundclinical application ofacquiring cardiac images
Parameters	Caption GuidanceDEN190040	Proposed UltraSight AlGuidance	Conclusion
			of adult patients. Sinceboth ultrasound devicesare cleared for therequired clinical use, theuse of different ultrasounddevices does not raisenew questions of safetyand/or effectiveness.
Clinical Features
Imageacquisitionguidance	Prescriptive Guidance:The prescriptive guidancefeature in CaptionGuidance providesdirection to the user toemulate how asonographer wouldmanipulate the transducerto acquire the optimal view	Probe Guidance: Theprobe guidance featureprovides graphic on-screen instructions forthe user to emulate howa sonographer wouldmanipulate thetransducer to acquire thetarget cardiac view.	Substantially Equivalent:While the user interface isslightly different, thefunctionality and type ofguidance provided to theuser will be the same.Specifically, both devicesprovide to the userinstructions on how tomanipulate the probe(translational, tilt androtation guidance cues)together with real-timefeedback on theexpected diagnosticquality of the resulting clipto direct the user to aprobe position that willenable acquisition of adiagnostic quality clip.Hence, this differencedoes not raise newquestions of safety and/oreffectiveness.
Real timefeedback onimage quality	Quality Meter: a real-timefeedback from the QualityMeter advises the user onthe expected diagnosticquality of the resulting clip,such that the user canmake decisions to furtheroptimize the quality, forexample by following theprescriptive guidance	Quality Bar: a real-timequality bar providingfeedback on the clip'squality advises the user onthe expected diagnosticquality of the currentimage. This informationcan be used by the usersto assess how close theyare to capturing a	Substantially Equivalent
Parameters	Caption GuidanceDEN190040	Proposed UltraSight AlGuidance	Conclusion

Automaticcapture of clipswith predicteddiagnosticquality	Auto-Capture: TheCaption Guidance Auto-Capture feature triggersan automatic capture of aclip when the quality ispredicted to be diagnostic,emulating the way inwhich a sonographerknows when an image isof sufficient quality to bediagnostic and records it.	None	Substantially Equivalent:This feature is notprovided; the UltraSightsoftware provides to theuser the view detectionand quality bar featureswhich enable any user toclearly identify when adiagnostic quality imagehas been achieved and tomanually save the clip.
Retrospectivelyrecording ofhighest qualityclip	Save Best Clip: Thisfeature continuallyassesses clip quality whilethe user is scanning and,in the event that the useris not able to obtain a clipsufficient for Auto-Capture, the softwareallows the user toretrospectively record thehighest quality clipobtained so far, mimickingthe choice a sonographermight make whenrecording an exam.	None	Substantially Equivalent:Although this feature is notprovided, theUltraSight softwareprovides to the user theview detection and qualitybar features which enableany user to clearly identifywhen a diagnostic qualityhas been achieved and tomanually save the clip.
Deep LearningBasedAlgorithm	Yes	Yes	Substantially Equivalent

{6}------------------------------------------------

{7}------------------------------------------------

{8}------------------------------------------------

Substantial Equivalence

The UltraSight AI Guidance is as safe and effective as the Caption Health's Caption Guidance. The UltraSight Al Guidance has the same intended uses and indications for use and similar technological characteristics, and principles of operation as its predicate device. The minor technological differences between the UltraSight Al Guidance and its predicate device raise no new issues of safety or effectiveness. Performance data demonstrate that the UltraSight Al Guidance is as safe and effective as Caption Health's Caption Guidance. Thus, UltraSight Al Guidance is substantially equivalent to Caption Health's Caption Guidance.

{9}------------------------------------------------

Non-Clinical Performance Data

The following testing was conducted to evaluate the device:

1. Software moderate level of concern verification and validation testing was conducted as required by IEC 62304 and FDA Guidance on General Principles of Software Validation, January 11, 2002.
1. UltraSight AI Guidance Algorithms' standalone performance was validated with a number of standalone performance tests, see detailed description below.
1. Human factors Validation: the system was validated with intended users in simulated use conditions to ensure the user needs and intended use requirements were met. All tasks, including a critical task, were met and no new issues of safety or effectiveness were raised.

Non-Clinical Standalone Performance Testing of AI algorithms

The Datasets used for algorithm development and performance testing are listed in Table 1:

Table 1: Data used for Algorithm Development and Performance Testing

Phase	Algorithm development	Performance tests for Quality Bar	Performance test for ViewDetection and Guidance tests
Number ofsubjects	580	312	75
Number ofsamples	5 million framesof ultrasound images	5,800 clips	2.3 million framesof ultrasound images

The data used for performance testing was collected at different sites, geographically separated, from the sites used for collection of the algorithm development data. The data used in the performance testing was collected from a population representative to the intended population.

Performance Tests Overview

The following features of the system were tested:

Quality bar .
View detection (part of the quality bar functionality as explained below) ●
Probe Guidance ●

Each feature of the system (Quality Bar, View Detection, Probe Guidance) was tested independently.

{10}------------------------------------------------

Quality Bar:

The quality bar is a score for image diagnosability. It represents the classification between high- and low-quality images, where high quality images are defined as grade 3 or more based on ACEP quidelines.

The test was performed on 312 clips. Each clip in the dataset is associated with a target cardiac view and was annotated with a clip diagnosability label (diagnosable / non-diagnosable), by three cardiologists.

The test assessed the classification performance between "diagnosable" clips of each view. The mean AUC was 0.86 with 95% Cl [0.85, 0.87] showing good classification performance, relative to the success criteria of AUC > 0.8. The mean PPV was 0.93 with 95% CI [0.92, 0.94] relative to the success criteria of PPV > 0.75, showing good classification performance. Stratified analysis was performed, for each target cardiac view the AUC was greater than 0.8, and the PPV was above 0.75.

View Detection:

The system provides view detection classification that distinguishes between three system states: "Hold Position", "Navigate", and "No Heart". The test was performed on 75 subjects on a total of 2.3 million frames of ultrasound images.

The view detection classification performance is quantified for the classification tasks of "Hold position" vs. "Navigate" and "Hold position" vs. "No heart". The ground truth labels were defined on the frame level using annotation of expert sonographers. The mean AUC was 0.988 with 95% Cl (0.985, 0.990) showing good classification performance, relative to the success criteria of AUC > 0.8. Stratified analysis was performed and showed that for each individual binary classification test the AUC was above 0.8.

Probe Guidance:

The system provides guidance cues when the heart is visible on the ultrasound image. To provide the correct guidance cues, the system needs to be able to identify the position and orientation of the probe relative to the target cardiac view. There are five possible guidance cues: rotations, tilts, rocks, and slides in the x (lateral-medial) and y (superior-inferior) directions.

The testing was performed on 75 subjects on a total of 2.3 million frames of ultrasound images. The testing evaluated the frame level accuracy of each guidance cue prediction. For each quidance cue, a series of tests was defined to check if the guidance cue is functioning correctly in a particular direction (for example, clockwise rotative to 2C view, tilts up from PLAX). Each test was formulated as a binary classification test.

The mean AUC was 0.821 with 95% Cl [0.813, 0.827] showing good classification performance, relative to the success criteria of AUC > 0.8. Stratified tests show that individual classifiers have reasonable discrimination ability, and acceptable classifiers.

{11}------------------------------------------------

Human factors Validation:

The UltraSight AI Guide application underwent summative usability validation in accordance with the FDA Guidance Document, "Applying Human Factors and Usability Engineering to Medical Devices" (2016). All user tasks associated with the device were evaluated and critical tasks were identified for usability evaluation, where if performed incorrectly or not performed at all would or could cause serious harm. All participants successfully completed the testing sessions and performed the critical task identified in the study without any use errors, close calls, or other issues.

Based on the results of the summative usability validation, it can be concluded that the UltraSight Al Guide application is deemed adequately safe and effective for its intended users, uses, and use environments.

Clinical Performance Data

Pilot Study

A prospective single center, multi-case (MRMC) study, primarily designed to obtain the quality of the clips obtained by medical professionals without specialized echocardiography training (non-expert users) when using the Ultrasound Al was conducted. Sixty-one (61) subjects were scanned twice, once by a non-expert user, using Philips Lumify with the UltraSight Al guidance; and, by a cardiac sonographer using the Philips Lumify hardware but without Al guidance. The clips acquired during those scans were reviewed by a panel of 5 expert cardiologists blinded to whether the clip was acquired by a non-expert user or a sonographer and to each other's evaluations. The cardiologists assessed whether the non-expert users' scans were of sufficient quality for the expert readers to make qualitative visual assessment of left ventricular (LV) size, LV function, right ventricular (RV) size, and the presence of nontrivial pericardial effusion. Six additional echocardiographic parameters were visually assessed as secondary endpoints: qualitative assessment of RV function; left atrium size; structural assessment of the aortic, mitral, and tricuspid valves; and qualitative assessment of IVC size.

The exams performed by the non-expert users had sufficient visual quality in 100% of cases based on majority agreement to assess LV size and function, RV size, and pericardial effusion. For the secondary endpoints, the exams performed by non-expert users had sufficient visual quality based on majority in 90-100% of cases to assess RV function, left atrium size, AV structure, MV structure, TV structure, and IVC size. Furthermore, the exams performed by non-expert users had sufficient diagnostic quality based on majority in 80% to 100% of cases to assess PLAX, PSAX MV, PSAX PM, AP2, AP3, AP5, Sub4C and SC-IVC. In addition, most clips (range 78.6-97.9%) taken by the non-expert users had a diagnostic quality score of 3 of higher based on American College of Emergency Physicians (ACEP) scale.

Pivotal Study

A prospective multi-center, multi-case (MRMC) pivotal clinical study was conducted to evaluate the use of UltraSight Al Guidance by medical professionals without specialized echocardiography training (non-expert/novice users).

{12}------------------------------------------------

A total of 240 subjects were scanned twice, once by a non-expert user, using the Philips Lumify with the UltraSight AI guidance; and, by a cardiac sonographer using the Philips Lumify hardware but without AI quidance. The clips acquired during the scans were reviewed by a panel of 5 expert cardiologists blinded to whether the clip was acquired by a non-expert user or a sonographer and to each other's evaluations. The cardiologists assessed whether the non-expert users' scans were of sufficient quality for the expert readers to make qualitative visual assessment of left ventricular (LV) size, LV function, right ventricular (RV) size, and the presence of nontrivial pericardial effusion. Six additional echocardiographic parameters were visually assessed as secondary endpoints: qualitative assessment of RV function; left atrium size; structural assessment of the aortic, mitral, and tricuspid valves; and qualitative assessment of IVC size. For the 4 co-primary endpoints, the non-expert users' scans were judged to have adequate visual quality on the majority agreement to assess the clinical parameters in nearly all subjects, in 93-100% of cases. For the secondary endpoints, the percentage of exams performed by non-expert users that had sufficient visual quality based on majority agreement was 98% for MV structure, 94% for RV function and left atrium size, 89% for AV structure, 74% for TV structure, and 67% for IVC size. Assessment of intra-cardiologists' variability using Cohen's kappa coefficient (k) was assessed on a randomly selected 10% of the examinations on which a repeated assessment was performed. There was very good agreement among the cardiologists when assessing the primary endpoints. The results of the study showed that exams performed by non-expert users using the UltraSight Al Guidance Software can provide clips with sufficient diagnostic quality even immediately upon completion of training.

Conclusions

The UltraSight Al Guidance is substantially equivalent to Caption Guidance. The UltraSight Al Guidance has the same intended use and indications for use and similar technological characteristics, and principles of operation as its predicate device. The minor technological differences between the UltraSight Al Guidance and its predicate device raise no new issues of safety or effectiveness. Non-clinical and clinical performance data demonstrate that the UltraSight Al Guidance performs as expected and in a manner that is substantially equivalent to its predicate device. Thus, the UltraSight Al Guidance is substantially equivalent.

Regulation Number and Section

892.2100 Radiological acquisition and/or optimization guidance system.

(a)
Identification. A radiological acquisition and/or optimization guidance system is a device that is intended to aid in the acquisition and/or optimization of images and/or diagnostic signals. The device interfaces with the acquisition system, analyzes its output, and provides guidance and/or feedback to the operator for improving image and/or signal quality.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Design verification and validation must include:
(i) A detailed, technical device description, including a detailed description of the impact of any software and hardware on the device's functions, the associated capabilities and limitations of each part, and the associated inputs and outputs.
(ii) A detailed, technical report on the non-clinical performance testing of the subject device in the intended use environments, using relevant consensus standards when applicable.
(iii) A detailed report on the clinical performance testing, obtained from either clinical testing, accepted virtual/physical systems designed to capture clinical variability, comparison to a closely-related device with established clinical performance, or other sources that are justified appropriately. The choice of the method must be justified given the risk of the device and the general acceptance of the test methods. The report must include the following:
(A) A thorough description of the testing protocol(s).
(B) A thorough, quantitative evaluation of the diagnostic utility and quality of images/data acquired, or optimized, using the device.
(C) A thorough, quantitative evaluation of the performance in a representative user population and patient population, under anticipated conditions and environments of use.
(D) A thorough discussion on the generalizability of the clinical performance testing results.
(E) A thorough discussion on use-related risk analysis/human factors data.
(iv) A detailed protocol that describes, in the event of a future change, the level of change in the device technical specifications or indications for use at which the change or changes could significantly affect the safety or effectiveness of the device and the risks posed by these changes. The assessment metrics, acceptance criteria, and analytical methods used for the performance testing of changes that are within the scope of the protocol must be included.
(v) Documentation of an appropriate training program, including instructions on how to acquire and process quality images and video clips, and a report on usability testing demonstrating the effectiveness of that training program on user performance, including acquiring and processing quality images.
(2) The labeling required under § 801.109(c) of this chapter must include:
(i) A detailed description of the device, including information on all required and/or compatible parts.
(ii) A detailed description of the patient population for which the device is indicated for use.
(iii) A detailed description of the intended user population, and the recommended user training.
(iv) Detailed instructions for use, including the information provided in the training program used to meet the requirements of paragraph (b)(1)(iv) of this section.
(v) A warning that the images and data acquired using the device are to be interpreted only by qualified medical professionals.
(vi) A detailed summary of the reports required under paragraphs (b)(1)(ii) and (iii) of this section.
(vii) A statement on upholding the As Low As Reasonably Achievable (ALARA) principle with a discussion on the associated device controls/options.