(119 days)
FETOLY-HEART is intended to analyse fetal ultrasound images and clips using machine learning techniques to automatically detect heart views and quality criteria within the views. The device is intended for use as a concurrent reading aid during the acquisition and interpretation of fetal ultrasound images.
FETOLY-HEART is indicated for use during routine fetal heart examination of 2nd and 3rd trimester pregnancy (gestational age: from 17 to 40 weeks).
FETOLY-HEART is a software that aims at helping sonographers, obstetricians, radiologists, maternal-fetal medicine specialists, and pediatric cardiologists (designated as healthcare professionals i.e. HCPs) to perform fetal ultrasound examinations of the fetal heart in real-time. FETOLY-HEART can be used by HCPs during fetal ultrasound examinations in the second and third trimesters (gestational age window: from 17 to 40 weeks). The software is intended to assist HCPs in the completeness assessment of the fetal heart ultrasound examination in accordance with national and international guidelines.
To utilize FETOLY-HEART, the software needs to be installed on a hardware device which is connected to an Ultrasound Machine through an HDMI connection. The software receives ultrasound images captured by the connected Ultrasound Machine in real-time. The software's frozen deep learning algorithm, which was trained by supervised learning, analyzes images of this ultrasound image stream to detect heart views and quality criteria within those views. The software provides the following user-accessible information:
- . Examination completeness: the software displays in real-time which heart views and quality criteria are verified by the software during the examination. It is the main and principal output of the FETOLY-HEART device. The verified heart views and quality criteria are accessible by clinicians at any moment of the ultrasound examination, in real-time.
- . Completeness illustration: the software selects an image subset that illustrates the verified views and quality criteria. These images can be reviewed by clinicians to verify the views and criteria's presence. This is a secondary output of the FETOLY-HEART device. Optionally, clinicians can display detected quality criteria localization on selected images.
Here's a breakdown of the acceptance criteria and the study proving FETOLY-HEART meets them, based on the provided FDA 510(k) summary:
Acceptance Criteria and Performance Study for FETOLY-HEART
The FETOLY-HEART device uses machine learning to automatically detect fetal heart views and quality criteria within those views from ultrasound images. The acceptance criteria focus on the device's accuracy in these detections, measured by sensitivity, specificity, and mean Intersection over Union (mIoU) for bounding box localization.
1. Table of Acceptance Criteria and Reported Device Performance
| Measured Metric | Acceptance Criteria | Reported Device Performance (Point Estimate) | Bootstrap CI (95%) |
|---|---|---|---|
| Fetal Heart View Detection | |||
| Sensitivity (for each view) | ≥ 85% | Abdomen: 0.976 | |
| Four Chamber: 0.987 | |||
| LVOT: 0.983 | |||
| RVOT: 0.987 | |||
| Three Vessels: 0.981 | Abdomen: (0.960, 0.990) | ||
| Four Chamber: (0.974, 0.997) | |||
| LVOT: (0.969, 0.994) | |||
| RVOT: (0.974, 0.996) | |||
| Three Vessels: (0.965, 0.993) | |||
| Specificity (for each view) | ≥ 85% | Abdomen: 0.998 | |
| Four Chamber: 1.00 | |||
| LVOT: 0.999 | |||
| RVOT: 0.998 | |||
| Three Vessels: 0.998 | Abdomen: (0.996, 1.000) | ||
| Four Chamber: (1.000, 1.000) | |||
| LVOT: (0.998, 1.000) | |||
| RVOT: (0.996, 1.000) | |||
| Three Vessels: (0.997, 1.000) | |||
| Quality Criteria Detection | |||
| Sensitivity | ≥ 90% | Ranges from 0.903 (Abdomen - Left rib) to 0.990 (Four Chamber - Left atrium) | (All reported lower bounds of CI for sensitivity met ≥ 0.85 acceptance criteria for views, and ≥ 0.90 for quality criteria.) |
| Specificity | ≥ 90% | Ranges from 0.990 (Four Chamber - Connection between crux and atrial septum) to 1.00 (Four Chamber - Right atrium/Foramen ovale flap) | (All reported lower bounds of CI for specificity met ≥ 0.85 acceptance criteria for views, and ≥ 0.90 for quality criteria.) |
| Quality Criteria Bounding Box Localization | |||
| Mean Intersection over Union (mIoU) | ≥ 50% | Values range from 0.512 (Abdomen - Inferior vena cava) to 0.792 (Three Vessels - Spine) | (All reported lower bounds of CI for mIoU met ≥ 0.50 acceptance criteria.) |
Note: For the detailed range of sensitivities, specificities, and mIoU for each of the 52 quality criteria, refer to the tables provided in the original document (pages 15-17).
2. Sample Size and Data Provenance
- Test Set Sample Size: 2,288 fetal ultrasound images across 480 patient cases.
- Data Provenance: The data originated from 7 distinct clinical sites in the United States. The data was collected retrospectively in reverse chronological order. It includes full examination still images, cardiac clip frames, and full examination video frames. The cases are stated to be representative of the intended use population.
3. Number of Experts and Qualifications for Ground Truth
- Number of Experts: Six annotators and additional adjudicators.
- Qualifications of Experts: 3 sonographers and 3 OB/GYN doctors. Specific experience levels (e.g., "10 years of experience") are not provided, but their professional titles indicate clinical expertise in relevant fields.
4. Adjudication Method for the Test Set
- View Classification: A 2+1 ground truth procedure was used.
- Images were assigned to pairs of annotators.
- If the two annotators agreed on the view classification, that was considered the ground truth.
- If the pair of annotators disagreed, an adjudicator reviewed the images and made the final decision.
- Quality Criteria Classification and Localization (Bounding Boxes):
- Each image was annotated by a pair of annotators who drew bounding boxes.
- Agreement on Localization: If their bounding boxes had at least 50% overlap, their coordinates were averaged to form the ground truth.
- Disagreement on Presence or Localization: If the overlap was lower or there was a disagreement on the criterion's presence, an adjudicator reviewed the boxes.
- The final decision regarding the presence of the criterion was based on majority consensus among the adjudicator and annotators.
- The final decision for the criteria localization (bounding box) was based on the adjudicator's decision to either keep one of the annotator's boxes or draw a new one.
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
- The document does not indicate that an MRMC comparative effectiveness study was done to evaluate how much human readers improve with AI vs. without AI assistance. The study described is a standalone performance test of the algorithm itself.
6. Standalone Performance (Algorithm Only) Study
- Yes, a standalone performance study was conducted. The results presented in the tables (sensitivity, specificity, mIoU) are for the FETOLY-HEART algorithm's performance without integration into a human reading workflow or human-in-the-loop performance measurement.
7. Type of Ground Truth Used
- The ground truth used was expert consensus based on a 2+1 adjudication method by a panel of sonographers and OB/GYN doctors. For quality criteria localization, it involved expert-drawn bounding boxes with an adjudication process.
8. Sample Size for the Training Set
- The document states that the testing dataset originated from distinct clinical sites from which the data used during model development (training/validation) was sourced, ensuring testing independence. However, the sample size for the training set is not explicitly provided in the given text.
9. How the Ground Truth for the Training Set Was Established
- The document states, "The software's frozen deep learning algorithm, which was trained by supervised learning...". While it confirms supervised learning was used (implying labeled data), it does not explicitly detail the method for establishing ground truth for the training set. It only describes the ground truth establishment for the test set. It is common for ground truth for training data to be established by experts, potentially through similar consensus or adjudication processes, but this specific document does not describe it for the training set.
§ 892.1550 Ultrasonic pulsed doppler imaging system.
(a)
Identification. An ultrasonic pulsed doppler imaging system is a device that combines the features of continuous wave doppler-effect technology with pulsed-echo effect technology and is intended to determine stationary body tissue characteristics, such as depth or location of tissue interfaces or dynamic tissue characteristics such as velocity of blood or tissue motion. This generic type of device may include signal analysis and display equipment, patient and equipment supports, component parts, and accessories.(b)
Classification. Class II.