(25 days)
Fetal EchoScan is a machine learning-based computer-assisted diagnosis (CADx) software device indicated as an adjunct to fetal heart ultrasound examination in pregnant women aged 18 or older undergoing second-trimester anatomic ultrasound exams.
When utilized by an interpreting physician, Fetal EchoScan provides information regarding the presence of any of the following suspicious radiographic findings:
- overriding artery
- septal defect at the cardiac crux
- abnormal relationship of the outflow tracts
- enlarged cardiothoracic ratio
- right ventricular to left ventricular size discrepancy
- tricuspid valve to mitral valve annular size discrepancy
- pulmonary valve to aortic valve annular size discrepancy
- cardiac axis deviation
Fetal EchoScan is to be used with cardiac fetal ultrasound video clips containing interpretable 4-chamber, left ventricular outflow tract, right ventricular outflow tract standard views.
Fetal EchoScan is intended for use as a concurrent reading aid for interpreting physicians (OB-GYN, MFM). It does not replace the role of the physician or of other diagnostic testing in the standard of care. When utilized by an interpreting physician, this device provides information that may be useful in rendering an accurate diagnosis regarding the potential presence of morphological abnormalities that might be suggestive of fetal congenital heart defects that may be useful in determining the need for additional exams.
Fetal EchoScan is not intended for use in multiple pregnancies, cases of heterotaxy and postnatal ultrasound exams.
Fetal EchoScan is a cloud-based software-only device which uses neural networks to detect suspicious cardiac radiographic findings for further review by trained and qualified physicians. Fetal EchoScan is intended to be used as an adjunct to the interpretation of the second-trimester fetal anatomic ultrasound exam performed between 18 and 24 weeks of gestation, for pregnant women aged 18 or more.
Here's a detailed breakdown of the acceptance criteria and the study that proves the device meets them, based on the provided FDA 510(k) Clearance Letter for Fetal EchoScan v1.1:
Acceptance Criteria and Device Performance
1. Table of Acceptance Criteria and Reported Device Performance
The document does not explicitly state "acceptance criteria" but rather presents the performance metrics achieved by the device in both standalone and reader studies. The implication is that these performance levels were deemed acceptable for clearance.
Table 1. Standalone Performance of Fetal EchoScan v1.1 for all suspicious radiographic findings Combined
| Metric | Acceptance Criteria (Implied) | Reported Device Performance (Worst-Case Sensitivity, Best-Case Specificity) | Reported Device Performance (Best-Case Sensitivity, Worst-Case Specificity) |
|---|---|---|---|
| Sensitivity for any suspicious findings | High (not numerically specified) | 0.977 (95% CI, 0.954 ; 0.989) | 0.987 (95% CI, 0.967 ; 0.995) |
| Specificity for any suspicious findings | High (not numerically specified) | 0.977 (95% CI, 0.961 ; 0.987) | 0.963 (95% CI, 0.944 ; 0.976) |
| Conclusive Output Rate | High (not numerically specified) | 98.8% (95% CI, 97.8 ; 99.3) | 98.8% (95% CI, 97.8 ; 99.3) |
Table 2. Reader Study Performance of Fetal EchoScan v1.1 for all suspicious radiographic findings Combined
| Metric | Acceptance Criteria (Implied) | Reported Device Performance (AI-Aided) | Reported Device Performance (Unaided) | Improvement (AI-Aided vs. Unaided) | DBM-OR p-value |
|---|---|---|---|---|---|
| ROC AUC for any suspicious findings | Significantly higher with aid | 0.974 (95% CI 0.957-0.990) | 0.825 (95% CI 0.741-0.908) | +0.149 (14.9%) | 0.002 |
| Mean Sensitivity for any suspicious findings | Improved with aid | 0.935 (95% CI 0.892-0.978) | 0.782 (95% CI 0.686-0.878) | +0.153 (15.3%) | Not explicitly stated for sensitivity/specificity |
| Mean Specificity for any suspicious findings | Improved with aid | 0.970 (95% CI 0.949-0.991) | 0.759 (95% CI 0.630-0.887) | +0.211 (21.1%) | Not explicitly stated for sensitivity/specificity |
Note: The numerical acceptance criteria for "high sensitivity" and "high specificity" are not explicitly defined in the provided document, but the reported performance values surpassed what was considered acceptable by the FDA for substantial equivalence.
2. Sample Size Used for the Test Set and Data Provenance
- Test Set Sample Size (Standalone Testing): 877 clinically acquired fetal ultrasound exams.
- Test Set Sample Size (Reader Study): 200 exams.
- Data Provenance:
- Country of Origin: U.S.A. and France.
- Retrospective or Prospective: The document doesn't explicitly state whether the data was retrospective or prospective, but it mentions "clinically acquired" exams, which often implies retrospective use of existing data.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications
- Number of Experts: Three (3) pediatric cardiologists.
- Qualifications of Experts: Pediatric cardiologists. No further details on years of experience or board certification are provided.
4. Adjudication Method for the Test Set
- Adjudication Method: Majority voting among the three pediatric cardiologists.
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
- Was an MRMC study done? Yes.
- Effect Size of Human Readers' Improvement with AI vs. without AI assistance:
- ROC AUC: Humans improved by +14.9% (from 0.825 unaided to 0.974 aided), with a p-value of 0.002.
- Mean Sensitivity: Humans improved by +15.3% (from 0.782 unaided to 0.935 aided).
- Mean Specificity: Humans improved by +21.1% (from 0.759 unaided to 0.970 aided).
6. Standalone Performance Study
- Was a standalone study done? Yes.
- Performance Metrics: Refer to Table 1 above. The AI system had a conclusive output rate of 98.8%. Sensitivity ranged from 0.977 to 0.987, and Specificity ranged from 0.963 to 0.977 for the detection of any suspicious findings, depending on how inconclusive outputs were treated.
7. Type of Ground Truth Used
- Ground Truth Type: Expert consensus. Specifically, it was derived from a "truthing process in which three pediatric cardiologists assessed the presence or absence of each of the eight findings, and majority voting was used." This constitutes expert consensus.
8. Sample Size for the Training Set
- The document states: "The ultrasound examinations used for training and validation are entirely distinct from the examinations used in standalone testing." However, the specific sample size for the training set is not provided in the clearance letter. It only mentions that the data used for standalone testing (877 exams) and the reader study (200 exams) were distinct from the training and validation data.
9. How the Ground Truth for the Training Set Was Established
- The document states: "The ultrasound examinations used for training and validation are entirely distinct from the examinations used in standalone testing." However, the methodology for establishing ground truth for the training set is not explicitly detailed in the provided text. It can be inferred that a similar expert review process would have been used, but no specific details are given.
§ 892.2060 Radiological computer-assisted diagnostic software for lesions suspicious of cancer.
(a)
Identification. A radiological computer-assisted diagnostic software for lesions suspicious of cancer is an image processing prescription device intended to aid in the characterization of lesions as suspicious for cancer identified on acquired medical images such as magnetic resonance, mammography, radiography, or computed tomography. The device characterizes lesions based on features or information extracted from the images and provides information about the lesion(s) to the user. Diagnostic and patient management decisions are made by the clinical user.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Design verification and validation must include:
(i) A detailed description of the image analysis algorithms including, but not limited to, a detailed description of the algorithm inputs and outputs, each major component or block, and algorithm limitations.
(ii) A detailed description of pre-specified performance testing protocols and dataset(s) used to assess whether the device will improve reader performance as intended.
(iii) Results from performance testing protocols that demonstrate that the device improves reader performance in the intended use population when used in accordance with the instructions for use. The performance assessment must be based on appropriate diagnostic accuracy measures (
e.g., receiver operator characteristic plot, sensitivity, specificity, predictive value, and diagnostic likelihood ratio). The test dataset must contain sufficient numbers of cases from important cohorts (e.g., subsets defined by clinically relevant confounders, effect modifiers, concomitant diseases, and subsets defined by image acquisition characteristics) such that the performance estimates and confidence intervals of the device for these individual subsets can be characterized for the intended use population and imaging equipment.(iv) Standalone performance testing protocols and results of the device.
(v) Appropriate software documentation (
e.g., device hazard analysis; software requirements specification document; software design specification document; traceability analysis; and description of verification and validation activities including system level test protocol, pass/fail criteria, results, and cybersecurity).(2) Labeling must include:
(i) A detailed description of the patient population for which the device is indicated for use.
(ii) A detailed description of the intended reading protocol.
(iii) A detailed description of the intended user and recommended user training.
(iv) A detailed description of the device inputs and outputs.
(v) A detailed description of compatible imaging hardware and imaging protocols.
(vi) Warnings, precautions, and limitations, including situations in which the device may fail or may not operate at its expected performance level (
e.g., poor image quality or for certain subpopulations), as applicable.(vii) Detailed instructions for use.
(viii) A detailed summary of the performance testing, including: Test methods, dataset characteristics, results, and a summary of sub-analyses on case distributions stratified by relevant confounders (
e.g., lesion and organ characteristics, disease stages, and imaging equipment).