(99 days)
Fetal EchoScan is a machine learning-based computer-assisted diagnosis (CADx) software device indicated as an adjunct to fetal heart ultrasound examination in pregnant women aged 18 or older undergoing second-trimester anatomic ultrasound exams.
When utilized by an interpreting physician, Fetal EchoScan provides information regarding the presence of any of the following suspicious radiographic findings:
- overriding artery
- septal defect at the cardiac crux
- abnormal relationship of the outflow tracts
- enlarged cardiothoracic ratio
- right ventricular to left ventricular size discrepancy
- tricuspid valve to mitral valve annular size discrepancy
- pulmonary valve to aortic valve annular size discrepancy
- cardiac axis deviation
Fetal EchoScan is to be used with cardiac fetal ultrasound video clips containing interpretable 4-chamber, left ventricular outflow tract, right ventricular outflow tract standard views.
Fetal EchoScan is intended for use as a concurrent reading aid for interpreting physicians (OB-GYN, MFM). It does not replace the role of the physician or of other diagnostic testing in the standard of care. When utilized by an interpreting physician, this device provides information that may be useful in rendering an accurate diagnosis regarding the potential presence of morphological abnormalities that might be suggestive of fetal congenital heart defects that may be useful in determining the need for additional exams.
Fetal EchoScan is not intended for use in multiple pregnancies, cases of heterotaxy, and postnatal ultrasound exams.
Fetal EchoScan is a cloud-based software-only device which uses neural networks to detect suspicious cardiac radiographic findings for further review by trained and qualified physicians. Fetal EchoScan is intended to be used as an adjunct to the interpretation of the second-trimester fetal anatomic ultrasound exam performed between 18 and 24 weeks of gestation, for pregnant women aged 18 or more.
Here's a breakdown of the acceptance criteria and study details for the Fetal EchoScan device, based on the provided document:
1. Table of Acceptance Criteria and Reported Device Performance
The document does not explicitly state "acceptance criteria" as a set of predefined thresholds. Instead, it presents the performance of the device in various metrics and then concludes that these results demonstrate substantial equivalence. For the purpose of this request, I will infer the implied acceptance criteria from the reported performance and the conclusion of substantial equivalence.
Inferred Acceptance Criteria & Reported Device Performance
| Metric / Finding | Inferred Acceptance Criteria (Implicit from conclusion of substantial equivalence) | Fetal EchoScan Performance (Worst-Case Sensitivity / Best-Case Specificity) | Fetal EchoScan Performance (Best-Case Sensitivity / Worst-Case Specificity) | Aided Reader Performance (ROC AUC) |
|---|---|---|---|---|
| Standalone Performance | ||||
| Any suspicious findings | High Sensitivity & High Specificity | Sensitivity: 0.977 (0.954-0.989) | ||
| Specificity: 0.977 (0.961-0.987) | Sensitivity: 0.987 (0.967-0.995) | |||
| Specificity: 0.963 (0.944-0.976) | N/A | |||
| Overriding artery | High Sensitivity & High Specificity | Sensitivity: 0.894 (0.820-0.940) | ||
| Specificity: 0.989 (0.977-0.995) | Sensitivity: 0.942 (0.880-0.973) | |||
| Specificity: 0.979 (0.963-0.988) | 0.953 (0.916-0.990) | |||
| Cardiac crux septal defect | High Sensitivity & High Specificity | Sensitivity: 0.905 (0.823-0.951) | ||
| Specificity: 0.995 (0.985-0.998) | Sensitivity: 0.917 (0.838-0.959) | |||
| Specificity: 0.989 (0.977-0.995) | 0.971 (0.943-0.999) | |||
| Abnormal OT relationship | High Sensitivity & High Specificity | Sensitivity: 0.869 (0.781-0.925) | ||
| Specificity: 0.991 (0.979-0.996) | Sensitivity: 0.952 (0.884-0.981) | |||
| Specificity: 0.989 (0.977-0.995) | 0.972 (0.953-0.992) | |||
| Enlarged CTR | High Sensitivity & High Specificity | Sensitivity: 0.955 (0.876-0.985) | ||
| Specificity: 1.000 (0.993-1.000) | Sensitivity: 0.955 (0.876-0.985) | |||
| Specificity: 1.000 (0.993-1.000) | 0.960 (0.930-0.989) | |||
| Cardiac axis deviation | High Sensitivity & High Specificity | Sensitivity: 0.945 (0.851-0.981) | ||
| Specificity: 1.000 (0.993-1.000) | Sensitivity: 0.945 (0.851-0.981) | |||
| Specificity: 1.000 (0.993-1.000) | 0.967 (0.932-1.000) | |||
| PV/AV size discrepancy | High Sensitivity & High Specificity | Sensitivity: 0.954 (0.914-0.975) | ||
| Specificity: 0.989 (0.977-0.995) | Sensitivity: 0.954 (0.914-0.975) | |||
| Specificity: 0.989 (0.977-0.995) | 0.979 (0.962-0.997) | |||
| RV/LV size discrepancy | High Sensitivity & High Specificity | Sensitivity: 0.950 (0.900-0.975) | ||
| Specificity: 1.000 (0.993-1.000) | Sensitivity: 0.950 (0.900-0.975) | |||
| Specificity: 1.000 (0.993-1.000) | 0.991 (0.983-0.999) | |||
| TV/MV size discrepancy | High Sensitivity & High Specificity | Sensitivity: 0.943 (0.896-0.970) | ||
| Specificity: 1.000 (0.993-1.000) | Sensitivity: 0.943 (0.896-0.970) | |||
| Specificity: 1.000 (0.993-1.000) | 0.964 (0.938-0.990) | |||
| MRMC Study Performance | ||||
| ROC AUC (any suspicious finding) | Significantly higher with aid than unaided | N/A | N/A | Aided: 0.974 (0.957-0.990) |
| Unaided: 0.825 (0.741-0.908) | ||||
| Mean Sensitivity (Any finding) | Increased with aid | N/A | N/A | Aided: 0.935 (0.892-0.978) |
| Unaided: 0.782 (0.686-0.878) | ||||
| Mean Specificity (Any finding) | Increased with aid | N/A | N/A | Aided: 0.970 (0.949-0.991) |
| Unaided: 0.759 (0.630-0.887) | ||||
| Conclusive output rate | High | 98.8% (95% CL, 97.8-99.3) | N/A | N/A |
2. Sample Size and Data Provenance for the Test Set
- Sample Size for Standalone Test Set: 877 clinically acquired fetal ultrasound exams.
- Sample Size for MRMC Test Set: 200 exams.
- Data Provenance: The data was collected from 11 centers in the U.S.A. and France. It was retrospectively collected as it refers to "clinically acquired fetal ultrasound exams".
3. Number of Experts and Qualifications for Ground Truth
- Number of Experts: Three (3) pediatric cardiologists.
- Qualifications of Experts: The document specifies "pediatric cardiologists" but does not provide details on their years of experience or other specific qualifications beyond their specialty.
4. Adjudication Method for the Test Set
- Adjudication Method: Majority voting. This means that if at least two out of the three pediatric cardiologists agreed on the presence or absence of a finding, that was established as the ground truth.
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
- Was an MRMC study done? Yes.
- Effect size of human readers improvement with AI vs. without AI assistance:
- ROC AUC for any suspicious finding: +14.9% increase (from 0.825 unaided to 0.974 aided, p=0.002).
- Mean Sensitivity for any suspicious finding: +15.3% increase (from 0.782 unaided to 0.935 aided).
- Mean Specificity for any suspicious finding: +21.1% increase (from 0.759 unaided to 0.970 aided).
6. Standalone (Algorithm Only) Performance Study
- Was a standalone study done? Yes.
- The results are presented in Table 1, showing sensitivity and specificity for "Any suspicious findings" and each individual finding, calculated under two scenarios for inconclusive outputs.
7. Type of Ground Truth Used
- Type of Ground Truth: Expert consensus. Specifically, it was derived from a truthing process by three pediatric cardiologists using majority voting.
8. Sample Size for the Training Set
- The document states that "The ultrasound examinations used for training and validation are entirely distinct from the examinations used in standalone testing," but it does not explicitly provide the sample size for the training set.
9. How the Ground Truth for the Training Set Was Established
- The document states that the "ultrasound examinations used for training and validation are entirely distinct from the examinations used in standalone testing." However, similar to the training set sample size, it does not explicitly describe how the ground truth for the training set was established. It only details the ground truth establishment for the test sets (standalone and MRMC).
§ 892.2060 Radiological computer-assisted diagnostic software for lesions suspicious of cancer.
(a)
Identification. A radiological computer-assisted diagnostic software for lesions suspicious of cancer is an image processing prescription device intended to aid in the characterization of lesions as suspicious for cancer identified on acquired medical images such as magnetic resonance, mammography, radiography, or computed tomography. The device characterizes lesions based on features or information extracted from the images and provides information about the lesion(s) to the user. Diagnostic and patient management decisions are made by the clinical user.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Design verification and validation must include:
(i) A detailed description of the image analysis algorithms including, but not limited to, a detailed description of the algorithm inputs and outputs, each major component or block, and algorithm limitations.
(ii) A detailed description of pre-specified performance testing protocols and dataset(s) used to assess whether the device will improve reader performance as intended.
(iii) Results from performance testing protocols that demonstrate that the device improves reader performance in the intended use population when used in accordance with the instructions for use. The performance assessment must be based on appropriate diagnostic accuracy measures (
e.g., receiver operator characteristic plot, sensitivity, specificity, predictive value, and diagnostic likelihood ratio). The test dataset must contain sufficient numbers of cases from important cohorts (e.g., subsets defined by clinically relevant confounders, effect modifiers, concomitant diseases, and subsets defined by image acquisition characteristics) such that the performance estimates and confidence intervals of the device for these individual subsets can be characterized for the intended use population and imaging equipment.(iv) Standalone performance testing protocols and results of the device.
(v) Appropriate software documentation (
e.g., device hazard analysis; software requirements specification document; software design specification document; traceability analysis; and description of verification and validation activities including system level test protocol, pass/fail criteria, results, and cybersecurity).(2) Labeling must include:
(i) A detailed description of the patient population for which the device is indicated for use.
(ii) A detailed description of the intended reading protocol.
(iii) A detailed description of the intended user and recommended user training.
(iv) A detailed description of the device inputs and outputs.
(v) A detailed description of compatible imaging hardware and imaging protocols.
(vi) Warnings, precautions, and limitations, including situations in which the device may fail or may not operate at its expected performance level (
e.g., poor image quality or for certain subpopulations), as applicable.(vii) Detailed instructions for use.
(viii) A detailed summary of the performance testing, including: Test methods, dataset characteristics, results, and a summary of sub-analyses on case distributions stratified by relevant confounders (
e.g., lesion and organ characteristics, disease stages, and imaging equipment).