(191 days)
EchoGo Pro v1.0.2 is a machine learning-based decision support system, indicated as an adjunct to diagnotic stress echocardiography for patients undergoing assessment for coronary artery disease (CAD). When utilized by an interpreting physician, this device provides information that may be useful in rendering an accurate diagnosis. Patient management decisions should not be made solely on the results of the EchoGo Pro v 1.0.2 analysis. EchoGo Pro v 1.0.2 is to be used with stress echo exam protocols that contain A2C, A4C and mid-ventricular short-axis views at rest and at peak stress. EchoGo Pro v1.0.2 is not intended for the assessment of mild or moderate myocardial ischemia, or localization of coronary artery disease, or for the assessment of myocardial perfusion, myocardial viability or valve disease.
EchoGo Pro v1.0.2 is a standalone software application that utilizes anonymized DICOM 3.0 compliant stress echo (SE) datasets to provide a categorical assessment as to whether the data are suggestive of a higher or lower possibility of significant CAD. The software automatically registers images, and segments and analyses selected regions of interest (ROI). EchoGo Pro v1.0.2 utilizes standard clinical SE protocols that provide apical 2 chamber and parasternal short axis (SAX) views.
Ultrasound images are acquired from a third-party acquisition device. The incoming DICOM study is checked for consistency and completeness, i.e. whether all required views labels are present in metadata. Once the technical QC has been performed on the DICOM datasets, the algorithm for automated contour detection of the endocardium of the LV is applied and presented for review and approval by trained Operators. An auto-contouring algorithm places a trace around the LV that sufficiently captures the LV shape. Outlining is detected for all frames in between and including end-systole (ES) and end-diastole (ED) for AP2, AP4, and mid-ventricular short axis (SAX) views of the LV at both rest and peak stress. Trained operators review and approve of the contour traces. The approved contour traces are used in calculations for geometric parameters.
Geometric parameters are calculated from the approved contours and are fed into a fixed classification model that has been previously trained on datasets with known outcomes. The output of the pre-trained model generates a report which contains a categorical assessment as to whether the data are consistent with significant CAD or not. Significant CAD as determined by EchoGo Pro, based on LV segmentation (ROI) analysis, and was defined as ≥70% stenosis in the proximal to mid LAD, proximal left circumflex or proximal to mid RCA as measured by invasive angiography performed within 6 months of stress echocardiogram.
Here's an analysis of the acceptance criteria and study findings for the EchoGo Pro device based on the provided text:
1. Table of Acceptance Criteria and Reported Device Performance
| Acceptance Criteria Category | Specific Acceptance Criteria | Reported Device Performance and Confidence Intervals |
|---|---|---|
| Primary Endpoint: Reader Performance Improvement (AUROC) | The difference between the diagnostic performance of readers when interpreting ultrasound studies with and without the assistance of EchoGo Pro v1.0.2 is equivalent or better than that of the predicate device (Koios DS, K190442), which reported a mean reader improvement in AUROC of 0.034. | The difference in AUROC (USE + EGP vs. USE Alone) was 0.054 (95% CI 0.032, 0.077) at p = 0.02. This satisfied the acceptance criteria as it exceeded the mean reader improvement (0.034) reported for the predicate device. |
| Standalone Performance (Native System AUROC) | The system achieved a native system performance equivalent to or better than the predicate device (K190442), which reported a native system performance AUROC of 0.882. | The system achieved a "native system performance" of 0.927 AUROC. This is greater than the native system performance reported for the predicate device (0.882). |
| Standalone Performance (Native System Specificity) | (Implicit, based on standalone AUROC and sensitivity, demonstrating robust performance) | Specificity was 0.927 (95% CI 0.878, 0.976). |
| Standalone Performance (Native System Sensitivity) | (Implicit, based on standalone AUROC and specificity, demonstrating robust performance) | Sensitivity was 0.844 (95% CI 0.739, 0.950). |
| Secondary Endpoint: Inter-operator Agreement Improvement (Kendall Tau-B) | The difference between inter-operator agreement when interpreting ultrasound studies with and without the assistance of EchoGo Pro v1.0.2 is equivalent or better than that of the predicate device (Koios DS, K190442), which reported a mean Kendall Tau-B improvement of 0.1393. | The average Kendall Tau-B of USE Alone was 0.58 (0.48, 0.69). The average Kendall Tau-B of USE + EGP was 0.82 (0.72, 0.93). The increase in the metric was significant (p 0 for all reader pairs. |
| Software Verification and Validation | Documentation provided as recommended by FDA's Guidance for Industry and FDA Staff, "Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices." Software considered a "moderate" level of concern. | Software verification and validation testing were conducted, and documentation was provided as recommended by FDA's Guidance. The software was considered to be of "moderate" level of concern. (No specific performance metrics are given for V&V beyond compliance). |
2. Sample Size Used for the Test Set and Data Provenance
The document does not explicitly state the specific numerical sample size (number of patients/cases) for the test set used in the performance/clinical testing. It only mentions that "ROC curves were generated and analyzed" and refers to "readers" interpreting "ultrasound studies."
The data provenance (country of origin, retrospective/prospective) is not specified in the provided text.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications of Those Experts
The document does not explicitly state the number of experts used to establish the ground truth for the test set, nor does it specify their qualifications (e.g., "radiologist with 10 years of experience"). It mentions that ground truth was based on "known outcomes" for the training set and that "Significant CAD... was defined as ≥70% stenosis in the proximal to mid LAD, proximal left circumflex or proximal to mid RCA as measured by invasive angiography." This implies that the ground truth for CAD status was likely derived from invasive angiography reports, which are interpreted by cardiologists or interventional cardiologists.
4. Adjudication Method for the Test Set
The document does not explicitly state the adjudication method used for the test set. It mentions "known outcomes" and diagnostic performance with readers, but not how ground truth disagreements were resolved or if a consensus panel was used for the test set.
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study Was Done, and the Effect Size of How Much Human Readers Improve with AI vs. Without AI Assistance
Yes, a multi-reader comparative effectiveness study was done. This is evident from the "primary endpoint of the study," which assessed "the difference between the diagnostic performance of readers when interpreting ultrasound studies with and without the assistance of EchoGo Pro v1.0.2."
The effect size of improvement for human readers with AI assistance (USE + EGP) vs. without AI assistance (USE Alone) in terms of AUROC was 0.054. This means that, on average, the area under the receiver operating characteristic curve for readers improved by 0.054 when using EchoGo Pro.
6. If a Standalone (Algorithm Only Without Human-in-the-Loop Performance) Was Done
Yes, a standalone performance evaluation was done. The document states: "The system achieved a native system performance of 0.927 AUROC, with specificity of 0.927 (95% CI 0.878, 0.976) and a sensitivity of 0.844 (95% CI 0.739, 0.950)." This "native system performance" refers to the algorithm's performance without direct human-in-the-loop diagnostic interpretation.
7. The Type of Ground Truth Used
The ground truth for "significant CAD" was defined as ≥70% stenosis in the proximal to mid LAD, proximal left circumflex, or proximal to mid RCA as measured by invasive angiography performed within 6 months of stress echocardiogram. This is a form of outcome data (angiographically confirmed disease status), considered a high-fidelity ground truth for CAD.
8. The Sample Size for the Training Set
The sample size for the training set is not specified in the provided text. It is mentioned that the "fixed classification model... has been previously trained on datasets with known outcomes," but no quantitative details about these datasets are given.
9. How the Ground Truth for the Training Set Was Established
The ground truth for the training set was established based on known outcomes, specifically "Significant CAD as determined... based on LV segmentation (ROI) analysis, and was defined as ≥70% stenosis in the proximal to mid LAD, proximal left circumflex or proximal to mid RCA as measured by invasive angiography performed within 6 months of stress echocardiogram." This indicates that the training data also utilized invasive angiography results to confirm the CAD status, similar to the ground truth definition for the test set.
§ 892.2060 Radiological computer-assisted diagnostic software for lesions suspicious of cancer.
(a)
Identification. A radiological computer-assisted diagnostic software for lesions suspicious of cancer is an image processing prescription device intended to aid in the characterization of lesions as suspicious for cancer identified on acquired medical images such as magnetic resonance, mammography, radiography, or computed tomography. The device characterizes lesions based on features or information extracted from the images and provides information about the lesion(s) to the user. Diagnostic and patient management decisions are made by the clinical user.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Design verification and validation must include:
(i) A detailed description of the image analysis algorithms including, but not limited to, a detailed description of the algorithm inputs and outputs, each major component or block, and algorithm limitations.
(ii) A detailed description of pre-specified performance testing protocols and dataset(s) used to assess whether the device will improve reader performance as intended.
(iii) Results from performance testing protocols that demonstrate that the device improves reader performance in the intended use population when used in accordance with the instructions for use. The performance assessment must be based on appropriate diagnostic accuracy measures (
e.g., receiver operator characteristic plot, sensitivity, specificity, predictive value, and diagnostic likelihood ratio). The test dataset must contain sufficient numbers of cases from important cohorts (e.g., subsets defined by clinically relevant confounders, effect modifiers, concomitant diseases, and subsets defined by image acquisition characteristics) such that the performance estimates and confidence intervals of the device for these individual subsets can be characterized for the intended use population and imaging equipment.(iv) Standalone performance testing protocols and results of the device.
(v) Appropriate software documentation (
e.g., device hazard analysis; software requirements specification document; software design specification document; traceability analysis; and description of verification and validation activities including system level test protocol, pass/fail criteria, results, and cybersecurity).(2) Labeling must include:
(i) A detailed description of the patient population for which the device is indicated for use.
(ii) A detailed description of the intended reading protocol.
(iii) A detailed description of the intended user and recommended user training.
(iv) A detailed description of the device inputs and outputs.
(v) A detailed description of compatible imaging hardware and imaging protocols.
(vi) Warnings, precautions, and limitations, including situations in which the device may fail or may not operate at its expected performance level (
e.g., poor image quality or for certain subpopulations), as applicable.(vii) Detailed instructions for use.
(viii) A detailed summary of the performance testing, including: Test methods, dataset characteristics, results, and a summary of sub-analyses on case distributions stratified by relevant confounders (
e.g., lesion and organ characteristics, disease stages, and imaging equipment).