(135 days)
The Caption Interpretation Automated Ejection Fraction software is used to process previously acquired transthoracic cardiac ultrasound images, to store images, and to manipulate and make measurements on images using an ultrasound device, personal computer, or a compatible DICOM-compliant PACS system in order to provide automated estimation of left ventricular ejection. This measurement can be used to assist the clinician in a cardiac evaluation.
The Caption Interpretation Automated Ejection Fraction Software is indicated for use in adult patients.
The Caption Interpretation Automated Ejection Fraction Software applies machine learning algorithms to process echocardiography images in order to calculate left ventricular ejection fraction. Caption Interpretation AutoEF performs left ventricular ejection fraction measurements using the apical four chamber, apical two chamber or parasternal long-axis cardiac ultrasound views or a combination of those views. The software selects the image clips to be used, performs the AutoEF calculation, and forwards the results to the desired destination for clinician viewing. The output of the program is the Ejection Fraction estimate stated as a percentage, along with an indications of confidence regarding that estimate.
Here's a breakdown of the acceptance criteria and the study proving the device meets them, based on the provided text:
Acceptance Criteria and Reported Device Performance
| Acceptance Criteria Category | Specific Criteria | Reported Device Performance |
|---|---|---|
| Clip Annotator Study | PPV for identification of imaging mode and view > 97% | Observed PPV point estimates > 97% |
| Clip Annotator Study | Sensitivity across views and imaging mode > 90% | Observed sensitivity point estimates > 90% |
| Pivotal Clinical Study (Primary Endpoint) | RMSD between AutoEF derived values (best available view combination) and reference method | Met predetermined acceptance criteria |
| Pivotal Clinical Study (Secondary Endpoint) | RMSD for combinations of views (AP2, AP4, PLAX) | Met same predetermined acceptance criteria |
| Pivotal Clinical Study (Single-View AP4) | RMSD for AP4 view | Observed results less than acceptance criterion |
| Pivotal Clinical Study (Single-View AP4) | Performance compared to physicians in qualitative and quantitative visual assessment | Superior to physicians |
| Pivotal Clinical Study (Single-View AP2) | RMSD for AP2 view | Observed results did not meet acceptance criteria, but performed superior on a quantitative visual assessment |
| Pivotal Clinical Study (Single-View PLAX) | RMSD for PLAX view | Observed results did not meet acceptance criteria, but performed superior on a quantitative visual assessment |
| Pivotal Clinical Study (Single-View AP2/PLAX RMSD compared to sonographers) | AP2-only and PLAX-only RMSD compared to sonographers' biplane tracing before cardiologist overread | Observed RMSD lower than sonographers' biplane tracing before cardiologist overread |
| Confidence Metric Functionality | Successful performance in estimating error range of EF estimates around reference EF with normally distributed difference (estimated vs. reference EF) | Verified successful performance |
Detailed Study Information:
-
Sample sizes used for the test set and data provenance:
- The document does not explicitly state the sample sizes (number of patients or echocardiograms) used for the test sets in either the Clip Annotator study or the pivotal clinical investigation.
- Data Provenance: The document does not specify the country of origin of the data. It implies the data was "previously acquired transthoracic cardiac ultrasound images," but does not explicitly state if it was retrospective or prospective data.
-
Number of experts used to establish the ground truth for the test set and the qualifications of those experts:
- Clip Annotator Study: "a panel of expert readers" was used. The exact number of experts and their specific qualifications are not provided beyond "expert readers."
- Pivotal Clinical Study: The "reference method" is mentioned, implying a ground truth was established, but the number and qualifications of experts involved in creating this reference (e.g., conventional EF calculation methods by cardiologists) are not detailed. It mentions "sonographers' biplane tracing before a cardiologist overread" in the context of comparison, suggesting cardiologist input in the reference.
-
Adjudication method for the test set:
- Clip Annotator Study: "Results of the Clip Annotator were compared to evaluation by a panel of expert readers." This suggests a direct comparison rather than a specific multi-reader adjudication method like 2+1 or 3+1. It's unclear if there was an adjudication for disagreements among the panel.
- Pivotal Clinical Study: The document refers to a "reference method" for EF calculation. It doesn't specify an adjudication method for this reference, nor for any disagreements in the human interpretations used for comparison (e.g., sonographers' tracings).
-
If a Multi-Reader Multi-Case (MRMC) comparative effectiveness study was done, and the effect size of how much human readers improve with AI vs without AI assistance:
- The document implies a comparison between the AI's performance and human performance, stating the AP4 view was "superior to physicians in making a qualitative and quantitative visual assessment" and that AP2-only and PLAX-only RMSD was "lower than the RMSD of sonographers' biplane tracing before a cardiologist overread."
- However, it does not explicitly describe a formal MRMC comparative effectiveness study where human readers' performance with AI assistance is directly compared to their performance without AI assistance. The comparisons made seem to be between the AI's standalone performance and human performance (or a component of human performance like tracing).
- Therefore, an effect size of how much human readers improve with AI vs. without AI assistance cannot be extracted from this text.
-
If a standalone (i.e., algorithm only without human-in-the-loop performance) was done:
- Yes, the core objective of the studies described is to evaluate the standalone performance of the "Caption Interpretation Automated Ejection Fraction Software" in calculating EF and selecting clips. The reported "AutoEF accuracy" and RMSD values are indicative of standalone algorithm performance.
-
The type of ground truth used:
- Clip Annotator Study: Expert consensus/evaluation by a panel of expert readers concerning the correct imaging mode and view.
- Pivotal Clinical Study: "Conventional EF calculation methods" served as the "reference method." This likely implies expert-derived EF measurements, potentially involving cardiologist overread of sonographer tracings, and possibly other established clinical methods.
-
The sample size for the training set:
- The document states, "The algorithms for estimating ejection fraction have been further optimized though additional training." and "Images and cases used for verification testing were carefully separated from training algorithms."
- However, the specific sample size for the training set is not provided in this document.
-
How the ground truth for the training set was established:
- The document states that the algorithms were "optimized though additional training" but does not detail how the ground truth for this training data was established. It only mentions that "Images and cases used for verification testing were carefully separated from training algorithms," which relates to data splitting, not ground truth establishment for training.
§ 892.2050 Medical image management and processing system.
(a)
Identification. A medical image management and processing system is a device that provides one or more capabilities relating to the review and digital processing of medical images for the purposes of interpretation by a trained practitioner of disease detection, diagnosis, or patient management. The software components may provide advanced or complex image processing functions for image manipulation, enhancement, or quantification that are intended for use in the interpretation and analysis of medical images. Advanced image manipulation functions may include image segmentation, multimodality image registration, or 3D visualization. Complex quantitative functions may include semi-automated measurements or time-series measurements.(b)
Classification. Class II (special controls; voluntary standards—Digital Imaging and Communications in Medicine (DICOM) Std., Joint Photographic Experts Group (JPEG) Std., Society of Motion Picture and Television Engineers (SMPTE) Test Pattern).