(216 days)
AISAP CARDIO V1.0 is a software platform that automatically processes and analyzes acquired cardiac POCUS images, producing a report with diagnostic assessment and measurements of several key cardiac structural and functional parameters, including: presence of valvular pathology (regurgitations of the mitral, tricuspid, aortic valves and aortic stenosis), and measurements of the Left Ventricle Ejection Fraction (LVEF), right and left ventricular dimensions, right ventricular fractional area change (RV FAC), atrial areas, ascending aorta diameter, and inferior vena cava (IVC) diameter.
The device outputs are provided in a report that is intended to support qualified physicians in their analysis and interpretation of adult cardiac POCUS images, using FDA-cleared ultrasound devices. Physicians should be trained and privileged by their organization following education processes and should perform cardiac POCUS according to their specialty professional society clinical guidelines.
AISAP CARDIO V1.0 has not been validated for the assessment of congenital heart disease, and/or intra-cardiac lesions (e.g., tumors, thrombi, vegetations), prosthetic valves, and in the presence of ventricular assist devices.
AISAP CARDIO V1.0 is indicated for use in adult patients only.
AISAP CARDIO V1.0 is a machine learning-based decision support software device, indicated as an adjunct to diagnostic Cardiac point of care ultrasound (C-POCUS) for adult patients undergoing assessment for cardiac disease. This device performs automated analysis of ultrasound images and generates valvular assessments and measurements of standard cardiac structural and functional parameters.
-
- Inform the user of a suspected cardiac valvular regurgitation (mitral, tricuspid, or aortic), and/or aortic stenosis is: either greater than mild severity or none to mild severity.
-
- Inform the user of the 4 class American Society of Echocardiography (ASE) recommended category for valvular regurgitation (mitral, tricuspid, or aortic), and or aortic stenosis. Each finding categorizes according to none, mild, moderate, or severe.
-
- Measurements of the following standard cardiac structural or functional parameters:
- Left Ventricular Ejection Fraction (LVEF) (percent) a.
- Left ventricular end diastolic diameter (cm) b.
- Right ventricular area change (RV FAC [ratio]) C.
- Inferior vena cava (IVC) maximal diameter (mm) d.
- e. Aortic root diameter (cm)
- ਿ Right atrium (RA) area (cm2)
- Left atrium (LA) area (cm²) g.
AISAP CARDIO V1.0 assists the physician in assessing 4 major valvular findings in adults, along with providing information on several correlated cardiac ultrasound measurements frequently found to be abnormal in association with valvular heart disease. Used together and interpreted by the physician, the device provides information that may assist in rendering an accurate diagnosis of selected cardiac findings. AISAP CARDIO V1.0 is adjunctive to cardiac POCUS (C-POCUS) use by privileged physicians in use scenarios supported by clinical guidelines. Specifically, patient management decisions are not intended to be and should not be made solely on the results of the software analysis of the proposed device. When significant valve pathology is suspected comprehensive echocardiography should be considered in accordance with the relevant professional guidelines.
AISAP CARDIO V1.0 uses machine learning NN (neural network) models trained to recognize patterns and make decisions. AISAP CARDIO V1.0 contains classification models which identify categories within data, regression models which predict numerical values, and instance segmentation models that detect and segment objects within images.
Here's a breakdown of the acceptance criteria and supporting studies for the AISAP Cardio V1.0 device, based on the provided text:
1. Table of Acceptance Criteria and Reported Device Performance
| Feature/Measurement | Acceptance Criteria | Reported Device Performance | Study Type |
|---|---|---|---|
| Structural & Functional Measurements | Standalone Model Performance | ||
| LVEF | RMSE 0.80 | MR: 0.975; AS: 0.969; AR: 0.993; TR: 0.973 | Standalone Model Performance |
| Clinical Reader Performance | Multi-Reader Study | ||
| MR (Aided vs. Un-aided) | Lower bound of 95% CI for (AUC_aided - AUC_unaided) > 0 | AUC_aided (0.963) > AUC_unaided (0.870) | Clinical Reader Performance (MRMC) |
| TR (Aided vs. Un-aided) | Lower bound of 95% CI for (AUC_aided - AUC_unaided) > 0 | AUC_aided (0.937) > AUC_unaided (0.851) | Clinical Reader Performance (MRMC) |
| AR (Aided vs. Un-aided) | Lower bound of 95% CI for (AUC_aided - AUC_unaided) > 0 | AUC_aided (0.947) > AUC_unaided (0.868) | Clinical Reader Performance (MRMC) |
| AS (Aided vs. Un-aided) | Lower bound of 95% CI for (AUC_aided - AUC_unaided) > 0 | AUC_aided (0.925) > AUC_unaided (0.897) | Clinical Reader Performance (MRMC) |
| View Classification | Accuracy > 95% | PLAX: 100%; PSAX: 99.2%; A4C: Not reported; SC IVC: 98.8% | View Classification Validation Study |
2. Sample Size for Test Set and Data Provenance
- Standalone Structural and Functional Measurements Study: 200 cases
- Standalone Valvular Pathology Study: 329 cases
- Clinical Reader Performance (MRMC) Study: 260 cases
- View Classification Validation Study: 500 sampled loops per cardiac view (from the clinical study dataset)
Data Provenance:
The test data was collected prospectively at 4 clinical reader sites located in the United States (51% of cases) and Israel (49% of cases). Images were acquired with different US device vendors (Philips, GE, Wisonic, EchoNous) from both in-patient and out-patient settings. Both physicians and sonographers performed the POCUS exams.
3. Number of Experts and Qualifications for Test Set Ground Truth
- Structural and Functional Measurements Study: 3 US board-certified cardiologists with a minimum of 5 years of experience.
- Valvular Pathology Study: Cardiologist interpretations (number not specified, but the context implies multiple experts as ground truth for other studies).
- Clinical Reader Performance (MRMC) Study: 3 US Board Certified cardiologists (for the severity grade of valvular pathologies).
- View Classification Validation Study: 2 certified experienced echo technicians, with over-read by a lead technician and a senior cardiologist.
4. Adjudication Method for Test Set
- Structural and Functional Measurements Study: Ground truth was established by the mean value determined by the 3 cardiologists' measurements following ASE guidelines.
- Valvular Pathology Study: Not explicitly stated, but implies expert interpretation as the ground truth.
- Clinical Reader Performance (MRMC) Study: "2+1" annotation strategy. The 260 cases were interpreted independently by two U.S. Board Certified ground truth cardiologists. Any discrepancies were interpreted by a third ground truth cardiologist. Any persistent disagreements were decided at a meeting of the three ground truth cardiologists.
- View Classification Validation Study: View verification by 2 certified experienced echo technicians, with an over-read of 30% of cases by a lead technician and an additional 10% over-read by a senior cardiologist.
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
Yes, a MRMC comparative effectiveness study was done. It was called the "Clinical Reader Performance" study.
Effect Size (Improvement with AI vs. without AI assistance):
The study demonstrated an improvement in AUC, Kappa, and Accuracy when readers were aided by the device.
-
AUC Improvement (AI-aided vs. Unaided):
- MR: 0.963 vs. 0.870 (Improvement: 0.093)
- TR: 0.937 vs. 0.851 (Improvement: 0.086)
- AR: 0.947 vs. 0.868 (Improvement: 0.079)
- AS: 0.925 vs. 0.897 (Improvement: 0.028)
(The passing criteria states the lower bound of the 95% CI for this difference lay entirely above zero, indicating a statistically significant positive effect.)
-
Kappa Improvement (AI-aided vs. Unaided):
- MR: 0.881 vs. 0.756 (Improvement: 0.125)
- TR: 0.881 vs. 0.765 (Improvement: 0.116)
- AR: 0.913 vs. 0.815 (Improvement: 0.098)
- AS: 0.850 vs. 0.792 (Improvement: 0.058)
-
Accuracy Improvement (AI-aided vs. Unaided):
- MR: 73.6% vs. 61.6% (Improvement: 12.0%)
- TR: 75.3% vs. 64.1% (Improvement: 11.2%)
- AR: 80.6% vs. 71.7% (Improvement: 8.9%)
- AS: 74.7% vs. 69.8% (Improvement: 4.9%)
6. Standalone (Algorithm Only) Performance Study
Yes, standalone performance studies were done for both:
- "Standalone Model Performance for Structural and Functional Measurements"
- "Standalone Model Performance for Valvular Pathology"
7. Type of Ground Truth Used
- Structural and Functional Measurements: Expert consensus (mean value of 3 cardiologists' measurements) following American Society of Echocardiography (ASE) guidelines.
- Valvular Pathology: Cardiologist interpretations (implied expert consensus).
- Clinical Reader Performance (MRMC): Expert consensus of 3 US Board Certified cardiologists, established via an adjudication process ("2+1" strategy and consensus meeting).
- View Classification: Expert consensus of 2 certified experienced echo technicians, with over-read by a lead technician and a senior cardiologist.
8. Sample Size for Training Set
Over 140,000 individual exams were used for training the machine learning models, representing > 1 billion frames.
9. How Ground Truth for Training Set Was Established
The AISAP CARDIO V1.0 algorithms were trained at 2 academic institutions that perform cardiac ultrasound examinations and interpretations according to ASE guidelines. This implies that the ground truth for the training data was established by expert interpretation and measurements conforming to these professional guidelines at the academic institutions.
§ 892.2060 Radiological computer-assisted diagnostic software for lesions suspicious of cancer.
(a)
Identification. A radiological computer-assisted diagnostic software for lesions suspicious of cancer is an image processing prescription device intended to aid in the characterization of lesions as suspicious for cancer identified on acquired medical images such as magnetic resonance, mammography, radiography, or computed tomography. The device characterizes lesions based on features or information extracted from the images and provides information about the lesion(s) to the user. Diagnostic and patient management decisions are made by the clinical user.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Design verification and validation must include:
(i) A detailed description of the image analysis algorithms including, but not limited to, a detailed description of the algorithm inputs and outputs, each major component or block, and algorithm limitations.
(ii) A detailed description of pre-specified performance testing protocols and dataset(s) used to assess whether the device will improve reader performance as intended.
(iii) Results from performance testing protocols that demonstrate that the device improves reader performance in the intended use population when used in accordance with the instructions for use. The performance assessment must be based on appropriate diagnostic accuracy measures (
e.g., receiver operator characteristic plot, sensitivity, specificity, predictive value, and diagnostic likelihood ratio). The test dataset must contain sufficient numbers of cases from important cohorts (e.g., subsets defined by clinically relevant confounders, effect modifiers, concomitant diseases, and subsets defined by image acquisition characteristics) such that the performance estimates and confidence intervals of the device for these individual subsets can be characterized for the intended use population and imaging equipment.(iv) Standalone performance testing protocols and results of the device.
(v) Appropriate software documentation (
e.g., device hazard analysis; software requirements specification document; software design specification document; traceability analysis; and description of verification and validation activities including system level test protocol, pass/fail criteria, results, and cybersecurity).(2) Labeling must include:
(i) A detailed description of the patient population for which the device is indicated for use.
(ii) A detailed description of the intended reading protocol.
(iii) A detailed description of the intended user and recommended user training.
(iv) A detailed description of the device inputs and outputs.
(v) A detailed description of compatible imaging hardware and imaging protocols.
(vi) Warnings, precautions, and limitations, including situations in which the device may fail or may not operate at its expected performance level (
e.g., poor image quality or for certain subpopulations), as applicable.(vii) Detailed instructions for use.
(viii) A detailed summary of the performance testing, including: Test methods, dataset characteristics, results, and a summary of sub-analyses on case distributions stratified by relevant confounders (
e.g., lesion and organ characteristics, disease stages, and imaging equipment).