(123 days)
Cleerly ISCHEMIA analysis software is an automated machine learning-based decision support tool, indicated as a diagnostic aid for patients undergoing CT analysis using Cleerly Labs software. When utilized by an interpreting healthcare provider, this software tool provides information that may be useful in detecting likely ischemia associated with coronary artery disease. Patient management decisions should not be made solely on the results of the Cleerly ISCHEMIA analysis.
Cleerly ISCHEMIA is an add-on software module to Cleerly Labs (K202280, K190868) that determines the likely presence or absence of coronal vessel ischemia based on quantitative measures of atherosclerosis, stenosis, and significant vascular morphology from typically-acquired Coronary Computed Tomography Angiography images (CCTA). Cleerly ISCHEMIA, in conjunction with Cleerly Labs, outputs a Cleerly ISCHEMIA Index (CII), a binary indication of negative CII (likely absence of ischemia) or positive CII (likely presence of ischemia) with its threshold equivalent to invasive FFR >0.80 vs. ≤0.80, respectively, as identified in professional societal practice guidelines.
The provided document describes the Cleerly ISCHEMIA device and its clinical validation. Here's a breakdown of the requested information based on the text:
1. A table of acceptance criteria and the reported device performance
The document does not explicitly state pre-defined acceptance criteria (e.g., minimum sensitivity of X% and specificity of Y%). Instead, it presents the results of the primary endpoint analysis from the CREDENCE Trial and then pooled results from additional studies. Therefore, the reported device performance serves as the basis for demonstrating acceptable clinical performance.
| Metric (Per-vessel territory) | Reported Device Performance (CREDENCE Trial, Primary Endpoint) |
|---|---|
| Sensitivity | 75.9% (167/220) |
| Specificity | 83.4% (521/625) |
Additional performance data from pooled US and OUS cohorts are also provided:
| Metric (Pooled US + OUS, Per-vessel territory) | Estimate | 95% CI |
|---|---|---|
| Sensitivity | 76.2% | 71.9%, 80.3% |
| Specificity | 85.2% | 82.8%, 87.4% |
| PPV | 65.9% | 61.2%, 70.3% |
| NPV | 90.5% | 88.5%, 92.3% |
| LR+ | 5.15 | - |
| LR- | 0.28 | - |
| Metric (Pooled US + OUS, Per patient territory) | Estimate | 95% CI |
|---|---|---|
| Sensitivity | 86.6% | 82.1%, 90.1% |
| Specificity | 69.8% | 64.4%, 74.7% |
| PPV | 73.2% | 68.2%, 77.7% |
| NPV | 84.6% | 79.5%, 88.6% |
| LR+ | 2.87 | - |
| LR- | 0.19 | - |
The conclusion states, "Cumulatively, these data demonstrate acceptable clinical performance," implying that the presented performance values met the internal acceptance standards for regulatory submission.
2. Sample size used for the test set and the data provenance (e.g. country of origin of the data, retrospective or prospective)
- Test Set (CREDENCE Trial Validation Set): 305 patients.
- Data Provenance: The CREDENCE Trial was a prospective, multicenter trial conducted across 17 centers (later mentioned as 23 centers in the pooled data description, implying evolution or different reporting) between 2014 and 2017. It recruited patients with stable symptoms and without a prior diagnosis of CAD, referred for non-emergent ICA. The primary endpoint analysis was based on the validation set from this trial. The document states a "US/OUS cohort population" was used for pooled data, and then breaks down the pooled data into "Pooled US" (N=149 subjects) and "Pooled OUS" (N=433 subjects). The CREDENCE trial, being a large multi-center study, likely spanned multiple countries, but the specific breakdown of US vs. OUS for the initial CREDENCE derivation/validation sets isn't explicitly detailed; however, subsequent pooled data clearly delineate US and OUS categories.
3. Number of experts used to establish the ground truth for the test set and the qualifications of those experts
The document states, "Clinical validation testing was done to validate the diagnostic performance of Cleerly ISCHEMIA for non-invasive determination of the functional significance of CAD, as referenced to direct invasive measurement of FFR as the reference standard." It also mentions, "All index tests were interpreted blindly by core laboratories."
- Number of Experts: Not explicitly stated for the interpretation of FFR.
- Qualifications of Experts: Not explicitly stated, though "core laboratories" implies a standard of expertise in cardiology and interventional procedures necessary for FFR measurement and interpretation.
4. Adjudication method (e.g. 2+1, 3+1, none) for the test set
The document mentions that FFR was the "reference standard." Invasive FFR measurement is an objective physiological assessment, rather than a subjective interpretation requiring adjudication. For the interpretation of the CCTA images that serve as input to Cleerly Labs (and subsequently Cleerly ISCHEMIA), it states, "All index tests were interpreted blindly by core laboratories." The specific adjudication method (e.g., consensual read vs. single reader) by these core laboratories for CCTA interpretation is not detailed. However, the ground truth for Cleerly ISCHEMIA is directly linked to the quantitative invasive FFR values.
5. If a multi-reader multi-case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance
No MRMC comparative effectiveness study involving human readers with and without AI assistance is described in this document. The study focuses on the standalone diagnostic performance of the Cleerly ISCHEMIA algorithm against an invasive reference standard (FFR). It is presented as a "diagnostic aid" for use by an interpreting healthcare provider, implying it provides information to the provider, but the study doesn't quantify interaction or improvement.
6. If a standalone (i.e. algorithm only without human-in-the-loop performance) was done
Yes, a standalone performance evaluation was clearly done. The clinical validation section explicitly describes the performance of the "Cleerly ISCHEMIA" device in detecting likely ischemia as referenced to invasive FFR. The results (sensitivity, specificity, etc.) are reported for the algorithm's output.
7. The type of ground truth used (expert consensus, pathology, outcomes data, etc)
The ground truth used was invasive fractional flow reserve (FFR), described as the "reference standard" for determining the functional significance of coronary artery disease. A Cleerly ISCHEMIA Index (CII) of positive (likely ischemia) corresponds to invasive FFR ≤0.80, and negative CII corresponds to FFR >0.80.
8. The sample size for the training set
The CREDENCE Trial cohort was divided into two subsets: "the first half of enrollees at each site assigned to the derivation (n = 307) and the second half to the validation (n = 305) data set." The derivation set (n=307) would typically serve as the training/development set for the algorithm. The document doesn't explicitly refer to it as the "training set," but "derivation" implies its use in developing/optimizing the algorithm.
9. How the ground truth for the training set was established
For the derivation set, the ground truth would have been established in the same manner as for the validation set: direct invasive measurement of FFR. The CREDENCE trial collected FFR data for all enrollees, which were then allocated to either the derivation or validation sets.
§ 870.2200 Adjunctive cardiovascular status indicator.
(a)
Identification. The adjunctive cardiovascular status indicator is a prescription device based on sensor technology for the measurement of a physical parameter(s). This device is intended for adjunctive use with other physical vital sign parameters and patient information and is not intended to independently direct therapy.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Software description, verification, and validation based on comprehensive hazard analysis must be provided, including:
(i) Full characterization of technical parameters of the software, including any proprietary algorithm(s);
(ii) Description of the expected impact of all applicable sensor acquisition hardware characteristics on performance and any associated hardware specifications;
(iii) Specification of acceptable incoming sensor data quality control measures; and
(iv) Mitigation of impact of user error or failure of any subsystem components (signal detection and analysis, data display, and storage) on accuracy of patient reports.
(2) Scientific justification for the validity of the status indicator algorithm(s) must be provided. Verification of algorithm calculations and validation testing of the algorithm using a data set separate from the training data must demonstrate the validity of modeling.
(3) Usability assessment must be provided to demonstrate that risk of misinterpretation of the status indicator is appropriately mitigated.
(4) Clinical data must be provided in support of the intended use and include the following:
(i) Output measure(s) must be compared to an acceptable reference method to demonstrate that the output measure(s) represent(s) the predictive measure(s) that the device provides in an accurate and reproducible manner;
(ii) The data set must be representative of the intended use population for the device. Any selection criteria or limitations of the samples must be fully described and justified;
(iii) Agreement of the measure(s) with the reference measure(s) must be assessed across the full measurement range; and
(iv) Data must be provided within the clinical validation study or using equivalent datasets to demonstrate the consistency of the output and be representative of the range of data sources and data quality likely to be encountered in the intended use population and relevant use conditions in the intended use environment.
(5) Labeling must include the following:
(i) The type of sensor data used, including specification of compatible sensors for data acquisition;
(ii) A description of what the device measures and outputs to the user;
(iii) Warnings identifying sensor reading acquisition factors that may impact measurement results;
(iv) Guidance for interpretation of the measurements, including warning(s) specifying adjunctive use of the measurements;
(v) Key assumptions made in the calculation and determination of measurements;
(vi) The measurement performance of the device for all presented parameters, with appropriate confidence intervals, and the supporting evidence for this performance; and
(vii) A detailed description of the patients studied in the clinical validation (
e.g., age, gender, race/ethnicity, clinical stability) as well as procedural details of the clinical study.