(267 days)
eCART is a software product that provides automated risk stratification and early warning for impending patient deterioration, signified as the composite outcome of death or ICU transfer. It is intended to be used on hospitalized ward patients 18 years of age or older by trained medical professionals.
As a clinical decision support device, eCART's risk score and trend analysis is intended to aid clinical teams in identifying which patients are most likely to clinically deteriorate. eCART provides additional information and does not replace the standard of care or clinical judgment.
eCART scoring is initiated by the documentation of any vital sign on an adult ward patient. The device calculates risk only from validated EHR data, such as vitals that have been confirmed by a registered nurse (RN); unvalidated data streaming from monitors/devices will not be used until confirmed by a healthcare professional. The product predictions are for reference only and no therapeutic decisions should be made based solely on the eCART scores.
The AgileMD eCARTv5 Clinical Deterioration Suite ("eCART") is a cloud-based software device that is integrated into the electronic health record ("EHR") in order to anticipate clinical deterioration in adult ward patients, which is signified as either of the following two predicted outcomes: (1) death or (2) ICU transfer. The tool synthesizes routine vital signs, laboratory data, and patient demographics into a single value that can be used to flag patients at-risk of the composite outcome of clinical deterioration for additional evaluation and monitoring. eCARTv5 requires the healthcare system within which it will be used, to provide an EHR connection and data interfaces through which the patient data necessary to run the software will be transmitted.
The primary functions of the system are imparted by the Gradient Boosted Machine ("GBM") learning algorithm that takes input directly from the EHR, in real time, to provide an assessment of patients and displays its outputs in an intuitive user interface which drives providers to follow standardized clinical workflows (established by their institutions) for elevated-risk patients.
eCARTv5's end users include med-surg nursing staff, physicians and other providers caring for these patients. The eCARTv5 composite score is determined from the model output (predicted probability of deterioration) scaled from 0-100, based on the specificity (true negative rate). The observed rate of deterioration at each eCART score threshold, displayed as the odds of deterioration in the next 24 hours, is presented to the user along with the scaled score. Default thresholds are set to an eCART of 93 and 97, respectively, for moderate and high risk categorization.
Here's a breakdown of the acceptance criteria and study details for the eCARTv5 Clinical Deterioration Suite, based on the provided FDA 510(k) summary:
Acceptance Criteria and Reported Device Performance
The acceptance criteria for the eCARTv5 device are implicitly defined by the performance metrics reported in the validation studies, specifically Area Under the Receiver Operating Characteristic curve (AUROC), Sensitivity, Specificity, Positive Predictive Value (PPV), and Negative Predictive Value (NPV) for two risk thresholds (Moderate-risk at eCART ≥93 and High-risk at eCART ≥97). The composite outcome of interest is "Deterioration" (death or ICU transfer within 24 hours).
Table: Acceptance Criteria (Implicit) and Reported Device Performance
| Performance Metric | Acceptance Criteria (Implicit Target) | Retrospective Cohort (Deterioration) | Prospective Cohort (Deterioration) |
|---|---|---|---|
| AUROC | (Target > 0.82) | 0.835 (0.834, 0.835) | 0.828 (0.827, 0.829) |
| Moderate-risk threshold (eCART ≥93) | |||
| Sensitivity | (Target ~48-52%) | 51.8% (51.7%, 51.8%) | 48.8% (48.7%, 49.0%) |
| Specificity | (Target ~93-94%) | 93.1% (93.1%, 93.1%) | 93.3% (93.3%, 93.3%) |
| PPV | (Target ~8-9%) | 9.0% (9.0%, 9.1%) | 8.9% (8.8%, 8.9%) |
| NPV | (Target ~99%) | 99.3% (99.3%, 99.3%) | 99.3% (99.3%, 99.3%) |
| High-risk threshold (eCART ≥97) | |||
| Sensitivity | (Target ~33-38 %) | 38.6% (38.5%, 38.7%) | 33.7% (33.6%, 33.9%) |
| Specificity | (Target ~96-97%) | 96.9% (96.9%, 96.9%) | 97.3% (97.3%, 97.3%) |
| PPV | (Target ~14%) | 14.2% (14.1%, 14.2%) | 14.2% (14.1%, 14.3%) |
| NPV | (Target ~99%) | 99.2% (99.2%, 99.2%) | 99.1% (99.1%, 99.1%) |
Note: The "Acceptance Criteria (Implicit Target)" values are inferred based on the consistently reported values that demonstrate performance above random chance and clinical utility for risk stratification. The document does not explicitly state pre-defined quantitative acceptance criteria but rather presents the achieved performance as a demonstration of substantial equivalence.
Study Details
-
Sample sizes used for the test set and data provenance:
- Retrospective Test Set:
- Encounters (N): 1,769,461 unique hospitalizations.
- Observations (n): 132,873,833 eCART scores.
- Unique Patients: 934,454
- Data Provenance: Admissions between 2009 and 2023 from three geographically distinct health systems. The specific countries are not mentioned, but "US" is inferred from typical FDA submissions. It is retrospective.
- Prospective Test Set:
- Encounters (N): 205,946 unique hospitalizations.
- Observations (n): 21,516,964 eCART scores.
- Unique Patients: 151,233
- Data Provenance: Non-overlapping admissions between 2023 and 2024 from the same three healthcare systems as the retrospective analysis. It is prospective.
- Retrospective Test Set:
-
Number of experts used to establish the ground truth for the test set and qualifications of those experts:
- The document does not specify the number of experts used or their qualifications for establishing ground truth. The ground truth (death or ICU transfer) appears to be derived directly from Electronic Health Record (EHR) data, which is objective outcome data, rather than requiring expert labeling.
-
Adjudication method for the test set:
- The document does not specify an adjudication method. Given that the ground truth is death or ICU transfer from EHR, a formal adjudication process involving multiple experts for each case may not have been necessary, as these are typically clear clinical outcomes documented in the EHR.
-
If a multi-reader multi-case (MRMC) comparative effectiveness study was done, and the effect size:
- A multi-reader multi-case (MRMC) comparative effectiveness study was not explicitly mentioned as being performed to compare human readers with and without AI assistance. The performance data presented is for the standalone algorithm.
-
If a standalone (i.e., algorithm only without human-in-the-loop performance) was done:
- Yes, a standalone study was presented. The performance metrics (AUROC, Sensitivity, Specificity, PPV, NPV) are reported for the eCART algorithm itself, without human intervention in the reported performance. The device is intended as a "clinical decision support device" to "aid clinical teams in identifying which patients are most likely to clinically deteriorate."
-
The type of ground truth used:
- The ground truth used is outcomes data derived from the Electronic Health Record (EHR). Specifically, "Deterioration is defined as death or ward to ICU transfer within 24 hours following a score." Mortality is defined as "death within 24 hours following a score." These are objective clinical events.
-
The sample size for the training set:
- The document states: "eCART's algorithm was trained on ward patients..." but does not explicitly provide the sample size of the training set. It only provides details for the retrospective and prospective validation cohorts which are distinct from the training set.
-
How the ground truth for the training set was established:
- The document does not explicitly detail how the ground truth for the training set was established. However, given the nature of the ground truth for the test set (death or ICU transfer from validated EHR data), it is reasonable to infer that the ground truth for the training set would have been established similarly using objective patient outcomes data from EHRs.
§ 870.2210 Adjunctive predictive cardiovascular indicator.
(a)
Identification. The adjunctive predictive cardiovascular indicator is a prescription device that uses software algorithms to analyze cardiovascular vital signs and predict future cardiovascular status or events. This device is intended for adjunctive use with other physical vital sign parameters and patient information and is not intended to independently direct therapy.(b)
Classification. Class II (special controls). The special controls for this device are:(1) A software description and the results of verification and validation testing based on a comprehensive hazard analysis and risk assessment must be provided, including:
(i) A full characterization of the software technical parameters, including algorithms;
(ii) A description of the expected impact of all applicable sensor acquisition hardware characteristics and associated hardware specifications;
(iii) A description of sensor data quality control measures;
(iv) A description of all mitigations for user error or failure of any subsystem components (including signal detection, signal analysis, data display, and storage) on output accuracy;
(v) A description of the expected time to patient status or clinical event for all expected outputs, accounting for differences in patient condition and environment; and
(vi) The sensitivity, specificity, positive predictive value, and negative predictive value in both percentage and number form.
(2) A scientific justification for the validity of the predictive cardiovascular indicator algorithm(s) must be provided. This justification must include verification of the algorithm calculations and validation using an independent data set.
(3) A human factors and usability engineering assessment must be provided that evaluates the risk of misinterpretation of device output.
(4) A clinical data assessment must be provided. This assessment must fulfill the following:
(i) The assessment must include a summary of the clinical data used, including source, patient demographics, and any techniques used for annotating and separating the data.
(ii) The clinical data must be representative of the intended use population for the device. Any selection criteria or sample limitations must be fully described and justified.
(iii) The assessment must demonstrate output consistency using the expected range of data sources and data quality encountered in the intended use population and environment.
(iv) The assessment must evaluate how the device output correlates with the predicted event or status.
(5) Labeling must include:
(i) A description of what the device measures and outputs to the user;
(ii) Warnings identifying sensor acquisition factors that may impact measurement results;
(iii) Guidance for interpretation of the measurements, including a statement that the output is adjunctive to other physical vital sign parameters and patient information;
(iv) A specific time or a range of times before the predicted patient status or clinical event occurs, accounting for differences in patient condition and environment;
(v) Key assumptions made during calculation of the output;
(vi) The type(s) of sensor data used, including specification of compatible sensors for data acquisition;
(vii) The expected performance of the device for all intended use populations and environments; and
(viii) Relevant characteristics of the patients studied in the clinical validation (including age, gender, race or ethnicity, and patient condition) and a summary of validation results.