(297 days)
CLEWICU provides the clinician with physiological insight into a patient's likelihood of future hemodynamic instability. CLEWICU is intended for use with intensive care unit (ICU) patients 18 years and over. CLEWICU is considered to provide additional information regarding the patient's predicted future risk for clinical deterioration, as well as identifying patients at low risk for deterioration. The product predictions are for reference only and no therapeutic decisions should be made based solely on the CLEWICU predictions.
The CLEWICU System is a stand-alone analytical software product that includes the ClewICUServer and the ClewICUnitor. It uses models derived from machine learning to calculate the likelihood of occurrence of certain clinically significant events for patients in the intensive care unit (ICU). ClewICUServer and ClewICUnitor are software-only devices that are installed on user-provided hardware. The ClewICUServer is a backend software platform that imports patient data from various sources including Electronic Health Record (EHR) data and medical device data. The data are then used by models operating within the ClewICUServer to compute and store the CLEWHI index (likelihood of hemodynamic instability requiring vasopressor / inotrope support), and CLEWLR (indication that the patient is at "low risk" for deterioration). The ClewICUnitor is the web-based user interface displaying CLEWHI, and CLEWLR associated notifications and related measures, as well as presenting the overall unit status.
Here's a breakdown of the acceptance criteria and the study details for the CLEWICU System, based on the provided text:
1. Acceptance Criteria and Reported Device Performance
The acceptance criteria are implied by the reported performance metrics of the device, particularly the ranges for the 95% Confidence Intervals (CI). The study aimed to demonstrate acceptable performance for predicting hemodynamic instability (CLEWHI) and identifying low-risk patients (CLEWLR).
| Metric | Acceptance Criteria (Implied by 95% CI) | Reported Device Performance |
|---|---|---|
| Hemodynamic Instability Model (CLEWHI) | ||
| Sensitivity | ≥ 56.9% | 60.6% |
| Positive Predictive Value (PPV) | ≥ 20.7% | 22.3% |
| Lead-time (true positive alerts) | Not explicitly quantified as a range, but reported as central tendencies for acceptable lead time. | Median: 3.0 hours, 25th Percentile: 1.6 hours, 75th Percentile: 4.8 hours |
| Low Risk Model (CLEWLR) | ||
| Specificity | ≥ 94.8% | 95.7% |
| Sensitivity | ≥ 21.2% | 21.4% |
2. Sample Size Used for the Test Set and Data Provenance
- Sample Size (Test Set): Not explicitly stated as a single number. The study utilized a dataset from the WakeMed Health System, including patient stays in 7 intensive care units across 2 hospitals. The number of patients or patient stays for the retrospective clinical validation is not precisely quantified, but it was used for both training and validation.
- Data Provenance:
- Country of Origin: United States (WakeMed Health System indicated).
- Retrospective or Prospective: Retrospective.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications of Those Experts
- Number of Experts: Not explicitly stated as a specific number. The text mentions a "tagging system was developed and validated (against human physician readers as ground truth)." This implies multiple physician readers were involved in establishing or validating the ground truth for selected cases.
- Qualifications of Experts: "Human physician readers." No further specific qualifications (e.g., years of experience, subspecialty) are provided.
4. Adjudication Method for the Test Set
- Adjudication Method: Not explicitly detailed. The text states "a tagging system was developed and validated (against human physician readers as ground truth)." This suggests an indirect method where the tagging system learned from or was compared against physician assessments, rather than direct physician adjudication of every case in the test set.
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
- MRMC Study: No, an MRMC comparative effectiveness study was not explicitly mentioned or described. The study focused on the performance of the standalone AI system.
- Effect Size with AI vs. Without AI Assistance: Not applicable, as an MRMC study was not performed.
6. Standalone (Algorithm Only Without Human-in-the-Loop Performance) Study
- Standalone Study: Yes, the described clinical validation is a standalone performance study. The reported metrics (Sensitivity, PPV, Specificity, Lead-time) directly reflect the algorithm's performance without explicit human intervention or assistance during the evaluation. The device is described as "a stand-alone analytical software product."
7. Type of Ground Truth Used
- Type of Ground Truth: The ground truth was established by "a tagging system... validated (against human physician readers as ground truth)." This suggests a hybrid approach where an automated tagging system, verified by human expert consensus (physician readers), was used to create the labels for the "events of interest" (hemodynamic instability, low risk).
8. Sample Size for the Training Set
- Sample Size (Training Set): Not explicitly stated as a specific number. The text mentions "the WakeMed dataset included patient stays in 7 intensive care units across 2 hospitals between 5 November 2019 and 30 June 2020." This dataset was used for "training of the CLEWICU predictive models," but the specific portion or number of cases allocated for training is not provided.
9. How the Ground Truth for the Training Set Was Established
- Ground Truth for Training Set: "Once validated, the tagging system was used to generate the clinical truth labels needed, both for training of the CLEWICU predictive models and for validation of the clinical performance of those models." This indicates that the same "tagging system" (validated against human physician readers) was used to establish the ground truth for the training set.
§ 870.2210 Adjunctive predictive cardiovascular indicator.
(a)
Identification. The adjunctive predictive cardiovascular indicator is a prescription device that uses software algorithms to analyze cardiovascular vital signs and predict future cardiovascular status or events. This device is intended for adjunctive use with other physical vital sign parameters and patient information and is not intended to independently direct therapy.(b)
Classification. Class II (special controls). The special controls for this device are:(1) A software description and the results of verification and validation testing based on a comprehensive hazard analysis and risk assessment must be provided, including:
(i) A full characterization of the software technical parameters, including algorithms;
(ii) A description of the expected impact of all applicable sensor acquisition hardware characteristics and associated hardware specifications;
(iii) A description of sensor data quality control measures;
(iv) A description of all mitigations for user error or failure of any subsystem components (including signal detection, signal analysis, data display, and storage) on output accuracy;
(v) A description of the expected time to patient status or clinical event for all expected outputs, accounting for differences in patient condition and environment; and
(vi) The sensitivity, specificity, positive predictive value, and negative predictive value in both percentage and number form.
(2) A scientific justification for the validity of the predictive cardiovascular indicator algorithm(s) must be provided. This justification must include verification of the algorithm calculations and validation using an independent data set.
(3) A human factors and usability engineering assessment must be provided that evaluates the risk of misinterpretation of device output.
(4) A clinical data assessment must be provided. This assessment must fulfill the following:
(i) The assessment must include a summary of the clinical data used, including source, patient demographics, and any techniques used for annotating and separating the data.
(ii) The clinical data must be representative of the intended use population for the device. Any selection criteria or sample limitations must be fully described and justified.
(iii) The assessment must demonstrate output consistency using the expected range of data sources and data quality encountered in the intended use population and environment.
(iv) The assessment must evaluate how the device output correlates with the predicted event or status.
(5) Labeling must include:
(i) A description of what the device measures and outputs to the user;
(ii) Warnings identifying sensor acquisition factors that may impact measurement results;
(iii) Guidance for interpretation of the measurements, including a statement that the output is adjunctive to other physical vital sign parameters and patient information;
(iv) A specific time or a range of times before the predicted patient status or clinical event occurs, accounting for differences in patient condition and environment;
(v) Key assumptions made during calculation of the output;
(vi) The type(s) of sensor data used, including specification of compatible sensors for data acquisition;
(vii) The expected performance of the device for all intended use populations and environments; and
(viii) Relevant characteristics of the patients studied in the clinical validation (including age, gender, race or ethnicity, and patient condition) and a summary of validation results.