(260 days)
The Cytovale IntelliSep test is a semi-quantitative test that assesses cellular host response via deformability cytometry of leukocyte biophysical properties and is intended for use in conjunction with clinical assessments and laboratory findings to aid in the early detection of sepsis with organ dysfunction manifesting within the first 3 days after testing. It is indicated for use in adult patients with signs and symptoms of infection who present to the Emergency Department. The test is performed on an EDTA anticoagulated whole blood sample.
The IntelliSep test generates an IntelliSep Index value that falls within one of three discrete interpretation bands based on the probability of sepsis with organ dysfunction manifesting within the first three days after testing. The IntelliSep test represents the probability of the clinical syndrome of sepsis and is intended to be used alongside other clinical information and clinical judgement. It does not identify the causative agent of infection and should not be used as the sole basis to determine the presence of sepsis. The IntelliSep test is intended for in vitro diagnostic use.
The Cytovale IntelliSep test is a short turn-around time (STAT) test, producing results in 10 minutes or less, to aid in the early identification of patients at risk for having or developing sepsis within three (3) days of testing. It assesses the state of immune activation in patients with clinical suspicion of infection who present in the Emergency Department (ED).
The IntelliSep test is run on the Cytovale System, a laboratory benchtop analyzer depicted in Figure 1. The Cytovale System is a closed system benchtop analyzer, comprised of three modules: Sample Preparation Module, Cell Imaging Module, and Imaging Analysis Module.
To run a test, the laboratory operator transfers 100 µL of whole blood into the sample preparation tube which is then placed into the Cytovale System. The system automatically lyses red blood cells, and washes the purified leukocytes in a diluent, producing a total volume of approximately 1mL of prepared sample, which the operator then transfers to the IntelliSep cartridge for analysis on the Cytovale System.
A microfluidic deformability cytometry technique is used to measure the biophysical properties of thousands of individual leukocytes in rapid succession. These properties have been shown to differ in quiescent white blood cell populations when compared to those in septic patients, enabling for rapid assessment of the host response and the likelihood of having or developing sepsis. Based on these measurements, the test provides a single score, the IntelliSep Index (ISI), ranging from 0.1-10.0, stratified into three discrete interpretation bands (Band 1, Band 2, Band 3) of sepsis likelihood.
Here's a breakdown of the acceptance criteria and study details for the Cytovale IntelliSep test, based on the provided document:
Acceptance Criteria and Device Performance
The document does not explicitly state pre-defined quantitative acceptance criteria in a pass/fail format for clinical performance endpoints (e.g., a specific PPV or likelihood ratio threshold). Instead, it focuses on demonstrating a clear relationship between the IntelliSep Index (ISI) and the likelihood of sepsis, and the statistical distinctness of the interpretation bands. For analytical performance, criteria are generally established for standard deviations within precision studies and confidence intervals for reproducibility and carry-over.
Here's a table summarizing the reported device performance, categorized by the relevant studies:
Clinical Performance
Implicit Acceptance Criterion: A clear relationship between ISI and the increasing likelihood of sepsis, and statistically distinct interpretation bands (non-overlapping 80% confidence intervals between bands).
| Performance Metric | IntelliSep Interpretation Band 1 (Lower Likelihood) | IntelliSep Interpretation Band 2 (Moderate Likelihood) | IntelliSep Interpretation Band 3 (Higher Likelihood) |
|---|---|---|---|
| Sepsis Predictive Value (Sepsis-3, Forced Adjudication) | 11.1% (8.6%, 14.1% CI) | 28.1% (23.5%, 33.2% CI) | 49.4% (44.0%, 54.7% CI) |
| Sepsis Likelihood Ratio (Sepsis-3, Forced Adjudication) | 0.35 | 1.08 | 2.69 |
| Sepsis Predictive Value (Sepsis-2, Forced Adjudication) | 28.0% (24.0%, 32.4% CI) | 59.5% (53.9%, 64.9% CI) | 72.9% (67.8%, 77.6% CI) |
| Sepsis Likelihood Ratio (Sepsis-2, Forced Adjudication) | 0.38 | 1.44 | 2.64 |
| Sepsis Predictive Value (Severe Sepsis-2, Forced Adjudication) | 15.9% (12.7%, 19.6% CI) | 43.9% (38.4%, 49.5% CI) | 62.6% (57.2%, 67.7% CI) |
| Sepsis Likelihood Ratio (Severe Sepsis-2, Forced Adjudication) | 0.31 | 1.28 | 2.74 |
Note: The clinical study explicitly states: "It is important to note that the algorithm for calculating the ISI ranges for the interpretation bands were defined based on prior sepsis-focused studies. No information from patients in this study influenced the calculations or interpretation bands of the ISI." and "Irrespective of the comparator scheme chosen, the probability of sepsis in the three ISI IntelliSep test Interpretation Bands was statistically distinct, defined as non-overlapping 80% confidence intervals between the bands." This suggests the demonstration of statistically distinct bands with appropriate clinical trends was the primary clinical performance criterion.
Analytical Performance
| Performance Metric | Stated Acceptance Criteria | Reported Performance |
|---|---|---|
| Within-Laboratory Precision (Total, across all samples) | (Implicitly, standard deviations demonstrating acceptable variability) | Standard Deviation: 0.48 for 144 tests across 3 sites (range from 0.26 to 0.96 for individual samples) |
| Reproducibility (Total) | (Implicitly, standard deviations demonstrating acceptable variability) | Standard Deviation: 0.57 (combining within-lab 0.48 and across sites 0.30) |
| Reagents Lot-to-Lot Reproducibility | The 90% CI Lower and Upper Bounds are within (-1.0, 1.0) units of ISI | QC Level 1: (-0.12, 0.24) - Pass |
| QC Level 2: (-0.21, 0.19) - Pass | ||
| Cartridge Lot-to-Lot Reproducibility | The 90% CI Lower and Upper Bounds are within (-1.0, 1.0) units of ISI | Donors: (-0.68, 0.26) - Pass |
| Interfering Substances | No impact on results observed (up to listed concentrations) | Bilirubin (Unconjugated/Conjugated), Triglycerides, Hemolysate, Hemoglobin: No impact observed on ISI. |
| Sample Carry-Over | No sample carry-over effect found | Total ISI Difference (LL-HL): 0.03 |
| Total ISI Difference (HH-LH): 0.02 (No effect found) | ||
| Onboard Stability of Reagents | Stable for up to 30 days | Stable for up to 30 days when stored onboard the Cytovale System. |
| Sample Stability (K2-EDTA) | May be used for up to 5 hours after blood draw | May be used for up to 5 hours after blood draw. |
Study Details
-
Sample sizes used for the test set and the data provenance:
- Test Set Sample Size: 572 evaluable subjects were included in the primary analyses for clinical performance.
- Data Provenance: The study was a "blinded, prospective, observational, multicenter cohort study" conducted at five hospitals at four geographically dispersed sites in the United States. The subjects were "representative of the adult population (≥ 18 years of age) typical of those presenting to EDs across the United States."
-
Number of experts used to establish the ground truth for the test set and the qualifications of those experts:
- Number of Experts: A multi-tiered adjudication process was used:
- Initially, two independent adjudicators reviewed each case.
- If there was disagreement, a third adjudicator was involved in a consensus meeting.
- Qualifications of Experts: Physicians with board certification in Medical or Surgical Critical Care (CC), Infectious Diseases (ID), Emergency Medicine (EM), or Internal Medicine (IM) or related fields, from different institutions. They were not participating as an Investigator in the Study.
- Number of Experts: A multi-tiered adjudication process was used:
-
Adjudication method (e.g. 2+1, 3+1, none) for the test set:
- The adjudication method was 2+1 (two independent adjudicators, with a third used for arbitration in case of disagreement).
- Outcomes were categorized as 'unanimous' (both initial adjudicators agreed), 'consensus' (third adjudicator facilitated agreement), or 'forced' (majority two out of three if consensus couldn't be reached). The performance data is presented for all three schemes (forced, consensus, unanimous).
-
If a multi-reader multi-case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance:
- No, an MRMC comparative effectiveness study was not explicitly stated or performed. The study evaluated the IntelliSep test as an aid in conjunction with clinical assessments and laboratory findings, but it did not directly compare human reader performance with and without AI assistance (i.e., IntelliSep test results). The test provides a quantitative index and interpretation bands, not an image for interpretation by human readers.
-
If a standalone (i.e. algorithm only without human-in-the-loop performance) was done:
- Yes, a standalone performance evaluation was done. The IntelliSep test generates an "IntelliSep Index" (ISI) automatically from the measured biophysical properties of leukocytes. The clinical performance section directly assesses the predictive values and likelihood ratios of this automatically generated ISI, independent of real-time human interpretation during the study (treating physicians were unaware of study enrollment and IntelliSep test results). The test is intended for use in conjunction with clinical assessments, but its performance metrics are derived from the algorithm's output itself.
-
The type of ground truth used (expert consensus, pathology, outcomes data, etc.):
- The ground truth was established through retrospective physician adjudication based on consensus definitions for Sepsis-2 and Sepsis-3. This involved a multi-tiered process by qualified physicians:
- Collection of detailed patient information (demographics, labs, vitals, medical history, hospital encounter, infectious disease, medications, discharge disposition, SOFA scores, clinical impression).
- Compilation into a Case Report Summary (CRS).
- Review by independent adjudicators with arbitration for discordant results.
- It's a form of expert consensus based on clinical data.
- The ground truth was established through retrospective physician adjudication based on consensus definitions for Sepsis-2 and Sepsis-3. This involved a multi-tiered process by qualified physicians:
-
The sample size for the training set:
- The document states: "It is important to note that the algorithm for calculating the ISI ranges for the interpretation bands were defined based on prior sepsis-focused studies. No information from patients in this study influenced the calculations or interpretation bands of the ISI."
- This implies the training of the algorithm and definition of interpretation bands occurred prior to and separate from this specific clinical validation study. The sample size for the training set is not provided in this document.
-
How the ground truth for the training set was established:
- Similar to the point above, the document only mentions that the ISI algorithm and interpretation bands were "defined based on prior sepsis-focused studies."
- The method for establishing ground truth for those prior training studies is not detailed in this document.
§ 866.3215 Device to detect and measure non-microbial analyte(s) in human clinical specimens to aid in assessment of patients with suspected sepsis.
(a)
Identification. A device to detect and measure non-microbial analyte(s) in human clinical specimens to aid in assessment of patients with suspected sepsis is identified as an in vitro device intended for the detection and qualitative and/or quantitative measurement of one or more non-microbial analytes in human clinical specimens to aid in the assessment of patients with suspected sepsis when used in conjunction with clinical signs and symptoms and other clinical and laboratory findings.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Premarket notification submissions must include the device's detailed Indications for Use statement describing what the device detects and measures, the results provided to the user, whether the measure is qualitative and/or quantitative, the clinical indications for which the test is to be used, and the specific population(s) for which the device use is intended.
(2) Premarket notification submissions must include detailed documentation of the device description, including (as applicable), all device components, software, ancillary reagents required but not provided, explanation of the device principle and methodology, and for molecular devices include detailed documentation of the primer/probe sequence, design, and rationale for sequence selection.
(3) Premarket notification submissions must include detailed documentation of applicable analytical studies, such as, analytical sensitivity (Limit of Detection, Limit of Blank, and Limit of Quantitation), precision, reproducibility, analytical measuring range, interference, cross-reactivity, and specimen stability.
(4) Premarket notification submissions must include detailed documentation of a prospective clinical study or, if appropriate, results from an equivalent sample set. This detailed documentation must include the following information:
(i) Results must demonstrate adequate device performance relative to a well-accepted comparator.
(ii) Clinical sample results must demonstrate consistency of device output throughout the device measuring range likely to be encountered in the Intended Use population.
(iii) Clinical study documentation must include the original study protocol (including predefined statistical analysis plan), study report documenting support for the Indications for Use(s), and results of all statistical analyses.
(5) Premarket notification submissions must include evaluation of the level of the non-microbial analyte in asymptomatic patients with demographic characteristics (
e.g., age, racial, ethnic, and gender distribution) similar to the Intended Use population.(6) As part of the risk management activities performed under 21 CFR 820.30 design controls, you must document an appropriate end user device training program that will be offered as part of your efforts to mitigate the risk of failure to correctly operate the instrument.
(7) A detailed explanation of the interpretation of results and acceptance criteria must be included in the device's 21 CFR 809.10(b)(9) compliant labeling, and a detailed explanation of the interpretation of the limitations of the samples (
e.g., collected on day of diagnosis) must be included in the device's 21 CFR 809.10(b)(10) compliant labeling.