(204 days)
The Unyvero LRT Application is a qualitative nucleic acid multiplex test intended for the simultaneous detection and identification of nucleic acid sequences from the following microorganisms and antibiotic resistance markers in endotracheal aspirates from adult hospitalized patients with suspected lower respiratory tract infections.
The Unyvero LRT Application performed on the Unyvero System is indicated as an aid in the diagnosis of lower respiratory tract infection in adult hospitalized patients with signs and symptoms of lower respiratory infection: results should be used in conjunction with other clinical and laboratory findings. As tracheal aspirates commonly contain colonizing microorganisms, detection of Unyvero LRT microbial targets does not indicate that the microorganism is the cause of the disease. Unyvero positive results do not rule out co-infection with microorganisms not detected by the Unyvero LRT Application. Negative results do not preclude lower respiratory infection, as the causative agent may be a microorganism not detected by this test.
A negative result for any antibiotic resistance marker does not indicate that detected microorganisms are susceptible to applicable antimicrobial agents. Detected resistance markers cannot be definitively linked to specific microorganisms, and may be present in organisms that are not detected by the Unyvero LRT Application such as organisms present as colonizing or normal flora.
Microbiology cultures of aspirates should be performed to obtain isolates for species identification and antimicrobial susceptibility testing, to differentiate quantities of identified microorganisms as well as normal flora present in the specimen and to identify potential microorganisms not targeted by the Unyvero LRT Application.
The Unyvero LRT Application is a qualitative test that includes specimen processing, genomic bacterial DNA isolation and purification, multiplex PCR and array hybridization and detection. The Unyvero LRT Application performed using the Unyvero System detects specific nucleic acid sequences from microorganisms and resistance markers in tracheal aspirates collected from patients with signs and symptoms of lower respiratory infection.
The Unyvero LRT Application consists of the following components:
- Unyvero LRT Cartridge: Contains DNA isolation and purification reagents, a DNA isolation column, eight separate PCR chambers with eight corresponding detection arrays. The Cartridge also contains fluorescently-labeled primers, hybridization and wash buffers and oligonucleotide probes for detection of targeted PCR products using array hybridization technology.
- Unyvero T1 Sample Tube: Contains glass beads and buffers to lyse bacteria and liquefy the sample.
- Unyvero T1 Sample Tube Cap (with Internal Control): Contains proteinase K and a synthetic internal control gene for process monitoring. The T1 Sample Tube Cap seals the Unyvero Sample Tube after which the internal control is combined with each patient specimen. The internal control DNA sequence does not have significant homology to targeted sequences and is amplified independently in each of the eight PCR chambers and the amplified internal control product is hybridized on each array.
- Unyvero M1 Master Mix: Contains reagents for DNA amplification.
- Unyvero T1 Transfer Tool: The Transfer tool can be used to transfer viscous specimens from the primary sample container to the Unyvero Sample Tube.
The Unyvero System consists of the following components:
- Unyvero Lysator: The Lysator lyses the specimen and can process up to four specimens simultaneously in four separate slots.
- Unyvero Analyzer: The Analyzer automates DNA purification, amplification and detection. Each Analyzer can simultaneously process up to two Unyvero Cartridges with each slot available using random access.
- Unyvero Cockpit: The Cockpit provides the main user interface for the Unyvero System, guides the user through the steps to run the Unvvero LRT Application and automatically generates and displays test results. The Cockpit is equipped with a high-resolution touch screen and a barcode reader.
- Unyvero Sample Tube Holder: The Sample Tube holder holds the Sample Tube securely while the specimen is transferred into the Sample Tube.
Here's a breakdown of the acceptance criteria and study proving the device meets them, based on the provided text:
Device: Unyvero Lower Respiratory Tract (LRT) Application and Unyvero System (Qualitative nucleic acid amplification assay)
Purpose: Simultaneous detection and identification of nucleic acid sequences from specified microorganisms and antibiotic resistance markers in endotracheal aspirates from adult hospitalized patients with suspected lower respiratory tract infections.
1. Table of Acceptance Criteria and Reported Device Performance
The document does not explicitly state pre-defined acceptance criteria in a quantitative table format (e.g., "PPA must be >90%"). However, it describes the performance observed in the validation studies and then concludes that this performance is "acceptable" in conjunction with various limitations and recommended practices. The implicit acceptance criteria appear to be the demonstrated performance characteristics from the clinical studies.
Below is a table summarizing the reported device performance for organism detection (compared to a composite comparator, which is generally the most stringent comparison) and resistance marker detection (with software masking). The document doesn't provide a single, overarching table of "acceptance criteria" but rather details performance metrics across various studies.
Table of Reported Device Performance (Key Clinical Metrics)
| Target (Type) | Performance Metric (vs. Composite Comparator) | Value (95% CI) | Notes |
|---|---|---|---|
| Microorganism Targets | (Prospective Study, comparison to Composite Comparator unless otherwise noted) | ||
| Acinetobacter spp. | PPA | 95.8% (79.8 - 99.3%) | Also 100% PPA vs. Culture for numerous, moderate, abundant categories; 37.0% PPV vs. Culture (likely due to DNA vs. viable organism) |
| Chlamydia pneumoniae | PPA | na (0/0 positive cases) | 100.0% NPA (99.4 - 100.0%) for 603 cases |
| Citrobacter freundii | PPA | 16.7% (3.0 - 56.3%) | Very low PPA in prospective study; complemented by 86.0% PPA in contrived study. |
| Enterobacter cloacae complex | PPA | 94.4% (74.2 - 99.0%) | |
| Escherichia coli | PPA | 97.4% (86.5 - 99.5%) | |
| Haemophilus influenzae | PPA | 88.2% (65.7 - 96.7%) | |
| Klebsiella oxytoca | PPA | 77.8% (45.3 - 93.7%) | Complemented by 89.3% PPA in contrived study. |
| Klebsiella pneumoniae | PPA | 90.9% (76.4 - 96.9%) | |
| Klebsiella variicola | PPA | 100.0% (34.2 - 100.0%) | Low number of positive cases (2/2 prospectively); complemented by 96.4% PPA in contrived study. |
| Legionella pneumophila | PPA | 100.0% (34.2 - 100.0%) | Low number of positive cases (2/2 prospectively); complemented by 100.0% PPA in contrived study. |
| Moraxella catarrhalis | PPA | 52.2% (33.0 - 70.8%) | Notably low PPA in prospective study; 96.0% PPA in contrived study. High FP rate against culture. |
| Morganella morganii | PPA | 85.7% (48.7 - 97.4%) | Complemented by 95.6% PPA in contrived study. |
| Mycoplasma pneumoniae | PPA | 100.0% (34.2 - 100.0%) | Low number of positive cases (2/2 prospectively); complemented by 100.0% PPA in contrived study. |
| Proteus spp. | PPA | 96.0% (80.5 - 99.3%) | |
| Pseudomonas aeruginosa | PPA | 89.4% (81.1 - 94.3%) | |
| Serratia marcescens | PPA | 87.5% (69.0 - 95.7%) | |
| Staphylococcus aureus | PPA | 91.6% (85.2 - 95.4%) | |
| Stenotrophomonas maltophilia | PPA | 89.3% (78.5 - 95.0%) | |
| Streptococcus pneumoniae | PPA | 62.5% (38.6 - 81.5%) | Notably low PPA in prospective study. |
| Antibiotic Resistance Markers | (Prospective Study, comparison to molecular comparator with software masking applied for host microorganism) | ||
| ctx-M | PPA | 93.8% (71.7 - 98.9%) | |
| kpc | PPA | 100.0% (61.0 - 100.0%) | |
| ndm | PPA | na (0/0 positive cases) | |
| oxa-23 | PPA | 85.7% (48.7 - 97.4%) | |
| oxa-24 | PPA | 100.0% (34.2 - 100.0%) | Low number of positive cases (2/2) |
| oxa-48 | PPA | na (0/0 positive cases) | |
| oxa-58 | PPA | na (0/0 positive cases) | |
| tem | PPA | 100.0% (67.6 - 100.0%) | This is for tem detected when H. influenzae is also detected. Overall tem PPA (without masking) was 100.0% (93.4 - 100.0%) for 54/54 cases. |
| vim | PPA | 100.0% (34.2 - 100.0%) | Low number of positive cases (2/2) |
| mecA | PPA | 91.5% (81.6 - 96.3%) | This is for mecA detected when S. aureus is also detected. Overall mecA PPA (without masking) was 87.1% (80.1 - 91.9%) for 108/124 cases. |
2. Sample Size Used for the Test Set and Data Provenance
- Test Set (Clinical Studies):
- Prospective Study: 603 evaluable tracheal aspirate specimens.
- Provenance: Multi-center study at nine clinical sites in the United States.
- Nature: Prospectively collected.
- Retrospective (Archived) Study: 369 evaluable archived specimens (211 from US study sites, 158 from other US or European sites tested in-house at Curetis).
- Provenance: US and European sites.
- Nature: Previously frozen, retrospective.
- Contrived Clinical Study: Ranged from 21 to 50 specimens for each microorganism or resistance marker.
- Provenance: Prepared and tested at three sites in the United States and in-house at Curetis.
- Nature: Contrived specimens (spiked into natural tracheal aspirate matrices).
- Prospective Study: 603 evaluable tracheal aspirate specimens.
3. Number of Experts Used to Establish Ground Truth for the Test Set and Qualifications of those Experts
The document does not specify the number or qualifications of experts involved in establishing the ground truth for the clinical studies.
- For "typical" microorganisms in the prospective study, the primary ground truth was standard-of-care (SoC) tracheal aspirate culture. For the composite comparator, it also included independent and validated multiplexed PCR assays with bi-directional sequencing.
- For "atypical" microorganisms, the ground truth was two independent and validated multiplex PCR assays with bi-directional sequencing.
- For antibiotic resistance markers, the ground truth was validated multiplex PCR assays followed by bi-directional sequencing.
- For comparative phenotypic analysis of mecA, phenotypic antimicrobial susceptibility (AST) testing results from established culture methods were used.
The validation of these comparator methods (LoDs, inclusivity) is mentioned, implying expert oversight, but specific expert count or detailed qualifications are not provided.
4. Adjudication Method for the Test Set
The document describes the "Composite Comparator" method as an adjudication of sorts:
- For 'typical' microorganisms: A specimen was considered positive if culture was positive OR if the validated comparator PCR and follow-up bi-directional sequencing was positive. Any specimen negative by both culture and PCR was considered negative. This is a form of "consensus" or "best clinical practice" ground truth, where molecular confirmation is used to bolster culture results, or establish positivity where culture might miss (e.g., non-viable organisms, difficult to culture).
- For 'atypical' microorganisms: Specimens positive by either of the two PCR/sequencing assays were considered positive. Those negative by both were considered negative. This implies a "molecular consensus" approach.
- For 'false positives' against culture: False positive LRT results were analyzed with molecular assays (PCR/bi-directional sequencing) using sample DNA extracts to confirm presence or absence of microorganisms. This acts as a re-adjudication/confirmation step for discordant results.
There is no mention of human expert adjudication panels in the way it might be done for image-based AI devices (e.g., 2+1 radiologist review). The adjudication is inherently built into the definition of the reference/comparator method, often relying on molecular methods to resolve discrepancies or provide more sensitive ground truth than traditional culture.
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was done
No, an MRMC comparative effectiveness study was not done for this device. This device is a diagnostic assay for molecular detection, not an AI-assisted diagnostic aid for human readers/interpreters in the typical MRMC study design (e.g., radiology AI). The study proves the standalone performance of the device.
6. If a Standalone (algorithm only without human-in-the-loop performance) was done
Yes, the study primarily evaluates the standalone performance of the Unyvero LRT Application. The reported PPA and NPA values for specific targets are measures of the device's accuracy in detecting nucleic acid sequences in the specimens, independent of human interpretation of the device's raw output. The device itself (Unyvero System with LRT Application) automates DNA purification, amplification, and detection, and automatically generates and displays results. The human involvement is in sample preparation and loading, and interpreting the final qualitative "positive/negative" result the device provides, rather than interpreting raw data alongside or without AI assistance.
7. The Type of Ground Truth Used
The ground truth used varied depending on the target:
- For 'typical' microorganisms:
- Primary Reference: Standard-of-care (SoC) tracheal aspirate culture.
- Composite Comparator: A combination of SoC culture PLUS independent and validated multiplexed PCR assays with bi-directional sequencing. This represents a more sensitive and comprehensive "true presence" ground truth, especially for organisms that may be difficult to culture or present as non-viable DNA.
- For 'atypical' microorganisms: Two independent and validated multiplex PCR assays with bi-directional sequencing.
- For antibiotic resistance markers: Validated multiplex PCR assays followed by bi-directional sequencing.
- For genotypic linkage and phenotypic correlation: Molecular sequencing of cultured isolates for the presence of the resistance marker gene, and phenotypic antimicrobial susceptibility testing (AST) results from those isolates.
8. The Sample Size for the Training Set
The document does not explicitly mention a "training set" in the context of machine learning or AI models. Given that this is a PCR-based diagnostic assay, it's developed through traditional molecular assay design and optimization, not typically through supervised machine learning with distinct training and test sets as seen in AI imaging. The "development" or "optimization" of the assay (e.g., primer design, LoD determination, inclusivity/exclusivity testing) would constitute its "training" or optimization phase, but no sample size for this phase is provided. The data presented relates to the validation of the final assay.
9. How the Ground Truth for the Training Set Was Established
As noted above, there's no explicitly defined "training set" in the machine learning sense. The ground truth for the analytical studies and clinical validation studies (which would inform refinement if developmental iterations were considered "training") was established as follows:
- Analytical Studies (LoD, Reproducibility, Inclusivity, Exclusivity, Interference): Samples were contrived by spiking known concentrations of well-characterized reference strains (ATCC, NCTC, DSM, JMI, Micromyx, NRZ, UCLA, RKI, clinical isolates) into artificial respiratory matrix (ARM) or PBS. The "ground truth" here is the known identity and concentration of the spiked microorganisms/resistance markers.
- Clinical Studies (Prospective and Retrospective): As detailed in point 7, ground truth was established by:
- Standard-of-care (SoC) tracheal aspirate culture.
- Validated multiplex PCR assays with bi-directional sequencing.
- A composite of SoC culture and PCR/sequencing.
- Molecular sequencing of isolates and phenotypic AST.
This means the "ground truth" was established by conventional microbiological and molecular diagnostic methods, which were considered the gold standard for comparing the performance of the new Unyvero system.
§ 866.3985 Device to detect and identify microorganisms and associated resistance marker nucleic acids directly in respiratory specimens.
(a)
Identification. A device to detect and identify microorganisms and associated resistance marker nucleic acids directly from respiratory specimens is an in vitro diagnostic device intended for the detection and identification of microorganisms and associated resistance markers in respiratory specimens collected from patients with signs or symptoms of respiratory infection. The device is intended to aid in the diagnosis of respiratory infection in conjunction with clinical signs and symptoms and other laboratory findings. These devices do not provide confirmation of antibiotic susceptibility since mechanisms of resistance may exist other than those detected by the device.(b)
Classification. Class II (special controls). The special controls for this device are:(1) The intended use for the 21 CFR 809.10 labeling must include a detailed description of what the device detects, the type of results provided to the user, the clinical indications appropriate for test use, and the specific population(s) for which the device is intended.
(2) The 21 CFR 809.10(b) labeling must include:
(i) A detailed device description, including all device components, control elements incorporated into the test procedure, instrument requirements, ancillary reagents required but not provided, and a detailed explanation of the methodology, including all pre-analytical methods for processing of specimens.
(ii) Performance characteristics from analytical studies, including, but not limited to, limit of detection, inclusivity, reproducibility, cross reactivity, interfering substances, competitive inhibition, carryover/cross contamination, specimen stability, and linearity, as applicable.
(iii) A limiting statement that the device is intended to be used in conjunction with clinical history, signs and symptoms, and results of other diagnostic tests, including culture and antimicrobial susceptibility testing.
(iv) A detailed explanation of the interpretation of test results for clinical specimens and acceptance criteria for any quality control testing.
(v) A limiting statement that negative results for microorganisms do not preclude the possibility of infection, and should not be used as the sole basis for diagnosis, treatment, or other patient management decisions.
(vi) If applicable, a limiting statement that detected microorganisms may not be the cause of lower respiratory tract infection and may be indicative of colonizing or normal respiratory flora.
(vii) If applicable, a limiting statement that detection of resistance markers cannot be definitively linked to specific microorganisms and that the source of a detected resistance marker may be an organism not detected by the assay, including colonizing flora.
(viii) If applicable, a limiting statement that detection of antibiotic resistance markers may not correlate with phenotypic gene expression.
(3) The 21 CFR 809.10(b) labeling and any test report generated by the device must include a limiting statement that negative results for resistance markers do not indicate susceptibility of detected microorganisms.
(4) Design verification and validation must include:
(i) Performance characteristics from clinical studies that include prospective (sequential) samples and, if appropriate, additional characterized samples. The study must be performed on a study population consistent with the intended use population and compare the device performance to results obtained from an FDA accepted reference method and/or FDA accepted comparator method, as appropriate. Results from the clinical studies must include the clinical study protocol (including predefined statistical analysis plan, if applicable), clinical study report, and results of all statistical analyses.
(ii) A detailed device description including the following:
(A) Thorough description of the assay methodology including, but not limited to, primer/probe sequences, primer/probe design, and rationale for target sequence selection, as applicable.
(B) Algorithm used to generate a final result from raw data (e.g., how raw signals are converted into a reported result).
(iii) A detailed description of device software, including, but not limited to, validation activities and outcomes.
(iv) As part of the risk management activities, an appropriate end user device training program must be offered as an effort to mitigate the risk of failure from user error.