(147 days)
The LIAISON PLEX Respiratory Flex (RSP Flex) Assay is a multiplexed qualitative test for the simultaneous in vitro detection and identification of multiple bacterial and viral nucleic acids in nasopharyngeal swabs (NPS) obtained from individuals with clinical signs and symptoms of respiratory tract infection, including SARS-CoV-2. The test is performed on the automated LIAISON PLEX System utilizing reverse transcription (RT), polymerase chain reaction (PCR), and array hybridization to detect specific nucleic acid gene sequences of the following organism types and subtypes:
Viruses: Adenovirus Human Coronavirus (HKU1, NL63, OC43, and 229E not differentiated) Human Enterovirus/Rhinovirus (not differentiated) Human Metapneumovirus, Influenza A Influenza A (subtype H1) Influenza A (subtype H3) Influenza B Parainfluenza 1 Parainfluenza 2 Parainfluenza 3 Parainfluenza 4 Respiratory Syncytial Virus Severe Acute Respiratory Syndrome Coronavirus (SARS-CoV-2)
Bacteria: Bordetella holmesii Bordetella parapertussis Bordetella pertussis Chlamydia pneumoniae Mycoplasma pneumoniae
Nucleic acids from the bacterial and viral organisms identified by this test are generally detectable in NPS specimens during the acute phase of infection. Detecting and identifying specific bacterial and viral nucleic acids from individuals exhibiting signs and symptoms of respiratory infection aids in the diagnosis of respiratory infection, if used in conjunction with other clinical, epidemiological, and laboratory findings. The results of this test should not be used as the sole basis for diagnosis, treatment, or patient management decisions.
Negative results in the presence of a respiratory illness may be due to infection with pathogens that are not detected by this test or due to lower respiratory that is not detected by an NPS specimen. Conversely, positive results do not rule out infection or co-infection with organisms not detected by the LIAISON PLEX Respiratory Flex (RSP Flex) Assay. The agent(s) detected may not be the definite cause of disease.
The use of additional laboratory testing (e.g., bacterial and viral culture, immunofluorescence, and radiography), may be necessary when evaluating a patient with possible respiratory tract infection.
The LIAISON PLEX® Respiratory Flex Assay is a multiplexed nucleic acid test system composed of the LIAISON PLEX Instrument, the LIAISON PLEX® System Software (preinstalled on the LIAISON PLEX Instrument), the LIAISON PLEX® Respiratory Flex Assay cartridge, and the LIAISON PLEX® Respiratory Flex Assay File. The LIAISON PLEX® Respiratory Flex Assay cartridge contains the reagents to perform nucleic acid extraction and purification, reverse transcription, PCR, and array hybridization. Specifically, the LIAISON PLEX® Respiratory Flex Assay detects bacteria and viruses from nasopharyngeal swab (NPS) specimens collected from individuals with signs and symptoms of respiratory infection.
The LIAISON PLEX System consists of a touchscreen user interface that includes the software for running and analyzing assay results, one to six processing/imaging LIAISON PLEX modules, and a handheld barcode reader. Each LIAISON PLEX module processes one sample at a time under the control of the LIAISON PLEX System software.
LIAISON PLEX® automates the sample processing through analysis within a single cartridge. Processing steps include 1.) Sample Preparation: Nucleic acid extraction from organisms by chemical and mechanical means and isolation of nucleic acid on magnetic beads 2.) Target Amplification: Multiplex PCR and RT-PCR based amplification of extracted nucleic acid to generate target specific amplicons 3.) Hybridization: Amplicons hybridize with their target specific DNA probe arranged in a microarray format and that are attached to mediator and gold nanoparticles 4.) Analysis: Gold nanoparticles specifically bound to target amplicons are silver enhanced and the light scatter from microarray spot is measured and analyzed to confirm presence (Detected) or absence (not Detected) of a target.
The LIAISON PLEX Respiratory Flex Assay has the option of creating and processing results for custom panels using Flex® Software. Flex Software allows users to randomly select and group targets in tiers for result processing. Up to 7 targets may be selected for the initial test tier. After the first tier, each additional tier requires a specific number of credits. Flex™ credits allow the end-user to create custom panels and pay for a smaller subset of results tailored to the individual patient's clinical presentation. Alternatively, a laboratory may choose the fixed price option where all target results are processed at the same time.
The LIAISON PLEX Respiratory Flex Assay is a multiplexed qualitative test for the simultaneous in vitro detection and identification of multiple bacterial and viral nucleic acids in nasopharyngeal swabs (NPS) from individuals with clinical signs and symptoms of respiratory tract infection, including SARS-CoV-2.
Here's an analysis of the acceptance criteria and study proving its performance:
1. Table of Acceptance Criteria and Reported Device Performance
The provided document doesn't explicitly state "acceptance criteria" with numerical thresholds for performance metrics. However, regulatory bodies like the FDA typically expect high sensitivity (Positive Percent Agreement - PPA) and specificity (Negative Percent Agreement - NPA) for diagnostic assays. Based on the clinical performance summary, we can infer the achieved performance.
Infered Acceptance Criteria and Reported Device Performance (Summary for Key Analytes):
| Analyte (Overall Performance) | Infered Acceptance Criteria (Typical) | Reported Device Performance (PPA) | Reported Device Performance (NPA) |
|---|---|---|---|
| Adenovirus | High PPA, High NPA | 100% (96.1-100% CI) | 95.7% (94.7-96.6% CI) |
| Bordetella parapertussis | High PPA, High NPA | 80.0% (37.6-96.4% CI) | 99.8% (99.5-99.9% CI) |
| Human Coronavirus | High PPA, High NPA | 90.0% (83.6-94.1% CI) | 99.5% (99.1-99.8% CI) |
| Enterovirus/Rhinovirus | High PPA, High NPA | 93.7% (90.5-95.8% CI) | 97.8% (96.9-98.4% CI) |
| Human Metapneumovirus (hMPV) | High PPA, High NPA | 95.4% (90.4-97.9% CI) | 99.6% (99.2-99.8% CI) |
| Influenza A | High PPA, High NPA | 100% (97.1-100% CI) | 99.1% (98.5-99.4% CI) |
| Influenza A Subtype H1 | High PPA, High NPA | 100% (90.6-100% CI) | 99.9% (99.7-100% CI) |
| Influenza A Subtype H3 | High PPA, High NPA | 97.2% (92.1-99.0% CI) | 99.8% (99.4-99.9% CI) |
| Influenza B | High PPA, High NPA | 100% (67.6-100% CI) | 100% (99.8-100% CI) |
| Parainfluenza 1 | High PPA, High NPA | 91.7% (64.6-98.5% CI) | 100% (99.8-100% CI) |
| Parainfluenza 2 | High PPA, High NPA | 92.3% (66.7-98.6% CI) | 100% (99.8-100% CI) |
| Parainfluenza 3 | High PPA, High NPA | 93.2% (81.8-97.7% CI) | 99.9% (99.7-100% CI) |
| Parainfluenza 4 | High PPA, High NPA | 88.9% (56.5-98.0% CI) | 99.9% (99.7-100% CI) |
| Respiratory Syncytial Virus (RSV) | High PPA, High NPA | 95.9% (90.8-98.3% CI) | 100% (99.8-100% CI) |
| SARS-CoV-2 | High PPA, High NPA | 96.5% (93.4-98.1% CI) | 99.5% (99.0-99.7% CI) |
Note: For analytes with 0/0 positive cases (Bordetella holmesii, Bordetella pertussis, Chlamydia pneumoniae, Mycoplasma pneumoniae in the prospective study), performance is "Not Evaluable" (NE) but the NPA is 100%. These were supplemented with contrived specimens.
2. Sample Size and Data Provenance
Prospective Clinical Study:
- Sample Size (Test Set): 1843 unique clinical specimens initially enrolled, with 1832 specimens yielding valid results after retests.
- Data Provenance: Prospectively collected between October 2022 to April 2023 from six geographically diverse clinical sites within the United States. Specimens were remnant and de-identified, collected from pediatric and adult patients.
Archived Specimen Testing (Retrospective):
- Sample Size (Test Set): 256 pre-selected, left-over, frozen, de-identified specimens, all yielding valid results after retests.
- Data Provenance: Retrospectively collected from November 2013 through June 2023 from four sites/vendors in the United States.
Contrived Specimen Testing:
- Sample Size (Test Set): 300 contrived specimens, all yielding valid results after retests.
- Data Provenance: Artificially created samples to cover low prevalence targets. Tested at two US sites (details about origin of base matrix not specified beyond "simulated NPS matrix").
3. Number of Experts and Qualifications for Ground Truth
The document does not specify the number of experts used and their exact qualifications (e.g., "radiologist with 10 years of experience") for establishing ground truth. Instead, it indicates that the ground truth was established by comparator methods, which are themselves FDA-cleared molecular panels or analytically validated assays. These methods inherently rely on expertise for their development and validation but do not require additional human expert adjudication for each case in this study.
4. Adjudication Method for the Test Set
Specimens that obtained discordant results between the LIAISON PLEX Respiratory Flex Assay and the comparator method underwent additional testing for investigation.
- For targets typically compared against FDA-cleared molecular respiratory panels, discordant samples were re-tested with an FDA-cleared molecular respiratory panel or PCR/BDS.
- For Bordetella holmesii, Bordetella parapertussis, Bordetella pertussis, comparator performance was based on "well-validated Fragment Analysis (FA) assays followed by PCR/Bi-Directional Sequencing (PCR/BDS) assays."
This implies an adjudication method involving a third, more definitive or confirmatory test (often a "tie-breaker" or "gold standard" method like PCR/BDS) for discordant results. This is a common practice in diagnostic device studies. The document does not specify a "2+1" or "3+1" structure as those typically refer to multiple human readers or interpretations.
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
No, a Multi-Reader Multi-Case (MRMC) comparative effectiveness study was not done. This study evaluates the performance of a diagnostic assay itself for detecting nucleic acids, not the impact of AI assistance on human readers' interpretation of images or other data. Therefore, there is no effect size reported for how much human readers improve with AI vs. without AI assistance.
6. Standalone (Algorithm Only) Performance
Yes, a standalone performance was done. The entire clinical performance study (prospective, archived, and contrived) evaluates the LIAISON PLEX Respiratory Flex Assay as an algorithm-only or device-only diagnostic tool without human-in-the-loop interpretation being part of the primary evaluation. The results presented for PPA and NPA are based solely on the device's output compared to the ground truth.
7. Type of Ground Truth Used
The ground truth used was primarily established through comparator laboratory methods:
- FDA-cleared molecular respiratory panels for most viral and bacterial targets.
- FDA-cleared molecular SARS-CoV-2 assay for SARS-CoV-2.
- Analytically Validated Fragment Analysis (FA) assays followed by PCR/Bi-Directional Sequencing (PCR/BDS) assays for Bordetella species.
- For discordant results, additional testing with FDA-cleared molecular respiratory panels or PCR/BDS was performed for investigation.
This combination of highly sensitive and specific molecular diagnostic assays, with confirmatory testing for discordance, serves as the ground truth.
8. Sample Size for the Training Set
The document does not explicitly state the sample size for a separate "training set" for the LIAISON PLEX Respiratory Flex Assay. This is characteristic of molecular diagnostic assays which are typically developed and optimized through analytical studies (e.g., limit of detection, inclusivity, exclusivity) and then validated in clinical performance studies without a distinct "AI training set" in the common machine learning sense. The performance characteristics (analytical and clinical) presented are for the final, locked-down assay.
9. How the Ground Truth for the Training Set was Established
Since a distinct "training set" in the AI/machine learning context is not specified, the method for establishing its ground truth is not detailed. However, the development of such assays involves extensive analytical testing using characterized isolates, strains, and clinical samples to define reactivity, specificity, and sensitivity. The ground truth for this analytical development would typically be based on:
- Known concentrations of purified nucleic acids or organisms.
- Well-characterized reference materials and clinical specimens with confirmed presence/absence of targets via established methods (e.g., culture, sequencing, reference PCR).
§ 866.3981 Device to detect and identify nucleic acid targets in respiratory specimens from microbial agents that cause the SARS-CoV-2 respiratory infection and other microbial agents when in a multi-target test.
(a)
Identification. A device to detect and identify nucleic acid targets in respiratory specimens from microbial agents that cause the SARS-CoV-2 respiratory infection and other microbial agents when in a multi-target test is an in vitro diagnostic device intended for the detection and identification of SARS-CoV-2 and other microbial agents when in a multi-target test in human clinical respiratory specimens from patients suspected of respiratory infection who are at risk for exposure or who may have been exposed to these agents. The device is intended to aid in the diagnosis of respiratory infection in conjunction with other clinical, epidemiologic, and laboratory data or other risk factors.(b)
Classification. Class II (special controls). The special controls for this device are:(1) The intended use in the labeling required under § 809.10 of this chapter must include a description of the following: Analytes and targets the device detects and identifies, the specimen types tested, the results provided to the user, the clinical indications for which the test is to be used, the specific intended population(s), the intended use locations including testing location(s) where the device is to be used (if applicable), and other conditions of use as appropriate.
(2) Any sample collection device used must be FDA-cleared, -approved, or -classified as 510(k) exempt (standalone or as part of a test system) for the collection of specimen types claimed by this device; alternatively, the sample collection device must be cleared in a premarket submission as a part of this device.
(3) The labeling required under § 809.10(b) of this chapter must include:
(i) A detailed device description, including reagents, instruments, ancillary materials, all control elements, and a detailed explanation of the methodology, including all pre-analytical methods for processing of specimens;
(ii) Detailed descriptions of the performance characteristics of the device for each specimen type claimed in the intended use based on analytical studies including the following, as applicable: Limit of Detection, inclusivity, cross-reactivity, interfering substances, competitive inhibition, carryover/cross contamination, specimen stability, precision, reproducibility, and clinical studies;
(iii) Detailed descriptions of the test procedure(s), the interpretation of test results for clinical specimens, and acceptance criteria for any quality control testing;
(iv) A warning statement that viral culture should not be attempted in cases of positive results for SARS-CoV-2 and/or any similar microbial agents unless a facility with an appropriate level of laboratory biosafety (
e.g., BSL 3 and BSL 3+, etc.) is available to receive and culture specimens; and(v) A prominent statement that device performance has not been established for specimens collected from individuals not identified in the intended use population (
e.g., when applicable, that device performance has not been established in individuals without signs or symptoms of respiratory infection).(vi) Limiting statements that indicate that:
(A) A negative test result does not preclude the possibility of infection;
(B) The test results should be interpreted in conjunction with other clinical and laboratory data available to the clinician;
(C) There is a risk of incorrect results due to the presence of nucleic acid sequence variants in the targeted pathogens;
(D) That positive and negative predictive values are highly dependent on prevalence;
(E) Accurate results are dependent on adequate specimen collection, transport, storage, and processing. Failure to observe proper procedures in any one of these steps can lead to incorrect results; and
(F) When applicable (
e.g., recommended by the Centers for Disease Control and Prevention, by current well-accepted clinical guidelines, or by published peer-reviewed literature), that the clinical performance may be affected by testing a specific clinical subpopulation or for a specific claimed specimen type.(4) Design verification and validation must include:
(i) Detailed documentation, including performance results, from a clinical study that includes prospective (sequential) samples for each claimed specimen type and, as appropriate, additional characterized clinical samples. The clinical study must be performed on a study population consistent with the intended use population and compare the device performance to results obtained using a comparator that FDA has determined is appropriate. Detailed documentation must include the clinical study protocol (including a predefined statistical analysis plan), study report, testing results, and results of all statistical analyses.
(ii) Risk analysis and documentation demonstrating how risk control measures are implemented to address device system hazards, such as Failure Modes Effects Analysis and/or Hazard Analysis. This documentation must include a detailed description of a protocol (including all procedures and methods) for the continuous monitoring, identification, and handling of genetic mutations and/or novel respiratory pathogen isolates or strains (
e.g., regular review of published literature and periodic in silico analysis of target sequences to detect possible mismatches). All results of this protocol, including any findings, must be documented and must include any additional data analysis that is requested by FDA in response to any performance concerns identified under this section or identified by FDA during routine evaluation. Additionally, if requested by FDA, these evaluations must be submitted to FDA for FDA review within 48 hours of the request. Results that are reasonably interpreted to support the conclusion that novel respiratory pathogen strains or isolates impact the stated expected performance of the device must be sent to FDA immediately.(iii) A detailed description of the identity, phylogenetic relationship, and other recognized characterization of the respiratory pathogen(s) that the device is designed to detect. In addition, detailed documentation describing how to interpret the device results and other measures that might be needed for a laboratory diagnosis of respiratory infection.
(iv) A detailed device description, including device components, ancillary reagents required but not provided, and a detailed explanation of the methodology, including molecular target(s) for each analyte, design of target detection reagents, rationale for target selection, limiting factors of the device (
e.g., saturation level of hybridization and maximum amplification and detection cycle number, etc.), internal and external controls, and computational path from collected raw data to reported result (e.g., how collected raw signals are converted into a reported signal and result), as applicable.(v) A detailed description of device software, including software applications and hardware-based devices that incorporate software. The detailed description must include documentation of verification, validation, and hazard analysis and risk assessment activities, including an assessment of the impact of threats and vulnerabilities on device functionality and end users/patients as part of cybersecurity review.
(vi) For devices intended for the detection and identification of microbial agents for which an FDA recommended reference panel is available, design verification and validation must include the performance results of an analytical study testing the FDA recommended reference panel of characterized samples. Detailed documentation must be kept of that study and its results, including the study protocol, study report for the proposed intended use, testing results, and results of all statistical analyses.
(vii) For devices with an intended use that includes detection of Influenza A and Influenza B viruses and/or detection and differentiation between the Influenza A virus subtypes in human clinical specimens, the design verification and validation must include a detailed description of the identity, phylogenetic relationship, or other recognized characterization of the Influenza A and B viruses that the device is designed to detect, a description of how the device results might be used in a diagnostic algorithm and other measures that might be needed for a laboratory identification of Influenza A or B virus and of specific Influenza A virus subtypes, and a description of the clinical and epidemiological parameters that are relevant to a patient case diagnosis of Influenza A or B and of specific Influenza A virus subtypes. An evaluation of the device compared to a currently appropriate and FDA accepted comparator method. Detailed documentation must be kept of that study and its results, including the study protocol, study report for the proposed intended use, testing results, and results of all statistical analyses.
(5) When applicable, performance results of the analytical study testing the FDA recommended reference panel described in paragraph (b)(4)(vi) of this section must be included in the device's labeling under § 809.10(b) of this chapter.
(6) For devices with an intended use that includes detection of Influenza A and Influenza B viruses and/or detection and differentiation between the Influenza A virus subtypes in human clinical specimens in addition to detection of SARS-CoV-2 and similar microbial agents, the required labeling under § 809.10(b) of this chapter must include the following:
(i) Where applicable, a limiting statement that performance characteristics for Influenza A were established when Influenza A/H3 and A/H1-2009 (or other pertinent Influenza A subtypes) were the predominant Influenza A viruses in circulation.
(ii) Where applicable, a warning statement that reads if infection with a novel Influenza A virus is suspected based on current clinical and epidemiological screening criteria recommended by public health authorities, specimens should be collected with appropriate infection control precautions for novel virulent influenza viruses and sent to State or local health departments for testing. Viral culture should not be attempted in these cases unless a BSL 3+ facility is available to receive and culture specimens.
(iii) Where the device results interpretation involves combining the outputs of several targets to get the final results, such as a device that both detects Influenza A and differentiates all known Influenza A subtypes that are currently circulating, the device's labeling must include a clear interpretation instruction for all valid and invalid output combinations, and recommendations for any required followup actions or retesting in the case of an unusual or unexpected device result.
(iv) A limiting statement that if a specimen yields a positive result for Influenza A, but produces negative test results for all specific influenza A subtypes intended to be differentiated (
i.e., H1-2009 and H3), this result requires notification of appropriate local, State, or Federal public health authorities to determine necessary measures for verification and to further determine whether the specimen represents a novel strain of Influenza A.(7) If one of the actions listed at section 564(b)(1)(A) through (D) of the Federal Food, Drug, and Cosmetic Act occurs with respect to an influenza viral strain, or if the Secretary of Health and Human Services determines, under section 319(a) of the Public Health Service Act, that a disease or disorder presents a public health emergency, or that a public health emergency otherwise exists, with respect to an influenza viral strain:
(i) Within 30 days from the date that FDA notifies manufacturers that characterized viral samples are available for test evaluation, the manufacturer must have testing performed on the device with those influenza viral samples in accordance with a standardized protocol considered and determined by FDA to be acceptable and appropriate.
(ii) Within 60 days from the date that FDA notifies manufacturers that characterized influenza viral samples are available for test evaluation and continuing until 3 years from that date, the results of the influenza emergency analytical reactivity testing, including the detailed information for the virus tested as described in the certificate of authentication, must be included as part of the device's labeling in a tabular format, either by:
(A) Placing the results directly in the device's labeling required under § 809.10(b) of this chapter that accompanies the device in a separate section of the labeling where analytical reactivity testing data can be found, but separate from the annual analytical reactivity testing results; or
(B) In a section of the device's label or in other labeling that accompanies the device, prominently providing a hyperlink to the manufacturer's public website where the analytical reactivity testing data can be found. The manufacturer's website, as well as the primary part of the manufacturer's website that discusses the device, must provide a prominently placed hyperlink to the website containing this information and must allow unrestricted viewing access.