(431 days)
The DiaSorin Molecular Simplexa™ COVID-19 Direct is real-time RT-PCR assay intended for use on the LIAISON® MDX instrument for the in vitro qualitative detection of nucleic acid from severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in nasopharyngeal swabs (NPS) and nasal swabs (NS) from symptomatic individuals suspected of COVID 19 by their healthcare provider. The Simplexa™ COVID-19 Direct assay is an aid in the diagnosis of SARS-CoV-2 infection.
Positive results are indicative of the presence of SARS-CoV-2 RNA. Clinical correlation with patient history and other diagnostic information is necessary to determine patient infection status. Positive results do not rule out co-infection with other pathogens. Negative results do not preclude SARS-CoV-2 infection and should not be used as tor patient management decisions.
Results are meant to be used in conjunction with other clinical, epidemiologic, and laboratory data, in accordance with the guidelines provided by the relevant public health authorities.
The Simplexa COVID-19 Direct is a real-time RT-PCR (rRT-PCR) system that enables the direct amplification and detection of SARS-CoV-2 (COVID-19) RNA from nasopharyngeal swab or nasal swab that has not undergone nucleic acid extraction. The system consists of the Simplexa COVID-19 Direct reaction mix, the LIAISON MDX (with LIAISON MDX Studio Software), the Direct Amplification Disc and associated accessories. The assay uses forward and reverse primers and associated fluorescent probe(s) included in the reaction mix to amplify SARS-CoV-2 cDNA reverse transcribed from RNA. The primers and probe sets are designed to detect SARS-CoV-2 ORF 1ab and S gene from the viral RNA in nasopharyngeal swab or nasal swab. An RNA internal control, with associated primers and a fluorescent probe, is included in the reaction mix to detect RT-PCR failure and/or inhibition.
Here's a breakdown of the acceptance criteria and study proving the device meets them, based on the provided text:
Device: Simplexa™ COVID-19 Direct
1. Table of Acceptance Criteria and Reported Device Performance
For the Simplexa™ COVID-19 Direct assay, the primary acceptance criteria revolve around its accuracy in detecting SARS-CoV-2 (COVID-19) RNA in patient samples, as well as its reproducibility, analytical sensitivity (Limit of Detection), analytical reactivity (ability to detect various strains), and specificity (cross-reactivity and interference).
| Acceptance Criteria Category | Specific Acceptance Criteria (Implicit from Study Design) | Reported Device Performance (Simplexa™ COVID-19 Direct) |
|---|---|---|
| Clinical Agreement (Total Specimens) | High Percent Positive Agreement (PPA) and Negative Percent Agreement (NPA) compared to an EUA NAAT Composite Reference Method. | PPA: 98.2% (108/110) (95% CI: 93.6% to 99.5%) |
| NPA: 99.6% (897/901) (95% CI: 98.9% to 99.8%) | ||
| Clinical Agreement (NPS) | High PPA and NPA for Nasopharyngeal Swabs. | PPA: 98.4% (60/61) (95% CI: 91.3% to 99.7%) |
| NPA: 99.6% (237/238) (95% CI: 97.7% to 99.9%) | ||
| Clinical Agreement (NS) | High PPA and NPA for Nasal Swabs. | PPA: 98.0% (48/49) (95% CI: 89.3% to 99.6%) |
| NPA: 99.5% (660/663) (95% CI: 98.7% to 99.8%) | ||
| Reproducibility (Low Positive) | High agreement with expected results across sites and operators for low positive samples. | S gene: 94.4% (85/90) agreement; Avg. Ct (All Sites) 31.6 ± 0.95 (3.0%) |
| ORF1ab: 95.6% (86/90) agreement; Avg. Ct (All Sites) 32.2 ± 0.97 (3.0%) | ||
| Total (algorithm based): 98.9% (89/90) agreement | ||
| Reproducibility (Moderate Positive) | High agreement with expected results across sites and operators for moderate positive samples. | S gene: 95.6% (86/90) agreement; Avg. Ct (All Sites) 30.5 ± 0.80 (2.6%) |
| ORF1ab: 100.0% (90/90) agreement; Avg. Ct (All Sites) 31.3 ± 0.87 (2.8%) | ||
| Total (algorithm based): 100.0% (90/90) agreement | ||
| Reproducibility (Negative) | 100% agreement with expected results for negative samples. | S gene: 100.0% (90/90) agreement |
| ORF1ab: 100.0% (90/90) agreement | ||
| Total (algorithm based): 100.0% (90/90) agreement | ||
| Reproducibility (Positive Control) | 100% agreement with expected results for positive control. | S gene: 100.0% (90/90) agreement |
| ORF1ab: 100.0% (90/90) agreement | ||
| Total (algorithm based): 100.0% (90/90) agreement | ||
| Analytical Sensitivity / Limit of Detection (NPS) | LoD confirmed as the lowest concentration with ≥95% positivity. | 500 copies/mL (100% detection for total algorithm based) |
| Analytical Sensitivity / Limit of Detection (NS) | LoD confirmed as the lowest concentration with ≥95% positivity. | 242 copies/mL (100% detection for total algorithm based) |
| Analytical Sensitivity / LoD (WHO International Standard) | LoD confirmed as the lowest concentration with ≥95% positivity (IU/mL). | 500 IU/mL (97.5% detection) |
| Analytical Reactivity / Inclusivity | Ability to detect various SARS-CoV-2 strains and variants. | All 5 wet-tested strains (Hong Kong, England, South Africa, Japan, hCoV19/USA) detected at 100% (3/3 replicates) at 1000 copies/mL. In silico analysis showed 98.6% - 99.99% sequence homology with broad variant coverage (Omicron BA.4/BA.5, BA.2.12.1, BA.2.75). |
| Cross-Reactivity | No false positives when challenged with common respiratory pathogens or human nucleic acid. | 0.0% detection across 47 tested organisms (viruses, bacteria, fungi, human genomic DNA, pooled human nasal fluid) for S gene and ORF1ab targets. IC detected at 100%. MERS-CoV showed 0.0% detection. |
| Potential Interfering Substances | No false negatives for COVID-19 detection in the presence of common nasal/respiratory substances. | 100% detection for most substances (antibiotics, antivirals, nasal corticosteroids, etc.). Saliva showed 83.3% detection at 10% (v/v) but 100% at 5% (v/v), indicating interference at higher concentrations. Zanamivir 83.3% IC detection for 5/6 replicates. |
| Interference by Other Microorganisms | No inhibition of SARS-CoV-2 detection by other microorganisms. | 100% detection of SARS-CoV-2 at 2x LoD for 46/47 co-present organisms. Lactobacillus plantarum 17-5 showed interference above 5x10^5 CFU/mL. |
| Carry-Over Contamination | No evidence of carry-over contamination. | No carry-over contamination observed during testing with high positive and negative samples. |
2. Sample Size Used for the Test Set and Data Provenance
-
Clinical Agreement Test Set:
- Total Samples: 1,150 prospective (fresh and/or frozen) samples collected.
- Samples analyzed: 1,011 samples (114 excluded due to insufficient evidence for media types, 24 invalid results, 1 indeterminate CRM result).
- Breakdown: 299 Nasopharyngeal Swabs (NPS) and 712 Nasal Swabs (NS).
- Provenance: Collected from four (4) geographically diverse collection sites, one of which was outside the United States (OUS). Samples were prospective (fresh and/or frozen).
- Timeframe: October 2020 to April 2021.
-
Reproducibility Test Set:
- Total replicates: 90 replicates per panel member (4 panel members), totaling 360 individual tests.
- Panel members: 2 contrived low positive (LP), 2 contrived moderate positive (MP), 1 positive control, 1 negative (UTM).
- Provenance: Tested at two (2) external clinical sites and one (1) internal site.
- Study Design: Each panel member tested in triplicate per run, for 2 runs per day, for 5 non-consecutive testing days. Each site had two operators.
-
Analytical Sensitivity (LoD) Test Set:
- NPS: 40 replicates for confirmation.
- NS: 20 replicates for confirmation.
- WHO International Standard: 40 replicates for confirmation.
-
Analytical Reactivity (Wet testing) Test Set:
- 3 replicates per strain for 5 SARS-CoV-2 strains.
-
Cross-Reactivity Test Set:
- 3 replicates per organism for 47 different viruses, bacteria, and fungi (some 6 replicates for Leptospira interrogans).
-
Potential Interfering Substances Test Set:
- 3 replicates per substance (some 6 replicates for saliva and Zanamivir).
-
Interference by Other Microorganisms Test Set:
- 3 replicates per organism (some 6 replicates for Lactobacillus plantarum 17-5).
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications of Those Experts
The ground truth for the clinical agreement test set was established using a "Composite Reference Method (CRM)" based on three (3) COVID-19 EUA approved NAAT assays. The rule for CRM agreement was: "Two out of three positive results determined 'Detected' CRM and two out of three negative results determined 'Not Detected' CRM."
The document does not specify the number or qualifications of experts (e.g., medical technologists, clinical lab scientists, or physicians) who performed these NAAT assays or interpreted their results for the CRM. It's implied that these were standard laboratory personnel qualified to run EUA-approved molecular diagnostic tests.
For analytical studies (LoD, reproducibility, reactivity, cross-reactivity, interference), the ground truth was based on the known composition and concentration of the samples (e.g., spiked RNA, cultured organisms, negative matrix). No external experts beyond the study design team would have been needed for this type of ground truth establishment.
4. Adjudication Method for the Test Set
For the clinical agreement test set, the adjudication method for the ground truth (CRM) was clearly defined: "Two out of three positive results determined 'Detected' CRM and two out of three negative results determined 'Not Detected' CRM." This is a form of consensus-based adjudication, specifically a majority rule.
For other analytical studies, adjudication was not described as it involved pre-defined positive/negative samples rather than interpretive human judgment.
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study Was Done
No, a Multi-Reader Multi-Case (MRMC) comparative effectiveness study was not done. This study focuses on the in vitro diagnostic performance of a molecular assay (RT-PCR) in a laboratory setting, not on the interpretative performance of human readers (e.g., radiologists) with or without AI assistance. Therefore, there is no discussion of human readers or an effect size of AI assistance.
6. If a Standalone (i.e. algorithm only without human-in-the loop performance) Was Done
Yes, the primary clinical performance and analytical studies are standalone (algorithm only) performance. The Simplexa™ COVID-19 Direct is an RT-PCR assay. Its "performance" refers to its ability to detect SARS-CoV-2 RNA based on its set algorithms for signal detection (Ct values for S gene and ORF1ab targets) and interpretation. The results (detected/not detected) are determined directly by the instrument and its software, not by a human interpreting images or complex patterns. The human involvement is in sample preparation and loading, and reviewing the qualitative output from the instrument.
7. The Type of Ground Truth Used
- Clinical Agreement Test Set: Composite Reference Method (CRM) using results from three (3) COVID-19 EUA approved NAAT assays, with a "two out of three" majority rule for determining "Detected" or "Not Detected." This is a form of expert consensus based on other validated diagnostic tests.
- Analytical Studies (Reproducibility, LoD, Reactivity, Cross-Reactivity, Interference): Known analytical truth established by spiking known concentrations of inactivated viral particles or other organisms into negative matrices. This is a laboratory-controlled ground truth.
8. The Sample Size for the Training Set
The provided document describes a premarket notification (510(k)) for an in vitro diagnostic device. For such devices, particularly RT-PCR assays, the "training set" typically refers to internal development and optimization data, rather than a distinct, formally defined "training set" for machine learning algorithms that would be tested on a separate "test set."
The document does not specify a numerical sample size for a training set. The assay's design (primers, probes, conditions) would have been developed and optimized internally by DiaSorin Molecular using various samples and experiments, but these are not enumerated as a specific "training set" in this regulatory submission. The "in silico inclusivity analysis" section points to the use of GISAID databases (millions of sequences) which could be considered a form of "training data" for validating the generalizability of the primer/probe design, but not as a conventional, labeled "training set" in a machine learning context.
9. How the Ground Truth for the Training Set Was Established
Since a formal "training set" with established ground truth is not explicitly detailed in the way a machine learning model's training data would be, we can infer the following:
- Assay Development & Optimization: The ground truth for the development phase would have been based on known positive and negative samples, viral loads, and various SARS-CoV-2 strains or synthetic genetic material. This involves standard molecular biology techniques where the presence or absence of the target nucleic acid, and its concentration, are experimentally determined and controlled.
- In Silico Inclusivity: For the evaluation of primer/probe design against genetic variants, the "ground truth" is the published, annotated SARS-CoV-2 genome sequences available in the GISAID database. This involves bioinformatic analysis to determine sequence homology and potential binding efficacy.
§ 866.3981 Device to detect and identify nucleic acid targets in respiratory specimens from microbial agents that cause the SARS-CoV-2 respiratory infection and other microbial agents when in a multi-target test.
(a)
Identification. A device to detect and identify nucleic acid targets in respiratory specimens from microbial agents that cause the SARS-CoV-2 respiratory infection and other microbial agents when in a multi-target test is an in vitro diagnostic device intended for the detection and identification of SARS-CoV-2 and other microbial agents when in a multi-target test in human clinical respiratory specimens from patients suspected of respiratory infection who are at risk for exposure or who may have been exposed to these agents. The device is intended to aid in the diagnosis of respiratory infection in conjunction with other clinical, epidemiologic, and laboratory data or other risk factors.(b)
Classification. Class II (special controls). The special controls for this device are:(1) The intended use in the labeling required under § 809.10 of this chapter must include a description of the following: Analytes and targets the device detects and identifies, the specimen types tested, the results provided to the user, the clinical indications for which the test is to be used, the specific intended population(s), the intended use locations including testing location(s) where the device is to be used (if applicable), and other conditions of use as appropriate.
(2) Any sample collection device used must be FDA-cleared, -approved, or -classified as 510(k) exempt (standalone or as part of a test system) for the collection of specimen types claimed by this device; alternatively, the sample collection device must be cleared in a premarket submission as a part of this device.
(3) The labeling required under § 809.10(b) of this chapter must include:
(i) A detailed device description, including reagents, instruments, ancillary materials, all control elements, and a detailed explanation of the methodology, including all pre-analytical methods for processing of specimens;
(ii) Detailed descriptions of the performance characteristics of the device for each specimen type claimed in the intended use based on analytical studies including the following, as applicable: Limit of Detection, inclusivity, cross-reactivity, interfering substances, competitive inhibition, carryover/cross contamination, specimen stability, precision, reproducibility, and clinical studies;
(iii) Detailed descriptions of the test procedure(s), the interpretation of test results for clinical specimens, and acceptance criteria for any quality control testing;
(iv) A warning statement that viral culture should not be attempted in cases of positive results for SARS-CoV-2 and/or any similar microbial agents unless a facility with an appropriate level of laboratory biosafety (
e.g., BSL 3 and BSL 3+, etc.) is available to receive and culture specimens; and(v) A prominent statement that device performance has not been established for specimens collected from individuals not identified in the intended use population (
e.g., when applicable, that device performance has not been established in individuals without signs or symptoms of respiratory infection).(vi) Limiting statements that indicate that:
(A) A negative test result does not preclude the possibility of infection;
(B) The test results should be interpreted in conjunction with other clinical and laboratory data available to the clinician;
(C) There is a risk of incorrect results due to the presence of nucleic acid sequence variants in the targeted pathogens;
(D) That positive and negative predictive values are highly dependent on prevalence;
(E) Accurate results are dependent on adequate specimen collection, transport, storage, and processing. Failure to observe proper procedures in any one of these steps can lead to incorrect results; and
(F) When applicable (
e.g., recommended by the Centers for Disease Control and Prevention, by current well-accepted clinical guidelines, or by published peer-reviewed literature), that the clinical performance may be affected by testing a specific clinical subpopulation or for a specific claimed specimen type.(4) Design verification and validation must include:
(i) Detailed documentation, including performance results, from a clinical study that includes prospective (sequential) samples for each claimed specimen type and, as appropriate, additional characterized clinical samples. The clinical study must be performed on a study population consistent with the intended use population and compare the device performance to results obtained using a comparator that FDA has determined is appropriate. Detailed documentation must include the clinical study protocol (including a predefined statistical analysis plan), study report, testing results, and results of all statistical analyses.
(ii) Risk analysis and documentation demonstrating how risk control measures are implemented to address device system hazards, such as Failure Modes Effects Analysis and/or Hazard Analysis. This documentation must include a detailed description of a protocol (including all procedures and methods) for the continuous monitoring, identification, and handling of genetic mutations and/or novel respiratory pathogen isolates or strains (
e.g., regular review of published literature and periodic in silico analysis of target sequences to detect possible mismatches). All results of this protocol, including any findings, must be documented and must include any additional data analysis that is requested by FDA in response to any performance concerns identified under this section or identified by FDA during routine evaluation. Additionally, if requested by FDA, these evaluations must be submitted to FDA for FDA review within 48 hours of the request. Results that are reasonably interpreted to support the conclusion that novel respiratory pathogen strains or isolates impact the stated expected performance of the device must be sent to FDA immediately.(iii) A detailed description of the identity, phylogenetic relationship, and other recognized characterization of the respiratory pathogen(s) that the device is designed to detect. In addition, detailed documentation describing how to interpret the device results and other measures that might be needed for a laboratory diagnosis of respiratory infection.
(iv) A detailed device description, including device components, ancillary reagents required but not provided, and a detailed explanation of the methodology, including molecular target(s) for each analyte, design of target detection reagents, rationale for target selection, limiting factors of the device (
e.g., saturation level of hybridization and maximum amplification and detection cycle number, etc.), internal and external controls, and computational path from collected raw data to reported result (e.g., how collected raw signals are converted into a reported signal and result), as applicable.(v) A detailed description of device software, including software applications and hardware-based devices that incorporate software. The detailed description must include documentation of verification, validation, and hazard analysis and risk assessment activities, including an assessment of the impact of threats and vulnerabilities on device functionality and end users/patients as part of cybersecurity review.
(vi) For devices intended for the detection and identification of microbial agents for which an FDA recommended reference panel is available, design verification and validation must include the performance results of an analytical study testing the FDA recommended reference panel of characterized samples. Detailed documentation must be kept of that study and its results, including the study protocol, study report for the proposed intended use, testing results, and results of all statistical analyses.
(vii) For devices with an intended use that includes detection of Influenza A and Influenza B viruses and/or detection and differentiation between the Influenza A virus subtypes in human clinical specimens, the design verification and validation must include a detailed description of the identity, phylogenetic relationship, or other recognized characterization of the Influenza A and B viruses that the device is designed to detect, a description of how the device results might be used in a diagnostic algorithm and other measures that might be needed for a laboratory identification of Influenza A or B virus and of specific Influenza A virus subtypes, and a description of the clinical and epidemiological parameters that are relevant to a patient case diagnosis of Influenza A or B and of specific Influenza A virus subtypes. An evaluation of the device compared to a currently appropriate and FDA accepted comparator method. Detailed documentation must be kept of that study and its results, including the study protocol, study report for the proposed intended use, testing results, and results of all statistical analyses.
(5) When applicable, performance results of the analytical study testing the FDA recommended reference panel described in paragraph (b)(4)(vi) of this section must be included in the device's labeling under § 809.10(b) of this chapter.
(6) For devices with an intended use that includes detection of Influenza A and Influenza B viruses and/or detection and differentiation between the Influenza A virus subtypes in human clinical specimens in addition to detection of SARS-CoV-2 and similar microbial agents, the required labeling under § 809.10(b) of this chapter must include the following:
(i) Where applicable, a limiting statement that performance characteristics for Influenza A were established when Influenza A/H3 and A/H1-2009 (or other pertinent Influenza A subtypes) were the predominant Influenza A viruses in circulation.
(ii) Where applicable, a warning statement that reads if infection with a novel Influenza A virus is suspected based on current clinical and epidemiological screening criteria recommended by public health authorities, specimens should be collected with appropriate infection control precautions for novel virulent influenza viruses and sent to State or local health departments for testing. Viral culture should not be attempted in these cases unless a BSL 3+ facility is available to receive and culture specimens.
(iii) Where the device results interpretation involves combining the outputs of several targets to get the final results, such as a device that both detects Influenza A and differentiates all known Influenza A subtypes that are currently circulating, the device's labeling must include a clear interpretation instruction for all valid and invalid output combinations, and recommendations for any required followup actions or retesting in the case of an unusual or unexpected device result.
(iv) A limiting statement that if a specimen yields a positive result for Influenza A, but produces negative test results for all specific influenza A subtypes intended to be differentiated (
i.e., H1-2009 and H3), this result requires notification of appropriate local, State, or Federal public health authorities to determine necessary measures for verification and to further determine whether the specimen represents a novel strain of Influenza A.(7) If one of the actions listed at section 564(b)(1)(A) through (D) of the Federal Food, Drug, and Cosmetic Act occurs with respect to an influenza viral strain, or if the Secretary of Health and Human Services determines, under section 319(a) of the Public Health Service Act, that a disease or disorder presents a public health emergency, or that a public health emergency otherwise exists, with respect to an influenza viral strain:
(i) Within 30 days from the date that FDA notifies manufacturers that characterized viral samples are available for test evaluation, the manufacturer must have testing performed on the device with those influenza viral samples in accordance with a standardized protocol considered and determined by FDA to be acceptable and appropriate.
(ii) Within 60 days from the date that FDA notifies manufacturers that characterized influenza viral samples are available for test evaluation and continuing until 3 years from that date, the results of the influenza emergency analytical reactivity testing, including the detailed information for the virus tested as described in the certificate of authentication, must be included as part of the device's labeling in a tabular format, either by:
(A) Placing the results directly in the device's labeling required under § 809.10(b) of this chapter that accompanies the device in a separate section of the labeling where analytical reactivity testing data can be found, but separate from the annual analytical reactivity testing results; or
(B) In a section of the device's label or in other labeling that accompanies the device, prominently providing a hyperlink to the manufacturer's public website where the analytical reactivity testing data can be found. The manufacturer's website, as well as the primary part of the manufacturer's website that discusses the device, must provide a prominently placed hyperlink to the website containing this information and must allow unrestricted viewing access.