(85 days)
The cobas® MRSA/SA Test on the cobas® 4800 system, is a qualitative in vitro diagnostic real-time PCR assay, for the direct detection of methicillin-resistant Staphylococus aureus (MRSA) and S.aureus (SA) DNA from nasal swabs to aid in the prevention and control of MRSA and SA infections in healthcare settings. The cobas® MRSA/SA Test is not intended to diagnose, guide or monitor treatment for MRSA or SA infections, or provide results of susceptibility to methicillin. A negative result does not preclude MRSA/SA nasal colonization. Concomitant cultures are necessary to recover organisms for epidemiology typing or for further susceptibility testing.
The Roche Molecular Systems (RMS) cobas® MRSA/SA Test utilizes real-time polymerase chain reaction (PCR) for the detection of Methicillin resistant Staphylococcus aureus (MRSA) and Staphylococcus aureus (SA) DNA from nasal swab specimens collected in MSwab medium to aid in the prevention and control of MRSA and SA infections in healthcare settings.
The cobas® MRSA/SA Test contains two major processes: (1) automated sample preparation to extract nucleic acids from the nasal swab specimens; (2) PCR amplification of target DNA sequences using MRSA and SA specific primers, and real-time detection of cleaved fluorescentlabeled MRSA and SA specific oligonucleotide detection probes. An Internal Control (IC), containing unrelated randomized DNA sequence, is added to all samples prior to automated sample preparation and is amplified and detected simultaneously with each sample to monitor the entire process.
The MSwab Collection, Transport and Preservation System (Copan Flock Technologies S.r.1.) is used for specimen collection, transportation and storage of specimen for the cobas MRSA/SA Test.
The cobas® MRSA/SA Test utilizes six reagent kits:
- cobas® 4800 MRSA/SA Amplification/Detection Kit
- cobas® 4800 MRSA/SA Controls and Cofactor Kit
- cobas® 4800 System Wash Buffer Kit
- cobas® 4800 System Lysis Kit 1
- cobas 4800 System Internal Control Kit 1
- cobas® 4800 System Sample Preparation Kit
The cobas® 4800 System uses the cobas x 480 Instrument for sample preparation, and the cobas z 480 Analyzer for amplification and detection. Both the cobas x 480 Instrument and the cobas z 480 Analyzer are controlled by a computer workstation running the cobas® 4800 System Software.
Here's an analysis of the acceptance criteria and study proving the device meets them, based on the provided text:
Device Name: cobas® MRSA/SA Test
1. Table of Acceptance Criteria and Reported Device Performance
The document does not explicitly present a table of acceptance criteria in the sense of predefined thresholds for performance metrics. Instead, it reports the achieved performance from various studies. However, based on the studies conducted, we can infer the performance metrics that were evaluated and for which the device demonstrated acceptable results.
Here's a table summarizing the reported device performance, categorized by the relevant studies:
| Performance Metric | Acceptance Criteria (Inferred) | Reported Device Performance | Study Section |
|---|---|---|---|
| Analytical Sensitivity (LOD) | ≥ 95% positive rate at lowest concentration | MRSA: 700 CFU/swab (highest among 2 strains), 175-750 CFU/swab (35 isolates) | 4.1, 4.2 |
| SA: 700 CFU/swab (highest among 1 strain), 175 CFU/swab (5 isolates) | 4.1, 4.2 | ||
| Genotype Inclusivity (MRSA) | Detection of diverse SCCmec types, spa types, PFGE types | 277 out of 281 MRSA isolates detected (diverse SCCmec, spa, PFGE types) | 4.3 |
| Genotype Inclusivity (SA) | Detection of diverse spa types | All 85 SA isolates detected (75 spa types) | 4.3 |
| Precision (MRSA) | Overall CV% of Ct values ≤ 1.3% | ≤ 1.3% (all concentrations) | 4.4, Table 8 |
| Precision (SA) | Overall CV% of Ct values ≤ 1.2% | ≤ 1.2% (all concentrations) | 4.4, Table 8 |
| Analytical Specificity (Non-MRSA/SA) | No cross-reactivity with common nasal flora and human cells, no interference | All 135 organisms and human cells produced negative results for targets and did not interfere with spiked targets | 4.5, Tables 9, 10 |
| Analytical Specificity (SA, BORSA, mecA dropouts) | No false negatives/positives for MRSA/SA in presence of these organisms | No false negatives/positives for MRSA with 2 SA, 10 BORSA, and 13/16 mecA dropouts (3 mecA dropouts caused false MRSA POS) | 4.5, Table 11 |
| Competitive Inhibition | No significant effect on Ct value from increasing concentrations of competing organisms | Relatively stable Ct values observed | 4.6, Table 12 |
| Interference (Common OTCs/Antibiotics) | No interference up to reasonable concentrations | No interference for most substances. Rhinaris® Nasal gel (15%), Releev™ (25%) showed interference above specified levels. Tolerated 75% whole blood, 10% mucin. | 4.7, Table 13 |
| Cross-Contamination | 0% cross-contamination rate | 0% (0/423 negative samples in checkerboard runs) | 4.8, Table 14 |
| Carry-over Contamination | 0% carry-over contamination rate | 0% (0/282 negative samples in last runs) | 4.8, Table 14 |
| Clinical Reproducibility (MRSA) PPA | High positive percent agreement for positive samples | 85.6% (Below LOD MRSA-384), 87.2% (Below LOD MRSA-43300), 100.0% (1x & 3x LOD) | 5.1.1, Table 15 |
| Clinical Reproducibility (MRSA) Ct Total CV | Low coefficient of variation for Ct values | ≤ 1.4% | 5.1.1, Table 16 |
| Clinical Reproducibility (SA) PPA | High positive percent agreement for positive samples | 50.0% (Below LOD SA), 99.4% (1x LOD SA), 100.0% (3x LOD SA) | 5.1.2, Table 17 |
| Clinical Reproducibility (SA) Ct Total CV | Low coefficient of variation for Ct values | ≤ 1.3% | 5.1.2, Table 18 |
| Clinical Performance (MRSA) Sensitivity | High sensitivity compared to reference method | 93.1% (149/160) | 5.2.1.1, Table 19 |
| Clinical Performance (MRSA) Specificity | High specificity compared to reference method | 97.5% (2281/2340) | 5.2.1.1, Table 19 |
| Clinical Performance (SA) Sensitivity | High sensitivity compared to reference method | 93.9% (620/660) | 5.2.1.1, Table 19 |
| Clinical Performance (SA) Specificity | High specificity compared to reference method | 94.2% (1734/1841) | 5.2.1.1, Table 19 |
2. Sample Size Used for the Test Set and Data Provenance
The primary clinical performance evaluation used the following test set:
- Sample Size: 2528 subjects, yielding 2504 evaluable results.
- Data Provenance:
- Country of Origin: United States. Specimens were collected at six geographically diverse sites across the US.
- Retrospective or Prospective: Prospective. Specimens were collected prospectively from eligible male and female subjects.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and the Qualifications of Those Experts
The document states that the ground truth for the clinical test set was established by a reference laboratory that specialized in culture and molecular detection of methicillin-resistant MRSA and SA. However, it does not specify the number of individual experts or their specific qualifications (e.g., "radiologist with 10 years of experience"). It implies that the laboratory's established practices for culture and molecular confirmation (including non-selective direct and enrichment culture, Gram-stain, latex agglutination, Kirby-Bauer cefoxitin disc diffusion for MRSA, and laboratory developed PCR assay for femA and mecA genes) served as the "expert" ground truth.
4. Adjudication Method for the Test Set
The ground truth was established by comparing the cobas® MRSA/SA Test results against a combined direct and enrichment culture (reference method).
- Discrepant Analysis: For samples with discordant results between the cobas® MRSA/SA Test and the combined direct and enrichment culture, a second, FDA-cleared nucleic acid amplification test (NAAT) and a non-selective direct and enrichment culture were used for further analysis.
- Control Samples: A randomly selected subset of samples with concordant results were also included in this discrepant analysis as controls.
This method resembles a 2+1 adjudication model in spirit for discordant cases, where the initial test (cobas® MRSA/SA) results are compared against a reference standard (combined culture), and any discrepancies are resolved by a third, independent method (second NAAT and non-selective culture).
5. If a Multi Reader Multi Case (MRMC) Comparative Effectiveness Study was Done
No, a MRMC comparative effectiveness study was not done. The device being evaluated is an in vitro diagnostic real-time PCR assay for direct detection of bacterial DNA, not an imaging device that requires human interpretation. Therefore, the concept of "human readers" or "AI assistance" in the context of improving human reader performance is not applicable to this device.
6. If a Standalone (i.e. algorithm only without human-in-the loop performance) was Done
Yes, the primary clinical performance study (Section 5.2) and all analytical studies (Section 4) represent standalone algorithm-only performance. The cobas® MRSA/SA Test is an automated diagnostic assay that reports qualitative results (Positive/Negative for MRSA/SA) based on real-time PCR amplification and detection. The interpretation of these results is done by the cobas 4800 System Software, which then reports out the final results (Section 6.1.2). Human involvement is in specimen collection, loading, and result interpretation, but the core detection and qualitative output is automated by the device itself.
7. The Type of Ground Truth Used
The ground truth used for the clinical performance evaluation was a combined microbiological culture method:
- Combined Direct and Enrichment Culture: This involved both direct plating onto selective and differential chromogenic media, and enrichment culture in tryptic soy broth.
- Confirmatory Tests: Suspect isolates from these cultures were further identified by Gram-stain, latex agglutination, and for MRSA, Kirby-Bauer cefoxitin disc diffusion.
- Discrepant Analysis: For discordant results, a second FDA-cleared NAAT and non-selective direct and enrichment culture were used, with identification of isolates confirmed using a laboratory developed PCR assay for femA and mecA genes.
This represents a robust, multi-faceted ground truth derived from established microbiological techniques and molecular confirmation.
8. The Sample Size for the Training Set
The document does not explicitly state the sample size for a "training set."
Medical device submissions for in vitro diagnostics like this PCR assay typically focus on analytical validation (LOD, specificity, precision) and clinical validation with an independent test set. The algorithms for interpreting PCR cycle thresholds are generally fixed and based on scientific principles rather than being "trained" on a large dataset in the same way machine learning models are. However, the analytical studies used various isolates to define parameters and confirm performance:
- Analytical Sensitivity (LOD): 2 MRSA culture isolates and 1 SA culture isolate for initial LOD, and 35 additional MRSA and 5 additional SA culture isolates for genotype LOD verification.
- Geographical Inclusivity: 281 MRSA and 85 SA isolates from diverse geographical locations.
- Precision: 2 MRSA culture isolates and 1 SA culture isolate.
- Analytical Specificity: 43 CoNS/MR-CoNS organisms, 93 non-MRSA/SA microorganisms and human cells, 2 SA isolates, 10 BORSA isolates, and 16 mecA dropouts.
These isolates were used to characterize the device's analytical performance across various conditions and strains, which could be considered akin to "training" or extensive characterization data, but not in the machine learning sense of model optimization.
9. How the Ground Truth for the Training Set Was Established
As noted above, there isn't a "training set" in the machine learning context described. The ground truth for the various analytical studies (e.g., sensitivity, specificity, inclusivity) was established through:
- Quantified Culture Isolates: For LOD and precision studies, specific MRSA and SA culture isolates with known concentrations (CFU/swab) were used.
- Characterized Clinical Isolates: For genotype LOD and inclusivity studies, isolates were selected based on known characteristics such as SCCmec type, spa type, PFGE type, MIC values, and geographical origin.
- Established Microbiological Methods: For analytical specificity, cultures of known microorganisms (both targets and non-targets) were used, often from ATCC or other reference collections, and their identity was presumably confirmed by standard microbiological identification techniques.
Essentially, for these analytical studies, the "ground truth" was defined by standard reference strains and well-characterized clinical isolates, obtained and verified through established microbiology laboratory practices.
§ 866.1640 Antimicrobial susceptibility test powder.
(a)
Identification. An antimicrobial susceptibility test powder is a device that consists of an antimicrobial drug powder packaged in vials in specified amounts and intended for use in clinical laboratories for determining in vitro susceptibility of bacterial pathogens to these therapeutic agents. Test results are used to determine the antimicrobial agent of choice in the treatment of bacterial diseases.(b)
Classification. Class II (performance standards).