(25 days)
The BD MAX™ MRSA XT assay performed on the BD MAX™ System is an automated qualitative in vitro diagnostic test for the direct detection of methicillin-resistant Staphylococcus aureus (MRSA) DNA from nasal swabs in patients at risk for nasal colonization. The test utilizes real-time polymerase chain reaction (PCR) for the amplification of MRSA DNA and fluorogenic target-specific hybridization probes for the detection of the amplified DNA. The BD MAX™ MRSA XT assay is intended to aid in the prevention and control of MRSA infections in healthcare settings. It is not intended to diagnose MRSA infections nor guide or monitor treatment for MRSA infections. A negative result does not preclude nasal colonization. Concomitant cultures are necessary to recover organisms for epidemiological typing or for further susceptibility testing.
The BD MAX™ System and the BD MAX™ MRSA XT assay are comprised of an instrument with associated hardware and accessories, disposable microfluidic cartridges, master mixes, unitized reagent strips, extraction reagents, and sample buffer tubes. The instrument automates sample preparation including target lysis, DNA extraction and concentration, reagent rehydration, and target nucleic acid amplification and detection using real-time PCR. The assay includes a Sample Processing Control (SPC) that is present in the Extraction Tube. The SPC monitors DNA extraction steps, thermal cycling steps, reagent integrity and the presence of inhibitory substances. The BD MAX™ System software automatically interprets test results. A test result may be called as MRSA NEG, MRSA POS or MRSA UNR (Unresolved)] based on the amplification status of the target and of the Sample Processing Control. IND (Indeterminate) or INC (Incomplete) results are due to BD MAX™ System failure.
Here's a breakdown of the requested information based on the provided text:
Acceptance Criteria and Device Performance Study for BD MAX™ MRSA XT Assay
1. Table of Acceptance Criteria and Reported Device Performance
The document does not explicitly state pre-defined acceptance criteria for all performance analytical and clinical metrics. However, it presents the results of various studies which implicitly define the performance achieved. For the purpose of this response, I've inferred "acceptance criteria" from the satisfactory performance reported by the manufacturer. The clinical performance is compared against a "Reference Method" (Direct/Enriched Culture).
| Metric / Study Category | Implied Acceptance Criteria (Based on reported performance considered acceptable) | Reported Device Performance (BD MAX™ MRSA XT) |
|---|---|---|
| Precision | High agreement for positive/negative samples, reasonable for low positives. | - True Negative (TN): 100% agreement |
- Moderate Positive (MP): 100% agreement
- Low Positive (LP): 97.9% agreement
- High Negative (HN): 60.4% agreement (No specific acceptance criterion defined for this category, but reported.) |
| Reproducibility (Site-to-Site) | High agreement for positive/negative samples, reasonable for low positives. | - MP: 100% agreement - TN: 100% agreement
- LP: 96.7% agreement
- HN: 63.3% agreement (No specific acceptance criterion defined for this category, but reported.) |
| Reproducibility (Lot-to-Lot) | High agreement for positive/negative samples, reasonable for low positives. | - MP: 100% agreement - TN: 100% agreement
- LP: 96.7% agreement
- HN: 61.1% agreement (No specific acceptance criterion defined for this category, but reported.) |
| Analytical Sensitivity (LoD) | Detection of MRSA strains at low concentrations. | LoD ranged from 64 to 343 CFU/swab at 95% positivity for various MRSA genotypes. |
| Analytical Inclusivity | Detection of a wide range of MRSA strains and types. | All 77 MRSA strains (across various MREJ genotypes, SCCmec types, PFGE types, geographic origins, and including VRSA/VISA) were detected at 2-3x LoD. |
| Analytical Specificity | No detection of non-target organisms (MSSA, CoNS, CoPS, other species, viruses). | - Empty cassette variant MSSA: 15/15 MRSA NEG - Non-staphylococcal species (57 strains): 57/57 MRSA NEG
- Coagulase-Negative/Positive staphylococcal strains (45 strains): 45/45 MRSA NEG
- MSSA (50 strains): 50/50 MRSA NEG
- Viruses (17 species): 17/17 MRSA NEG |
| Interfering Substances | No interference from common nasal substances/microorganisms. | No reportable interference from 28/29 tested substances/microorganisms. Tobramycin showed reportable interference at 4.5 x 10^-3 g/swab. |
| Carryover/Cross-Contamination | Low rate of false positives due to carryover. | 3 false positive results out of 210 expected negative results (1.4%). |
| Clinical Sensitivity (vs. Reference Method) | High agreement for positive specimens. | 93.1% (95% CI: 88.1%, 96.1%) |
| Clinical Specificity (vs. Reference Method) | High agreement for negative specimens. | 97.5% (95% CI: 96.8%, 98.1%) |
| Positive Predictive Value (PPV) (vs. Reference Method) | Reasonable PPV given prevalence. | 73.2% (95% CI: 67.8%, 78.3%) |
| Negative Predictive Value (NPV) (vs. Reference Method) | High NPV. | 99.5% (95% CI: 99.1%, 99.7%) |
| Positive Percent Agreement (vs. Direct Culture) | High agreement for positive specimens. | 96.5% (95% CI: 92.0%, 98.5%) |
| Negative Percent Agreement (vs. Direct Culture) | High agreement for negative specimens. | 96.9% (95% CI: 96.1%, 97.6%) |
| Unresolved Rate (Initial) | Low initial unresolved rate. | 0.6% (16/2399) |
| Unresolved Rate (After Repeat) | Very low unresolved rate after repeat. | 0.1% (2/2398) |
2. Sample Size Used for the Test Set and Data Provenance
-
Clinical Performance Study:
- Test Set Size:
- Reference Method comparison: 2352 compliant specimen results used for sensitivity/specificity calculations. A total of 2451 specimens were initially enrolled, with 94 noncompliant and 5 non-reportable PCR results removed.
- Direct Culture comparison: 2391 compliant specimen results used for positive/negative percent agreement. A total of 2451 specimens were initially enrolled, with 54 noncompliant and 6 non-reportable PCR results removed.
- Data Provenance: Multi-site prospective investigational study. Specimens were collected from patients at risk for nasal colonization from three investigational centers (countries not explicitly stated but typically within the regulatory region, e.g., North America for an FDA submission).
- Test Set Size:
-
Analytical Studies (e.g., Sensitivity, Specificity, Inclusivity, Precision, Reproducibility): These studies used prepared samples (simulated nasal matrix, bacterial suspensions) and well-characterized strains from public collections and clinical isolates. The Analytical Inclusivity study explicitly states that 77 MRSA strains were from 27 countries.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications of Those Experts
- The ground truth for the clinical performance study test set was established using a Comparative Reference Method consisting of direct culture complemented by enriched culture.
- The document does not explicitly state the number of experts or their specific qualifications (e.g., microbiologists, lab technologists) involved in performing or interpreting these culture methods at the clinical sites. However, it details the laboratory procedures: presumptive S. aureus colonies on selective media, subculture to Blood Agar, identification confirmation with an agglutination test, and methicillin-resistance confirmation by Cefoxitin disk diffusion susceptibility testing. For enriched culture, TSB 6.5% NaCl broth was used, followed by inoculation of chromogenic medium and BA plates, with MRSA confirmation as described. This implies trained laboratory personnel using standardized microbiological techniques.
4. Adjudication Method for the Test Set
- The document describes a "Reference Method" (Direct/Enriched Culture) as the independent ground truth standard.
- For discordant results between the BD MAX™ MRSA XT assay and the Reference Method in the clinical study, further investigation was performed.
- "12 of 54 MRSA False Positive BD MAX™ MRSA XT specimens were found to be MRSA POS after repeat of Reference Method."
- "5 of 11 MRSA False Negative BD MAX™ MRSA XT specimens were found to be MRSA NEG after repeat of Reference Method."
- This indicates a form of adjudication by repeat testing using the Reference Method (culture) for discordant clinical samples. It is not a 2+1 or 3+1 expert consensus model, but rather re-evaluation using the established gold standard.
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was Done, If So, What Was the Effect Size of How Much Human Readers Improve with AI vs. Without AI Assistance
- No, an MRMC comparative effectiveness study involving human readers/interpreters with and without AI assistance was not done.
- This device is an automated in vitro diagnostic test (NAAT) for direct detection of MRSA DNA. Its results are automatically interpreted by the BD MAX™ System software (MRSA NEG, MRSA POS, MRSA UNR).
- The comparison is between the automated device performance and traditional culture methods, not between human performance with and without AI assistance.
6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) Was Done
- Yes, a standalone performance evaluation was done.
- The entire premise of the analytical and clinical performance studies is to assess the BD MAX™ MRSA XT assay's performance as an automated qualitative in vitro diagnostic test. The device generates "MRSA NEG, MRSA POS or MRSA UNR (Unresolved)" results automatically.
- The "Substantial Equivalence" section explicitly states that the "Interpretation of Test Results" is "Automated (Diagnostic software of BD MAX™ System)."
- The clinical performance section directly compares "BD MAX™ MRSA XT Assay" results to the Reference Method and Direct Culture.
7. The Type of Ground Truth Used (Expert Consensus, Pathology, Outcomes Data, etc.)
- Clinical Performance Study: The ground truth for clinical performance was established by a Comparative Reference Method consisting of direct culture complemented by enriched culture. This is a laboratory-based microbiological gold standard.
- Analytical Performance Studies (e.g., LoD, Inclusivity, Specificity): The ground truth was established by known concentrations of characterized bacterial strains (e.g., CFU/swab), often from controlled laboratory settings or public reference collections.
8. The Sample Size for the Training Set
- The document does not provide details about a specific training set size for the BD MAX™ MRSA XT assay's algorithm.
- This is a molecular diagnostic test (PCR), and its performance is determined by the design and optimization of the primers and probes, and the reaction conditions, rather than a machine learning model that requires a "training set" in the conventional sense of AI. The analytical validation experiments (LoD, inclusivity, specificity) are used to characterize the assay's performance across a wide range of relevant targets and non-targets, ensuring its robustness and accuracy.
9. How the Ground Truth for the Training Set Was Established
- As noted above, the concept of a "training set" in the context of a machine learning algorithm is not directly applicable here.
- The development and optimization of the PCR assay would involve:
- Designing primers and probes to target specific MRSA DNA sequences (e.g., MREJ, mecA, mecC).
- Testing against characterized bacterial strains with known genotypes, SCCmec types, and concentrations to ensure accurate detection and specificity.
- Optimizing reaction parameters (e.g., temperatures, times, reagent concentrations).
- The ground truth for these developmental and optimization activities would be derived from established molecular biology techniques, bacterial genotyping, sequencing, and quantitative culturing methods to confirm the presence, identity, and concentration of target DNA.
§ 866.1640 Antimicrobial susceptibility test powder.
(a)
Identification. An antimicrobial susceptibility test powder is a device that consists of an antimicrobial drug powder packaged in vials in specified amounts and intended for use in clinical laboratories for determining in vitro susceptibility of bacterial pathogens to these therapeutic agents. Test results are used to determine the antimicrobial agent of choice in the treatment of bacterial diseases.(b)
Classification. Class II (performance standards).