(345 days)
BBL™ CHROMagar™ MRSA II is a selective and differential chromogenic medium for the qualitative direct detection of nasal colonization by methicillin-resistant Staphylococcus aureus (MRSA) to aid in the prevention and control of MRSA infections in healthcare settings. The test is performed on anterior nares swab specimens from patients to screen for MRSA colonization.
BBL CHROMagar MRSA II is not intended to diagnose, guide or monitor treatment for MRSA infections. A negative result does not preclude MRSA nasal colonization. Concomitant cultures are necessary for organism identification, susceptibility testing or epidemiological typing.
BBL™ CHROMagar™ MRSA II (CMRSA II) permits the direct detection and identification of MRSA through the incorporation of specific chromogenic substrates and cefoxitin. MRSA strains will grow in the presence of cefoxitin' and produce mauvecolored colonies resulting from hydrolysis of the chromogenic substrate. Selective agents are incorporated for the suppression of gram-negative organisms, yeast and enterococci and some other gram-positive cocci. Bacteria other than MRSA may utilize other chromogenic substrates in the medium resulting in the growth of colonies that are not mauve.
Here is an analysis of the provided text regarding the BBL™ CHROMagar™ MRSA II device:
Acceptance Criteria and Device Performance
The provided document describes the BBL™ CHROMagar™ MRSA II and its performance in detection of MRSA. The acceptance criteria and reported device performance are primarily presented in terms of Positive Percent Agreement (PPA) and Negative Percent Agreement (NPA) compared to established reference methods.
Table of Acceptance Criteria and Reported Device Performance
| Performance Metric | Acceptance Criteria (Implied/Reference) | Reported Device Performance (BBL™ CHROMagar™ MRSA II) | Notes |
|---|---|---|---|
| Reproducibility | >95% for inter-lot and overall testing intervals | 100% (Individual and combined site reproducibility agreements across three successive days for 10 masked S. aureus strains) | Analytical studies also showed >95% for both inter-lot and overall testing intervals for 20 test strains. |
| PPA vs. Cefoxitin Disk | Not explicitly stated, but typically high agreement is expected for diagnostic devices. | 92% (149/162) (95% CI: 86.7%, 95.7%) | Used as a primary reference method for MRSA status, particularly for the 13 discrepant samples from the Traditional Culture comparison. |
| NPA vs. Cefoxitin Disk | Not explicitly stated, but typically high agreement is expected for diagnostic devices. | 99.9% (1024/1025) (95% CI: 99.5%, 100%) | Used as a primary reference method for MRSA status. |
| PPA vs. Traditional Culture (Combined Sites) | Not explicitly stated, but typically high agreement is expected for diagnostic devices. | 92% (92/100) (95% CI: 84.8%, 96.5%) | This includes 9 samples that were positive on CMRSA II and negative by Traditional Culture, but were subsequently confirmed as MRSA by cefoxitin disk diffusion. This suggests the Traditional Culture might have missed some true positives. |
| NPA vs. Traditional Culture (Combined Sites) | Not explicitly stated, but typically high agreement is expected for diagnostic devices. | 98.8% (760/769) (95% CI: 97.8%, 99.5%) | |
| PPA vs. Traditional Culture (Third Site) | Not explicitly stated, but typically high agreement is expected for diagnostic devices. | 90.2% (46/51) (95% CI: 78.6%, 96.7%) | This includes 2 samples that were positive on CMRSA II and negative by Traditional Culture, but were subsequently confirmed as MRSA by cefoxitin disk diffusion. |
| NPA vs. Traditional Culture (Third Site) | Not explicitly stated, but typically high agreement is expected for diagnostic devices. | 98.9% (264/267) (95% CI: 96.8%, 99.8%) |
Study Details
2. Sample size used for the test set and the data provenance
- Clinical Accuracy Test Set Size: A total of 1187 compliant remnant, prospective, naive specimens.
- Data Provenance: The study was conducted at three diverse clinical laboratories in the U.S. The data is prospective, as it was collected from "remnant, prospective, nares specimens."
3. Number of experts used to establish the ground truth for the test set and the qualifications of those experts
The document does not specify the number of experts used in establishing the ground truth, nor does it detail their qualifications. The ground truth was based on laboratory methods (cefoxitin disk diffusion and traditional culture), which would have been performed and interpreted by trained laboratory personnel, but not necessarily "experts" in the sense of clinicians or radiologists.
4. Adjudication method (e.g. 2+1, 3+1, none) for the test set
The document does not explicitly describe an adjudication method for the test set in the conventional sense of multiple human readers reviewing results. Instead, any discrepancies between the BBL™ CHROMagar™ MRSA II and the Traditional Culture were reconciled by referring to the cefoxitin disk diffusion test method as the definitive reference. For example, 9 samples from the combined sites and 2 samples from the third site that were positive on CMRSA II but negative by Traditional Culture were confirmed as MRSA by cefoxitin disk diffusion.
5. If a multi reader multi case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance
No, a multi-reader multi-case (MRMC) comparative effectiveness study was not done. This study focuses on the performance of a culture medium device (BBL™ CHROMagar™ MRSA II), not an AI algorithm or an assistance tool for human readers. Therefore, there is no information on human reader improvement with or without AI assistance.
6. If a standalone (i.e. algorithm only without human-in-the-loop performance) was done
This is a standalone performance evaluation of a culture medium, not an algorithm. The device itself (BBL™ CHROMagar™ MRSA II) performs the detection based on phenotypic characteristics (mauve colonies and growth in cefoxitin). The interpretation of positive or negative (presence of mauve colonies) is done by laboratory personnel directly observing the plate, which could be considered a "human-in-the-loop" step, but the core analytical performance is derived from the medium itself reacting to the bacteria. However, it's not an "algorithm-only" or "AI" study in the typical sense.
7. The type of ground truth used (expert consensus, pathology, outcomes data, etc.)
The ground truth for the clinical accuracy study was primarily based on:
- Cefoxitin Disk Diffusion Test Method: This is a standard laboratory method for determining methicillin resistance in Staphylococcus aureus.
- Traditional Culture Media: This served as a comparative method.
- Adjudication by Cefoxitin Disk: For discrepancies between the BBL™ CHROMagar™ MRSA II and Traditional Culture, the cefoxitin disk diffusion test was used to confirm MRSA status.
This combination of laboratory-based tests represents the "gold standard" or reference method for determining the presence and methicillin resistance of S. aureus in clinical samples.
8. The sample size for the training set
The document does not mention a "training set" in the context of machine learning or AI. Rather, it refers to:
- Analytical Studies: Used 20 test strains for reproducibility, and 7 test strains for Limit of Detection (LOD) and analytical reactivity. A suspension of 1x10^5 CFU/mL was used for these studies.
- Challenge Strain Testing: Involved 20 challenge strains of S. aureus (14 MRSA, 6 MSSA).
- Reproducibility Testing (Clinical): Involved 10 masked strains of S. aureus (7 MRSA, 3 MSSA).
These "strains" or "test strains" function as internal controls and known samples to establish the device's inherent analytical capabilities, similar to a "development" or "validation" set for the device's formulation, but not a "training set" for an algorithm.
9. How the ground truth for the training set was established
As there is no "training set" in the AI sense, there isn't a ground truth established for one. For the analytical and challenge strain testing, the ground truth was inherently known because these were cultured strains with confirmed characteristics (MRSA/MSSA status). The document does not detail the exact methodology for confirming the MRSA status of these "training" or "challenge" strains, but it would have been through standard microbiological methods like those mentioned (e.g., genetic testing or other antimicrobial susceptibility testing to oxacillin/cefoxitin).
§ 866.1700 Culture medium for antimicrobial susceptibility tests.
(a)
Identification. A culture medium for antimicrobial susceptibility tests is a device intended for medical purposes that consists of any medium capable of supporting the growth of many of the bacterial pathogens that are subject to antimicrobial susceptibility tests. The medium should be free of components known to be antagonistic to the common agents for which susceptibility tests are performed in the treatment of disease.(b)
Classification. Class II (performance standards).