(207 days)
The Great Basin Staph ID/R Blood Culture Panel is a qualitative, multiplex, nucleic acid-based in vitro diagnostic assay intended for the simultaneous identification of nucleic acid from Staphylococcus lugdumensis and various Staphylococcus species to the genus level and the detection of the mecA gene for methicillin resistance directly from patient positive blood culture specimens. The test utilizes automated hot-start enabled polymerase chain reaction (PCR) for the amplification of specific DNA targets detected by hybridization probes immobilized on a silicon chip surface. The assay is performed directly on positive blood culture specimens identified as positive by continuous monitoring blood culture system that demonstrates the presence of organisms as determined by Gram stain to contain gram-positive cocci in clusters (GPCC) or gram-positive cocci in singles (GPC). The test may be performed using blood culture bottles. The Staph ID/R Blood Culture Panel identifies Staphylococcus aureus (SA), and Staphylococcus lugdunensis, and detects other Staphylococcus species without identification to species level.
The Portrait Staph ID/R Blood Culture Panel is indicated for use in conjunction with other clinical or laboratory findings to aid in the diagnosis of bacterial bloodstream infections; however, it is not used to monitor these infections. Subculturing positive blood cultures is necessary to recover viable organisms for further identification, susceptibility testing, or epidemiological typing to identify organisms in the blood culture that are not detected by the Great Basin Staph ID/R Blood Culture Panel. If detected, mecA may or may not be associated with Staphylococcus spp. detected or the agent responsible for the disease. Negative results for mecA antimicrobial resistance gene assays do not always indicate susceptibility, as other mechanisms of resistance to methicillin exist.
The Great Basin PA500 Portrait™ Analyzer System is a fully automated system that includes the Portrait Analyzer, single-use Staph ID/R Blood Culture Panel Test Cartridges, and the Portrait data analysis software. The PA500 Portrait Analyzer System is designed to perform automated sample preparation, PCR, and optical chip-based detection with integrated data analysis in approximately 110 minutes.
The Great Basin Staph ID/R Blood Culture Panel is a qualitative, multiplex, nucleic acid-based in vitro diagnostic assay intended for the simultaneous identification of nucleic acid from Staphylococcus aureus, Staphylococcus lugdunensis, and various Staphylococcus species to the genus level, and the detection of the mecA gene for methicillin resistance directly from patient positive blood culture specimens. The device uses automated hot-start enabled PCR for amplification and hybridization probes on a silicon chip for detection.
Here's an analysis of its acceptance criteria and the study that proves the device meets them:
1. Table of Acceptance Criteria and Reported Device Performance
| Feature/Metric | Acceptance Criteria (Stated or Implied) | Reported Device Performance |
|---|---|---|
| Analytical Sensitivity (LoD) - S. aureus | Not explicitly stated, but high accuracy expected at LoD. | LoDs for 6 S. aureus strains ± mecA range from 3.5-8.2 x 10⁵ CFU/mL, with 20/20 to 23/23 correct results for individual strains. |
| Analytical Sensitivity (LoD) - S. epidermidis | Not explicitly stated, but high accuracy expected at LoD. | LoDs for 6 S. epidermidis strains ± mecA range from 2.2-7.1 x 10⁵ CFU/mL, with 20/20 to 27/27 correct results for individual strains. |
| Analytical Sensitivity (LoD) - S. lugdunensis | Not explicitly stated, but high accuracy expected at LoD. | LoDs for 3 S. lugdunensis strains ± mecA range from 2.8-4.7 x 10⁵ CFU/mL, with 22/23 to 23/23 correct results for individual strains. |
| Analytical Reactivity (Inclusivity) | 100% detection of target strains. | Correctly detected all 48 additional well-characterized S. aureus strains (USA100-1200 and SCCmecA I-XI types) and 104 untyped strains (S. aureus, S. epidermidis, S. lugdunensis, and other Staphylococcus species) tested in the challenge panel. |
| Analytical Specificity (Exclusivity) | No cross-reactivity with off-panel organisms. | The vast majority of 116 off-panel microflora strains tested 'Staphylococcus NEGATIVE'. 6 'Staphylococcus POSITIVE' calls (identified as single contamination events and resolved upon re-testing as 'Staphylococcus NEGATIVE'). 27 'invalid' calls (resolved upon re-testing as 'Staphylococcus NEGATIVE'). |
| Microbial Interference (Off-Panel) | No interference with Staphylococcus detection. | For 'valid' runs, off-panel microorganisms did not interfere with detection of Staphylococcus strains (2/2 correct calls for most combinations). In cases of initial miscalls, re-testing at higher concentrations resolved the issue. |
| Microbial Interference (Staphylococcus) | No interference with Staphylococcus detection when co-cultured. | Staphylococcus interference observed in some cases at initial concentrations, but resolved upon re-testing at higher concentrations (within 2-3X LoD). The majority of tests showed no interference (2/2 correct calls). |
| Interfering Substances (Chemical Interference) | No interference with Staphylococcus detection. | All 16 tested substances did not interfere with the detection of Staphylococcus strains or the E. faecalis (Negative) control, resulting in expected Positive or Negative calls. A few 'Invalid' runs (max 1 invalid out of 2) were observed in specific cases, but those were still considered acceptable for the purpose of the study. |
| Carry-over/Cross-Contamination | No evidence of contamination. | All results were in concordance with expected calls; no evidence of contamination was found. |
| Reproducibility | High agreement with expected results across sites, operators, and lots. | - Overall valid results: 98.1% (630/642). |
- Staphylococcus positivity: 100% (540/540) for positive, 96.7% (87/90) for negative.
- Specific Staphylococcus detection: 99.6% (538/540) for positive, 99.3% (1341/1350) for negative.
- mecA detection (no organism): 99.7% (359/360) for present, 100% (270/270) for absent. |
| Evaluation of Blood Culture Bottle Types | Compatibility with various bottle types. | All 13 tested blood culture bottle types were compatible with the Staph ID/R Blood Culture Panel, with no false negative results. One false positive S. aureus and one false positive Staphylococcus species OTHER than S. aureus or S. lugdunensis were observed in a single test run each, attributed to contamination. Nine invalid calls for E. faecalis. |
| Prospective Clinical Performance - S. aureus | PPA ≥ 95%, NPA ≥ 95% (implied). | - PPA: 98.6% (211/214) (95% CI: 96.0 - 99.5%) - NPA: 99.5% (548/551) (95% CI: 98.4 - 99.8%) |
| Prospective Clinical Performance - S. lugdunensis | PPA ≥ 95%, NPA ≥ 95% (implied). | - PPA: 100.0% (3/3) (95% CI: 43.9 - 100%) - NPA: 99.9% (761/762) (95% CI: 99.3 - 99.9%) |
| Prospective Clinical Performance - Other Staphylococcus spp. | PPA ≥ 95%, NPA ≥ 95% (implied). | - PPA: 98.9% (444/449) (95% CI: 97.4 - 99.5%) - NPA: 97.2% (307/316) (95% CI: 94.7 - 98.5%) |
| Prospective Clinical Performance - mecA with S. aureus | PPA ≥ 90%, NPA ≥ 98% (implied). | - PPA: 94.4% (68/72) (95% CI: 86.6 - 97.8%) - NPA: 98.8% (682/690) (95% CI: 97.7 - 99.4%) |
| Prospective Clinical Performance - mecA with Other Staphylococcus spp. | PPA ≥ 90%, NPA ≥ 95% (implied). | - PPA: 92.7% (243/262) (95% CI: 88.1 - 97.1%) - NPA: 96.2% (481/500) (95% CI: 92.4 - 98.0%) |
| Invalid Rate - Clinical | Acceptable low rate, resolvable on retest. | Initial Invalid Rate: 1.39% (11/789). Final Invalid Rate: 0.00% after retest. |
| Abort Rate - Clinical | Acceptable low rate, resolvable on retest. | Initial Abort Rate: 3.30% (26/789). Final Abort Rate: 0.00% after retest. |
2. Sample Size Used for the Test Set and Data Provenance
-
Prospective Clinical Study:
- Sample Size: A total of 762 prospective specimens were tested. Additionally, 69 archived frozen specimens were tested after the prospective study, making the overall clinical sample set significantly larger. A 'Low Prevalence' panel of 102 contrived or 'simulated' blood culture specimens was also tested.
- Data Provenance: The prospective specimens were collected at three geographically diverse U.S. sites.
-
Analytical Studies (LoD, Inclusivity, Exclusivity, Interference, Bottle Types, Reproducibility):
- These studies used various numbers of cultured bacterial strains (ATCC, NCTC, CCUG, Clinical isolates where specified) and simulated blood culture specimens, often with multiple replicates per strain (ranging from 2 to 20+ replicates per strain for LoD, 2 replicates per strain for exclusivity, and 90 replicates per analyte for reproducibility). The specific number of unique strains and replicates varies per study as detailed in the tables (e.g., 22 strains for LoD, 116 off-panel strains for exclusivity, 7 simulated blood culture specimens for reproducibility).
- Data Provenance: Not explicitly stated for each analytical study, but context suggests laboratory-controlled experiments. Strains were from recognized culture collections (ATCC, NCTC, CCUG) or clinical sources.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications of those Experts
The document does not specify the number of experts or their qualifications used to establish the ground truth for the clinical or analytical test sets.
For the clinical study, the ground truth (Staphylococcus species identification and mecA gene presence) was established using "Reference Method(s)". While not explicitly defined, these reference methods typically involve conventional microbiology techniques such as:
- Sub-culturing of positive blood cultures.
- Phenotypic identification (e.g., Gram stain, catalase, coagulase tests, biochemical panels).
- Molecular methods (e.g., gene sequencing, PCR assays with a different target).
- Antimicrobial susceptibility testing (e.g., oxacillin MIC) to confirm methicillin resistance (though the device detects the mecA gene, not phenotypic resistance itself).
For analytical studies, bacterial strains from recognized collections (ATCC, NCTC, CCUG) and presumably well-characterized clinical isolates were used, implying their identity and mecA status were already established by standard microbiological and genetic methods.
4. Adjudication Method for the Test Set
The document does not explicitly describe an adjudication method (like 2+1 or 3+1) for the clinical test set. The comparison is made directly against "Reference Method(s)". For discrepant results in the exclusivity and microbial interference studies, repeat testing was performed (e.g., a minimum of six (6) repeat tests for 'Staphylococcus POSITIVE' calls in exclusivity, sometimes at higher CFU/mL input for interference studies to resolve miscalls).
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was Done
No, a Multi-Reader Multi-Case (MRMC) comparative effectiveness study was not conducted. This device is an automated in vitro diagnostic assay (algorithm only), not an AI-assisted human reading system. Therefore, there is no human-in-the-loop performance to measure improvement with AI assistance.
6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) was Done
Yes, the studies conducted demonstrate the standalone performance of the Great Basin Staph ID/R Blood Culture Panel. This device is an automated, qualitative, molecular diagnostic system, meaning its output is generated by the instrument itself (the Portrait Analyzer and its software) without direct human interpretation of complex images or data that would typically benefit from AI assistance. The performance metrics (PPA, NPA) directly reflect the device's accuracy against reference methods.
7. The Type of Ground Truth Used
The primary ground truth for both analytical and clinical performance studies was established through reference microbiology methods. This includes:
- Culture-based identification: Sub-culturing, Gram staining, and phenotypic tests for species identification.
- Molecular identification: Implied use of validated molecular techniques or genetic characterization (e.g., for mecA gene presence and confirmation of species for various strains).
- Antimicrobial Susceptibility Testing (AST): Specifically, oxacillin MIC was used in the Analytical Reactivity (Inclusivity) Challenge Panel to determine phenotypic resistance, which indirectly supports the genotypic mecA detection.
- For the analytical studies, well-characterized strains from culture collections (ATCC, NCTC, CCUG) served as a strong foundation for ground truth.
8. The Sample Size for the Training Set
The document does not explicitly mention a "training set" in the context of device development or any specific AI/machine learning models. This is typical for traditional molecular diagnostic assays, where assay design (primer/probe sequences, reaction conditions) is based on known genetic targets and verified through extensive analytical validation rather than data-driven machine learning training. The analytical studies (LoD, inclusivity, exclusivity) serve to validate the analytical performance of the developed assay against a wide range of relevant organisms and conditions.
9. How the Ground Truth for the Training Set Was Established
As no explicit "training set" is described for an AI/ML model, this question is not directly applicable. If one considers the development of the assay (design of specific DNA targets and hybridization probes) as analogous to "training," then the ground truth would have been established through extensive molecular biology research, genetic sequencing, and characterization of Staphylococcus species and the mecA gene. This would involve identifying conserved and variable regions of target genes across a broad collection of known Staphylococcus strains to ensure specificity and sensitivity.
§ 866.3365 Multiplex nucleic acid assay for identification of microorganisms and resistance markers from positive blood cultures.
(a)
Identification. A multiplex nucleic acid assay for identification of microorganisms and resistance markers from positive blood cultures is a qualitative in vitro device intended to simultaneously detect and identify microorganism nucleic acids from blood cultures that test positive by Gram stain or other microbiological stains. The device detects specific nucleic acid sequences for microorganism identification as well as for antimicrobial resistance. This device aids in the diagnosis of bloodstream infections when used in conjunction with other clinical and laboratory findings. However, the device does not replace traditional methods for culture and susceptibility testing.(b)
Classification. Class II (special controls). The special control for this device is FDA's guideline document entitled “Class II Special Controls Guideline: Multiplex Nucleic Acid Assay for Identification of Microorganisms and Resistance Markers from Positive Blood Cultures.” For availability of the guideline document, see § 866.1(e).