(11 days)
The Verigene® Gram Positive Blood Culture Nucleic Acid Test (BC-GP) performed using the sample-to-result Verigene System is a qualitative, multiplexed in vitro diagnostic test for the simultaneous detection and identification of potentially pathogenic gram-positive bacteria which may cause bloodstream infection (BSI). BC-GP is performed directly on positive blood culture using BACTEC™ Plus Aerobic/F and BacT/ALERT FA FAN® Aerobic blood culture bottles, which contain gram positive bacteria. BC-GP is indicated for use in conjunction with other clinical and laboratory findings, such as culture, to aid in the diagnosis of bacterial bloodstream infections; however, it is not used to monitor bloodstream infections.
BC-GP detects and identifies the following bacterial genera and species:
Staphylococcus spp.
Staphylococcus aureus
Staphylococcus epidermidis
Staphylococcus lugdunensis
Streptococcus spp.
Streptococcus pneumoniae
Streptococcus pyogenes
Streptococcus agalactiae
Streptococcus anginosus group
Enterococcus
faecalis
Enterococcus
faecium
Listeria spp.
In addition, BC-GP detects the mecA resistance marker, inferring mecA-mediated methicillin resistance, and the vanA and vanB resistance markers, inferring vanA/vanB-mediated vancomycin resistance. In mixed growth, BC-GP does not specifically attribute van-mediated vancomycin resistance to either E. faecium, or mecA-mediated methicillin resistance to either S. aureus or S. epidermidis.
BC-GP is indicated for use in conjunction with other clinical and laboratory findings to aid in the diagnosis of bacterial bloodstream infections; however, is not to be used to monitor these infections. Sub-culturing of positive blood cultures is necessary to recover organisms for susceptibility testing. identification of organisms not detected by BC-GP, differentiation of mixed growth, association of antimicrobial resistance marker genes to a specific organism, or for epidemiological typing.
The Verigene® Gram Positive Blood Culture Nucleic Acid Test (BC-GP) is a molecular assav which relies on detection of specific nucleic acid targets in a microarray format. For each of the bacterial nucleic acid sequences detected by the BC-GP test, Capture and Mediator oligonucleotides are utilized for gold nanoparticle probe-based endpoint detection. The Capture oligonucleotides bind to a specific portion of the nucleic acid target and are themselves bound onto a substrate in the microarray. The Mediator oligonucleotides bind to a different portion of the same nucleic acid target and allow binding of a gold nanoparticle probe to a portion complementary to a gold nanoparticle probe. Specific silver enhancement of the bound gold nanoparticle probes at the capture sites results in gold-silver aggregates that scatter light with high efficiency.
The BC-GP test is performed on the Verigene® System, a 'sample-to-result', fully automated, bench-top molecular diagnostics workstation. The Verigene System consists of two components: the Verigene Reader and the Verigene Processor SP. The BC-GP test utilizes single-use disposable test consumables and a self-contained Verigene Test Cartridge for each sample tested. For the BC-GP test, the Verigene System allows automated nucleic acid extraction from Gram-positive bacteria-containing blood culture specimens and target detection of bacteria-specific DNA.
The Reader is the Verigene System's user interface, which serves as the central control unit for all aspects of test processing and results generation. The Reader's graphical user interface guides the user through test processing and test results using a barcode scanner. The user inserts the Test Cartridge into the Verigene Processor SP, which executes the test procedure. automating the steps of (1) Sample Preparation - Cell lysis and magnetic bead-based bacterial DNA isolation from blood culture samples and (2) Verigene Hybridization Test - Detection and identification of bacterial-specific DNA in a microarray format by using gold nanoparticle probe-based technology.
After test processing is complete, to obtain the test results the user removes the Test Cartridge from the Processor SP, removes the reagent pack from the substrate holder, and inserts the substrate holder into the Reader for analysis. Light scatter from the capture spots is imaged by the Reader and intensities from the microarray spots are used to make decisions regarding the presence (Detected) or absence (Not Detected) of a bacterial nucleic acid sequence/analyte.
The provided text describes the evaluation of the Verigene® Gram Positive Blood Culture Nucleic Acid Test (BC-GP). While specific "acceptance criteria" defined as a numerical threshold for performance are not explicitly stated in a single section, the document outlines the performance requirements for various aspects of the device (limit of detection, inclusivity, specificity, interference, precision, and clinical agreement). The "reported device performance" is then presented in the clinical study results, allowing for an implicit comparison against acceptable ranges for such diagnostic assays.
Here's an organized breakdown of the requested information:
1. A table of acceptance criteria and the reported device performance
The document doesn't present explicit acceptance criteria with specific numerical thresholds for each analyte. Instead, it describes general expectations for "acceptable specificity," "expected results," and "high concordance." For the purpose of this table, I will infer general performance expectations from the overall context and the nature of diagnostic assay validation (e.g., generally aiming for very high agreement for positive and negative cases, especially for critical diagnoses like bloodstream infections). The reported device performance is directly extracted from the clinical study results.
Inferred Acceptance Criteria vs. Reported Device Performance
| Performance Metric | Inferred/Implicit Acceptance Criteria | Reported Device Performance (as per Clinical Study) |
|---|---|---|
| Analyte Detection (General) | High positive and negative agreement for all targeted organisms. | Staphylococcus aureus (SA): |
| - Positive Agreement: Fresh 99.1% (322/325), Frozen 100% (10/10), Total 99.1% (332/335) | ||
| - Negative Agreement: Fresh 100% (926/926), Frozen 100% (165/165), Total 100% (1091/1091) | ||
| Staphylococcus species: | ||
| - Positive Agreement: Fresh 97.9% (895/914), Frozen 100% (30/30), Total 98.0% (925/944), Simulated 100% (25/25) | ||
| - Negative Agreement: Fresh 99.4% (335/337), Frozen 99.3% (144/145), Total 99.4% (479/482), Simulated 100% (191/191) | ||
| Staphylococcus epidermidis (SE): | ||
| - Positive Agreement: Fresh 93.0% (294/316), Frozen 100% (2/2), Total 93.1% (296/318) | ||
| - Negative Agreement: Fresh 98.7% (923/935), Frozen 100% (173/173), Total 98.9% (1096/1108), Simulated 100% (214/214) | ||
| Listeria species: | ||
| - Positive Agreement: Fresh 100% (3/3), Frozen N/A, Total 100% (3/3), Simulated 100% (34/34) | ||
| - Negative Agreement: Fresh 100% (1248/1248), Frozen 100% (175/175), Total 100% (1423/1423), Simulated 100% (182/182) | ||
| Enterococcus faecalis (EFL): | ||
| - Positive Agreement: Fresh 96.0% (72/75), Frozen 100% (21/21), Total 96.9% (93/96), Simulated 92.3% (12/13) | ||
| - Negative Agreement: Fresh 99.9% (1175/1176), Frozen 100% (154/154), Total 99.9% (1329/1330), Simulated 100% (203/203) | ||
| Enterococcus faecium (EFC): | ||
| - Positive Agreement: Fresh 94.4% (34/36), Frozen 100% (32/32), Total 97.1% (66/68), Simulated 100% (46/46) | ||
| - Negative Agreement: Fresh 100% (1215/1215), Frozen 100% (143/143), Total 100% (1358/1358), Simulated 99.4% (169/170) | ||
| Streptococcus spp.: | ||
| - Positive Agreement: Fresh 91.7% (143/156), Frozen 98.4% (63/64), Total 93.6% (206/220), Simulated 100% (92/92) | ||
| - Negative Agreement: Fresh 99.5% (1090/1095), Frozen 100% (111/111), Total 99.6% (1201/1206), Simulated 99.2% (123/124) | ||
| S. pneumoniae: | ||
| - Positive Agreement: Fresh 100% (25/25), Frozen 100% (13/13), Total 100% (38/38), Simulated 100% (8/8) | ||
| - Negative Agreement: Fresh 99.6% (1221/1226), Frozen 100% (162/162), Total 99.6% (1383/1388), Simulated 100% (208/208) | ||
| S. agalactiae: | ||
| - Positive Agreement: Fresh 97.5% (39/40), Frozen 100% (31/31), Total 98.6% (70/71), Simulated 100% (6/6) | ||
| - Negative Agreement: Fresh 100% (1211/1211), Frozen 100% (144/144), Total 100% (1355/1355), Simulated 100% (210/210) | ||
| S. pyogenes: | ||
| - Positive Agreement: Fresh 100% (10/10), Frozen 92.9% (13/14), Total 95.8% (23/24), Simulated 98.2% (53/54) | ||
| - Negative Agreement: Fresh 100% (1241/1241), Frozen 100% (161/161), Total 100% (1402/1402), Simulated 100% (162/162) | ||
| S. anginosus group: | ||
| - Positive Agreement: Fresh 100% (9/9), Frozen 100% (3/3), Total 100% (12/12), Simulated 100% (23/23) | ||
| - Negative Agreement: Fresh 99.8% (1239/1242), Frozen 100% (172/172), Total 99.8% (1411/1414), Simulated 99.5% (192/193) | ||
| Resistance Marker Detection | Very high positive and negative agreement for mecA, vanA, and vanB. | mecA (S. aureus): |
| - Positive Agreement: 97.5% (157/161) | ||
| - Negative Agreement: 98.8% (172/174) | ||
| mecA (S. epidermidis): | ||
| - Positive Agreement: 92.0% (219/238) | ||
| - Negative Agreement: 81.5% (75/92) | ||
| vanA (E. faecalis): | ||
| - Positive Agreement: 85.7% (12/14) | ||
| - Negative Agreement: 100% (95/95) | ||
| vanA (E. faecium): | ||
| - Positive Agreement: 97.2% (69/71) | ||
| - Negative Agreement: 93.0% (40/43) | ||
| vanB (E. faecalis): | ||
| - Positive Agreement: 100% (7/7) | ||
| - Negative Agreement: 100% (102/102) | ||
| vanB (E. faecium): | ||
| - Positive Agreement: 97.0% (32/33) | ||
| - Negative Agreement: 100% (81/81) | ||
| Limit of Detection (LoD) | Lowest target concentration detectable ~95% of the time, with 20 replicates for confirmation. | Confirmed LoD ranges for various strains: S. aureus (1.9x10^6 - 5.7x10^6 CFU/mL), S. epidermidis (2.0x10^6 - 7.5x10^6 CFU/mL), E. faecalis (1.1x10^7 - 5.7x10^7 CFU/mL), E. faecium (2.4x10^7 - 3.7x10^7 CFU/mL), Streptococcus spp. (1.8x10^6 - 1.2x10^8 CFU/mL), S. lugdunensis (3.4x10^6 - 4.0x10^6 CFU/mL), S. pneumoniae (1.8x10^6 - 9.9x10^6 CFU/mL), Listeria spp. (7.5x10^6 - 1.2x10^7 CFU/mL). |
| Inclusivity | All targeted organisms and resistance markers are detected. Expected non-detection for specified exceptions. | All tested strains of targeted organisms and resistance markers detected, with the exception of Listeria grayi and BORSA strains (mecA negative). This was consistent with in silico analysis and expectation. |
| Analytical Specificity (Exclusivity) | No cross-reactivity with phylogenetically related organisms or common blood culture contaminants. | Acceptable specificity demonstrated with no cross-reactivity observed, except for two strains of Enterococcus avium (identified as E. faecium) and Lactococcus lactis (identified as Streptococcus species). |
| Interference | Presence of potential interferents (hemoglobin, triglycerides, bilirubin, γ-globulin, SPS) should not affect test results. | All replicates for all strains and negative controls gave expected results, indicating no interference by tested substances. Additional bottle types also showed no interference. |
| Competitive Inhibition | Detection of organisms at LoD levels should not be inhibited by other targeted organisms at high concentrations. | No evidence of competitive inhibition, except for one initial replicate of S. epidermidis (SE) not detected when mixed with S. agalactiae at high titer. Further testing (20/20 replicates) confirmed detection. |
| Carry-Over and Cross Contamination | No carryover/cross-contamination from high positive to true negative samples. | No evidence of carryover/cross-contamination observed. |
| Precision/Reproducibility | High agreement (e.g., >95% or >99%) across different operators, days, runs, instruments, and reagent lots. | Precision (In-house): |
| - S. aureus and S. epidermidis: 100% (576/576) | ||
| - E. faecalis and E. faecium: 99.8% (575/576) | ||
| - Streptococcus spp., S. lugdunensis, Listeria spp.: 99.9% (767/768) | ||
| Reproducibility (External Sites): | ||
| - S. aureus and S. epidermidis: 100% (720/720) | ||
| - E. faecalis and E. faecium: 100% (720/720) | ||
| - Streptococcus spp., S. lugdunensis, Listeria spp.: 100% (960/960) | ||
| Fresh versus Frozen Study | 100% agreement between fresh and frozen samples for all analytes. | 100% agreement demonstrated between fresh and frozen samples. |
| Initial No-Call Rate | Reasonably low rate. | Initial No-call rate: 4.7% (77/1642). Final No-call rate: 1.1% (18/1642). |
2. Sample size used for the test set and the data provenance (e.g., country of origin of the data, retrospective or prospective)
- Test Set Sample Size:
- Clinical Study: 1767 specimens in total were initially enrolled.
- 1251 fresh specimens
- 175 frozen specimens
- 216 contrived blood culture specimens (for rare organisms).
- 125 specimens were excluded for various reasons (e.g., training sample, culture/reference method results unavailable, sample not tested on BC-GP, QC/protocol error).
- The total evaluable specimens for analysis are not explicitly stated as a single number after exclusions, but performance tables typically reflect data from the 1426 (1251 fresh + 175 frozen) prospectively/retrospectively collected samples + 216 contrived samples, minus the excluded non-evaluable samples where applicable to each specific analyte calculation.
- Clinical Study: 1767 specimens in total were initially enrolled.
- Data Provenance:
- Country of Origin: Not explicitly stated, but the study was conducted at "five external, geographically-diverse clinical study sites," implying multiple locations within a single country (likely the USA, given the FDA context).
- Retrospective or Prospective: The clinical study incorporated both:
- Most specimens were collected and tested as "fresh" (implying prospective at time of collection relative to the BC-GP testing).
- Some were "stored frozen prior to testing" (implying retrospective collection for later testing).
- A portion was "contrived" (simulated specimens for rare organisms).
3. Number of experts used to establish the ground truth for the test set and the qualifications of those experts (e.g., radiologist with 10 years of experience)
- The document does not specify the number of experts or their specific qualifications (e.g., "radiologist with 10 years of experience").
- However, it details the reference methods used to establish ground truth:
- "conventional biochemical, culture, and bidirectional sequencing reference methods."
- "culture followed by testing blood culture isolates with conventional biochemicals, Vitek2, and cefoxitin disc testing."
- "Cefoxitin discs were used as the reference method for confirming mecA mediated resistance in S. aureus and S. epidermidis."
- "Vancomycin resistance and the presence of vanA or vanB in E. faecium and E. faecalis was performed using vancomycin E-tests followed by bidirectional sequencing on resistant organisms."
- This implies that the ground truth was established by laboratory professionals using established clinical microbiology techniques, but their individual expertise is not quantified.
4. Adjudication method (e.g., 2+1, 3+1, none) for the test set
- The document does not explicitly describe a formal adjudication method (like "2+1" or "3+1") for resolving discrepancies between the BC-GP test and the reference methods.
- It states that "BC-GP Test results were compared with results from traditional laboratory reference methods." Discrepancies are reported (e.g., in mixed cultures, 6 "false positives" and 25 "false negatives" compared to reference culture). However, the process for investigating or resolving these discrepancies to arrive at a final "ground truth" for the discrepant cases is not detailed.
- It does mention that the "initial No-call rate in the clinical study was 4.7% (77/1642), and the final No-call rate was 1.1% (18/1642)," implying that some initial no-calls were resolved or re-tested successfully, but this isn't an "adjudication" of BC-GP vs. reference method.
5. If a multi-reader multi-case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance
- No, a multi-reader multi-case (MRMC) comparative effectiveness study was NOT done.
- This device is an automated molecular diagnostic assay, not an imaging device that requires human "readers" or interpretation in the same way an AI for radiology would. Its performance is compared directly against established laboratory reference methods, not against human reader performance, nor is it designed to "assist" human readers in a direct interpretation task. The "Reader" component of the Verigene System is an automated instrument for analyzing the microarray, not a human.
6. If a standalone (i.e., algorithm only without human-in-the-loop performance) was done
- Yes, the provided study describes standalone performance of the algorithm.
- The Verigene BC-GP test is a "sample-to-result, fully automated, bench-top molecular diagnostics workstation." The entire process, from sample preparation to target detection and result generation, is automated by the Verigene System. The clinical study evaluates the performance of this automated system (algorithm only) against traditional culture-based reference methods. There is no human "in-the-loop" for interpreting the BC-GP test results; the system generates "Detected" or "Not Detected" calls.
7. The type of ground truth used (expert consensus, pathology, outcomes data, etc.)
- The ground truth for the clinical study was established using a combination of traditional microbiological reference methods and gold-standard molecular techniques:
- Culture: Isolation and identification of microorganisms through conventional culture methods.
- Biochemicals: Conventional biochemical tests for species identification.
- Vitek2: An automated system for microbial identification and antimicrobial susceptibility testing.
- Cefoxitin Disk Diffusion: For mecA-mediated methicillin resistance.
- Vancomycin E-tests: For vancomycin resistance.
- Bidirectional Sequencing: For vanA and vanB resistance markers.
- This represents a robust and well-accepted set of methods for establishing ground truth in clinical microbiology.
8. The sample size for the training set
- The document mentions "Training Sample" as a reason for exclusion from the evaluable clinical dataset (e.g., 5 at OSU, 4 at LIJ), but it does not specify the total sample size used for the training set for the BC-GP device itself.
- The "Assay Cutoff" section mentions that "a set of bacterial strains were tested to represent all the analytes detected by the BC-GP Test" to initially determine cutoff values, and "A set of retrospective blood culture specimens were then tested with the BC-GP Test to verify the previously determined cut off values." This internal process would constitute a form of training or optimization, but the specific number of samples for this phase is not provided.
- The clinical study mentions the exclusion of 125 specimens, some labeled as "Training Sample" (indicating they were likely used for internal development or training purposes prior to the formal clinical evaluation).
9. How the ground truth for the training set was established
- For the "Assay Cutoff" determination (an internal optimization process analogous to training), the ground truth was established by:
- "Blood culture bottles were spiked with each individual organism and grown in automated blood culture instruments to 'bottle ring.'"
- "The test decisions from each test were compiled to generate a data set that was used to initially determine BC-GP Assay Test cutoff values."
- The verification of these cutoffs used "retrospective blood culture specimens" where "Results were analyzed using logistic fit and ROC statistics. The results obtained from this the cutoff determination matched the results from culture-based biochemical results, thus verifying the final cutoff values."
- This indicates that for the internal optimization/training phases, the ground truth was established by known spiked organisms and confirmed by culture-based biochemical results.
§ 866.3365 Multiplex nucleic acid assay for identification of microorganisms and resistance markers from positive blood cultures.
(a)
Identification. A multiplex nucleic acid assay for identification of microorganisms and resistance markers from positive blood cultures is a qualitative in vitro device intended to simultaneously detect and identify microorganism nucleic acids from blood cultures that test positive by Gram stain or other microbiological stains. The device detects specific nucleic acid sequences for microorganism identification as well as for antimicrobial resistance. This device aids in the diagnosis of bloodstream infections when used in conjunction with other clinical and laboratory findings. However, the device does not replace traditional methods for culture and susceptibility testing.(b)
Classification. Class II (special controls). The special control for this device is FDA's guideline document entitled “Class II Special Controls Guideline: Multiplex Nucleic Acid Assay for Identification of Microorganisms and Resistance Markers from Positive Blood Cultures.” For availability of the guideline document, see § 866.1(e).