(291 days)
The BD MAX Vaginal Panel performed on the BD MAX System is an automated qualitative in vitro diagnostic test for the direct detection of DNA targets from bacteria associated with bacterial vaginosis (qualitative results reported based on detection and quantitation of targeted organism markers), Candida species associated with vulvovaginal candidiasis, and Trichomonas vaginalis from vaginal swabs in patients who are symptomatic for vaginitis/vaginosis. The test utilizes real-time polymerase chain reaction (PCR) for the amplification of specific DNA targets and utilizes fluorogenic target-specific hybridization probes to detect and differentiate DNA from:
- Bacterial vaginosis markers (Individual markers not reported)
- O Lactobacillus spp. (L. crispatus and L. jensenii)
- Gardnerella vaginalis о
- o Atopobium vaginae
- Bacterial Vaginosis Associated Bacteria-2 (BVAB-2) o
- o Megasphaera-1
- Candida spp. (C. albicans, C. tropicalis, C. parapsilosis, C. dubliniensis) ●
- Candida glabrata
- Candida krusei ●
- Trichomonas vaginalis
The BD MAX Vaginal Panel is intended to aid in the diagnosis of vaginal infections in women with a clinical presentation consistent with bacterial vaginosis, vulvovaginal candidiasis and trichomoniasis.
The BD MAX System and the BD MAX Vaginal Panel are comprised of an instrument with associated hardware and accessories, disposable microfluidic cartridges, master mixes, unitized reagent strips, and extraction reagents. The instrument automates sample preparation including target lysis. DNA extraction and concentration, reagent rehydration, target nucleic acid amplification and detection using real-time PCR. The assay includes a Sample Processing Control (SPC) that is present in the Extraction Tube. The SPC monitors DNA extraction steps, thermal cycling steps, reagent integrity and the presence of inhibitory substances. The BD MAX System software automatically interprets test results. For the BD MAX Vaginal Panel, a test result may be called as POS, NEG or UNR (Unresolved) based on the amplification status of the targets and of the Sample Processing Control. IND (Indeterminate) or INC (Incomplete) results are due to BD MAX System failure.
Acceptance Criteria and Study for BD MAX Vaginal Panel
The BD MAX Vaginal Panel is an automated qualitative in vitro diagnostic test for the direct detection of DNA targets from bacteria associated with bacterial vaginosis, Candida species associated with vulvovaginal candidiasis, and Trichomonas vaginalis from vaginal swabs in symptomatic patients. The device's performance was evaluated through a prospective clinical study and various analytical studies.
1. Table of Acceptance Criteria and Reported Device Performance
The acceptance criteria for this device are not explicitly stated as distinct numerical thresholds in the provided document for all metrics. Instead, the document presents detailed performance characteristics from various studies, implying that the observed performance was acceptable for De Novo classification. For the purpose of this response, I will infer "acceptance criteria" where possible from general regulatory expectations for diagnostic devices and the context of the reported results (e.g., successful detection, high agreement). The "Reported Device Performance" will reflect the results from the clinical and analytical studies.
| Performance Characteristic | Acceptance Criteria (Inferred/General Expectation) | Reported Device Performance |
|---|---|---|
| I. Clinical Performance | ||
| Bacterial Vaginosis (BV) | High sensitivity and specificity for both clinician-collected and self-collected samples, with confidence intervals indicating reliable performance. (Typically >80% sensitivity and specificity for nucleic acid-based tests in this context) | Clinician-collected: Sensitivity 90.5% (95% CI: 88.3, 92.2), Specificity 85.8% (95% CI: 83.0, 88.3) |
| Self-collected: Sensitivity 90.7% (95% CI: 88.6, 92.5), Specificity 84.5% (95% CI: 81.6, 87.0) | ||
| Cgroup (Candida spp.) | High sensitivity and specificity for both clinician-collected and self-collected samples. | Clinician-collected: Sensitivity 90.9% (95% CI: 88.1, 93.1), Specificity 94.1% (95% CI: 92.6, 95.4) |
| Self-collected: Sensitivity 92.2% (95% CI: 89.5, 94.2), Specificity 91.9% (95% CI: 90.2, 93.4) | ||
| Candida glabrata | High specificity, and acceptable sensitivity given potentially lower prevalence. Supplemented by contrived specimens. | Clinician-collected: Sensitivity 75.9% (95% CI: 57.9, 87.8), Specificity 99.7% (95% CI: 99.3, 99.9) |
| Self-collected: Sensitivity 86.7% (95% CI: 70.3, 94.7), Specificity 99.6% (95% CI: 99.2, 99.8) | ||
| Contrived specimens: 100% positive agreement for all spiked concentrations (High, Moderate, Low Positive). | ||
| Candida krusei | High specificity, and acceptable sensitivity through contrived specimens, especially given typically very low prevalence in clinical studies. | Clinician-collected: Specificity 99.8% (95% CI: 99.4, 99.9) |
| Self-collected: Specificity 100.0% (95% CI: 99.8, 100.0) | ||
| No positive clinical specimens identified for sensitivity calculation. | ||
| Contrived specimens: 100% positive agreement for all spiked concentrations (High, Moderate, Low Positive). | ||
| Trichomonas vaginalis | High sensitivity and specificity for both clinician-collected and self-collected samples. | Clinician-collected: Sensitivity 93.1% (95% CI: 87.4, 96.3), Specificity 99.3% (95% CI: 98.7, 99.6) |
| Self-collected: Sensitivity 93.2% (95% CI: 87.6, 96.4), Specificity 99.3% (95% CI: 98.6, 99.7) | ||
| Non-reportable Rates | Low rates of unresolved, indeterminate, and incomplete results, with further reduced rates after retesting. (Typically 95% for qualitative agreement for diagnostic assays) | Precision Study (Within-lab): Most categories 100% agreement, with BV high negative at 37.5%. Candida true negative ranges from 99.6-100%. |
| Reproducibility Study (Multi-site, Lot-to-lot): Overall Site-to-Site Reproducibility percent agreement for panel member results ranged from 98.5 % to 100% (true negatives), 99.0% to 100% (low positive samples), and 99.5% to 100% (moderate positive samples). Lot-to-lot results were similar. | ||
| Limit of Detection (LoD) | Consistent and sufficiently low LoD for all targeted organisms, demonstrating adequate detection capability for clinical relevance. (Typically defined as >95% detection rate) | Confirmed LoD values for each organism were established (e.g., C. albicans 17787 CFU/mL, T. vaginalis 22 Cells/mL), with 24 replicates tested at each dilution, showing >95% detection. |
| Analytical Inclusivity | Correct detection of a wide variety of strains for each targeted organism, including phylogenetic and geographic diversity, at clinically relevant concentrations. (Typically >95% detection of tested strains) | 60 of the initial strains correctly identified. Five strains (4 G. vaginalis, 1 L. crispatus) did not meet initial criteria but were detected at slightly higher, yet clinically relevant, concentrations upon repeat testing (G. vaginalis at 90% detection) |
| Analytical Specificity/Cross-reactivity | No significant cross-reactivity with phylogenetically related species or other organisms likely to be present in vaginal specimens. If cross-reactivity is observed, it should be at very high concentrations unlikely to be encountered clinically, or clearly disclosed as a limitation. (Typically shows 95% agreement with expected results) | Met acceptance criteria (>95% agreement) for dry swab up to 2 hours (2-30°C), in capped SBT up to 8 days (2-30°C) or 14 days (2-8°C), and post-vortex up to 4 hours (2-30°C), and post-testing up to 5 hours (2-30°C). |
2. Sample Sizes Used for the Test Set and Data Provenance
Test Set (Clinical Study):
- Total subjects enrolled: 1763
- Compliant subjects: 1740
- Compliant specimens with reportable results:
- Bacterial Vaginosis: 1559 (clinician-collected), 1582 (self-collected)
- Candida: 1618 (clinician-collected), 1628 (self-collected)
- Trichomonas vaginalis: 1600 (clinician-collected), 1610 (self-collected)
- Asymptomatic Women (separate evaluation): 202 women
- Contrived Specimens for C. glabrata and C. krusei: 50 strains each, developed at various concentrations, plus 50 true negative specimens for each organism.
Data Provenance:
- The data for the primary clinical study was prospective.
- Specimen collection occurred at 10 geographically diverse specimen collection sites. Seven sites performed collection only, and three performed both collection and testing with the BD MAX Vaginal Panel. (The country of origin is not explicitly stated but is implicitly the US given FDA submission context).
- Analytical studies (Precision, LoD, Inclusivity, Interference, Stability, Specificity) used simulated vaginal matrix and/or natural negative vaginal matrix, and commercially available organisms/plasmid DNA.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications
The document specifies the reference methods used to establish ground truth but does not explicitly state the number or qualifications of experts involved in the interpretation of these reference methods.
- BV status: Determined using a combination of Nugent Score and Amsel's criteria. These methods typically involve microscopic evaluation by trained laboratory personnel or clinicians, but specific expert qualifications (e.g., years of experience, specific certifications) are not detailed.
- Candida spp. status: Determined by selective (Candida) chromogenic medium and Sabouraud Dextrose Emmons plate cultures. PCR amplification targeting the its2 gene was performed followed by bi-directional sequencing to identify yeast isolates. Interpretation of cultures and sequencing results would be performed by trained microbiologists or laboratory specialists.
- Trichomonas vaginalis status: Determined by a composite of microscopic visualization of motile trichomonads in saline wet mounts of vaginal secretion and by culture. A positive result by either method categorized the patient as positive. Microscopic visualization implies examination by trained personnel.
4. Adjudication Method for the Test Set
The document does not explicitly describe an adjudication method for discordant results between the reference methods themselves or between the device and reference methods.
- For BV, "Specimens with normal flora as per the Nugent Score were considered negative: those positive for BV flora were considered positive while those with intermediate BV flora were segregated into positive or negative categories using Amsel's criteria." This implies a defined algorithm for combining the reference standards rather than a separate expert adjudication panel.
- For Trichomonas vaginalis, "A positive result either by wet mount or by culture was sufficient to categorize the patient as positive." This constitutes a composite reference standard.
- For analyses of discordant results (e.g., for T. vaginalis false negatives and false positives, or C. glabrata false negatives), the document mentions further evaluation with an "FDA-cleared molecular method" or assessing growth levels from chromagar, which indicates further diagnostic investigation rather than expert consensus adjudication of initial reference results.
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
No, a multi-reader multi-case (MRMC) comparative effectiveness study was not conducted. This device is an automated, standalone molecular diagnostic test. It does not involve human readers interpreting images or data to make a diagnosis that would then be compared with and without AI assistance. Therefore, there is no effect size reported for human readers improving with vs. without AI assistance.
6. Standalone Performance Study
Yes, a standalone (i.e., algorithm only without human-in-the-loop performance) was done. The entire premise of the clinical and analytical studies is to evaluate the BD MAX Vaginal Panel's performance on its own, comparing its direct output (qualitative detection of DNA targets) to established reference methods. The system automates sample preparation, DNA extraction, amplification, detection, and interpretation of test results (POS, NEG, or UNR). The clinical performance metrics (sensitivity, specificity, PPV, NPV) presented in Tables 17-36 represent the standalone performance of the device.
7. Type of Ground Truth Used
The ground truth used for the clinical study was a composite reference standard:
- Bacterial Vaginosis: A combination of Nugent Score and Amsel's criteria.
- Candida spp.: Culture (chromogenic medium and Sabouraud Dextrose Emmons plate cultures) followed by PCR amplification targeting the its2 gene and bi-directional sequencing for species identification.
- Trichomonas vaginalis: A composite of microscopic visualization of motile trichomonads in saline wet mounts of vaginal secretion and culture.
This approach combines multiple diagnostic methods to establish the most accurate possible "true" status for each patient's sample.
8. Sample Size for the Training Set
The document does not explicitly state the sample size for a separate "training set" in the context of machine learning or AI model development. The BD MAX Vaginal Panel is described as a nucleic acid-based test utilizing real-time PCR with fluorogenic target-specific hybridization probes, and software for automated interpretation based on amplification status. While there's an "Assay Cut-off" section mentioning use of pre-clinical studies and prospective clinical study data to "validate these cut-offs" and "ROC curve analysis was performed to confirm the optimal cutoffs," this typically refers to refining analytical thresholds based on observed performance from early testing rather than training a complex AI model in the conventional sense.
The "pre-clinical studies" and "LoD confirmation study" using simulated and natural vaginal matrices, as well as the "multi-site prospective clinical study" mentioned for validation of cut-offs, represent data used to establish and confirm the device's operational parameters. However, calling these a "training set" in the context of advanced AI algorithms (like those in machine learning) might be a misinterpretation given the nature of a PCR-based diagnostic with defined analytical cut-offs. The largest dataset mentioned for performance evaluation is the prospective clinical study (1763 enrolled subjects).
9. How the Ground Truth for the Training Set Was Established
As noted above, a distinct "training set" for an AI model (in the sense of supervised learning) is not explicitly described. However, if we interpret "training set" broadly as the data used to initially establish and refine the device's operational parameters and cut-offs, the ground truth was derived from the following:
- Pre-clinical studies: Utilized targeted organisms (or plasmid DNA) spiked into simulated vaginal matrix at varying, known concentrations (e.g., for LoD determination, precision studies). The "expected result" in these analytical studies served as the ground truth.
- Prospective clinical study data: Data from the 1763 enrolled subjects (used to "validate these cut-offs" and performing ROC analysis) employed the composite reference standards described in section 7 (Nugent/Amsel for BV, Culture/Sequencing for Candida, Wet Mount/Culture for T. vaginalis). These reference methods collectively established the ground truth for clinical specimens against which the device's performance, including its cut-offs, was evaluated and confirmed.
§ 866.3975 Device that detects nucleic acid sequences from microorganisms associated with vaginitis and bacterial vaginosis.
(a)
Identification. A device that detects nucleic acid sequences from microorganisms associated with vaginitis and bacterial vaginosis is a qualitative in vitro diagnostic device intended for the detection of microbial nucleic acid sequences in vaginal specimens collected from patients with signs and symptoms of vaginitis or bacterial vaginosis. This device is intended to aid in the diagnosis of vaginitis or bacterial vaginosis when used in conjunction with clinical signs and symptoms and other laboratory findings.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Design verification and validation must include:
(i) Documentation with a detailed device description of device components; ancillary reagents required but not provided; and explanation of the methodology including primer/probe sequence, design, and rationale for sequence selection.
(ii) Documentation with information that demonstrates the performance characteristics of the device, including:
(A) Limit of Detection;
(B) Precision (reproductivity);
(C) Analytical specificity;
(D) Analytical reactivity (inclusivity);
(E) Specimen stability; and
(F) Effects of interfering substances.
(iii) Detailed documentation from a prospective clinical study. As appropriate to the intended use, the prospective clinical study must be performed on an appropriate study population, including women of various ages and ethnicities. The prospective clinical study must compare the device performance to results obtained from well-accepted comparator methods.
(iv) Detailed documentation for device software, including software applications and hardware-based devices that incorporate software.
(2) The labeling required under § 809.10(b) of this chapter must include:
(i) A detailed explanation of the interpretation of results and acceptance criteria;
(ii) For devices with an intended use that includes detection of nucleic acid sequences from bacteria associated with bacterial vaginosis, clinical performance stratified by patient demographics such as race, ethnicity, age, and pregnancy status.
(iii) For devices with an intended use that includes detection of nucleic acid sequences from bacteria associated with bacterial vaginosis, a summary of device results in an asymptomatic population with demographic characteristics appropriate to the intended use population.
(iv) For devices with an intended use that includes detection of either Candida species or bacteria associated with bacterial vaginosis, a limitation that
Candida species and bacterial compositions associated with bacterial vaginosis can be present as part of normal vaginal flora and results should be considered in conjunction with available clinical information.