(386 days)
The Visby Medical Sexual Health Test is a single-use (disposable), fully-integrated, automated Polymerase Chain Reaction (PCR) in vitro diagnostic test intended for use in point-of-care or clinical laboratory settings for the rapid detection and differentiation of DNA from Chlamydia trachomatis, Neisseria gonorrhoeae, and Trichomonas vaginalis in self-collected female vaginal swab specimens using the Visby Medical Sexual Health Vaginal Specimen Collection Kit in a health care setting. The test results are to aid in the diagnosis of symptomatic infections with Chlamydia trachomatis, Neisseria gonorrhoeae, and Trichomonas vaginalis.
The Visby Medical Sexual Health Test (Visby Test) is a single-use (disposable), fully integrated, rapid, compact device containing a PCR-based assay for direct qualitative detection and differentiation of DNA from CT, NG, and TV. The test system includes the Visby Medical Sexual Health device, the Visby Medical power supply, the Visby Medical Vaginal Specimen Collection kit, and fixed-volume transfer pipettes. The device processes self-collected vaginal swab samples by automatically performing all steps required to complete lysis, polymerase chain reaction, and amplicon detection.
The patient uses the Visby Medical Vaginal Specimen Collection Kit to self-collect a vaginal specimen with the provided flocked swab, and then the patient elutes the specimen into the Visby Medical Collection Media. The test operator transfers the collection media containing the patient specimen into the sample port of the device using the provided fixed-volume pipette where it rehydrates a lyophilized internal process control. The sample enters a lysis module, where the DNA of the sample and the internal process control are extracted using a combination of chemical lysis and high temperature. The extracted DNA enters a mixing chamber where it rehydrates lyophilized PCR reagents, followed by thermocycling to amplify target DNA. If present, the amplified pathogen target (CT, NG, and/or TV) and internal process control hybridize to specific probes located on a flow channel. Detection of the target-specific PCR product is accomplished via an enzyme-linked colorimetric assay using streptavidin bound horseradish peroxidase (HRP) and a colorimetric substrate that forms a purple precipitate. Test results can be expected in approximately 30 minutes: a green check mark will appear, and a purple color will appear in the "Results Valid" spot, indicating a valid test. A purple spot adjacent to "Chlamydia," "Gonorrhea," and/or "Trichomoniasis" signifies the presence of amplified CT, NG, and/or TV DNA in the sample.
Vaginal self-collection kits containing a tube of collection media compatible with the Sexual Health test and a collection swab are also packaged separately. External controls recommended in the product labeling are commercially available from a different manufacturer.
Here's a breakdown of the acceptance criteria and study information based on the provided document:
1. Table of Acceptance Criteria and Reported Device Performance
The document does not explicitly state formal acceptance criteria in terms of specific PPA/NPA thresholds for substantial equivalence to the predicate device. Instead, the studies presented aim to demonstrate "comparability" or "equivalence" to the predicate device (Visby Medical Sexual Health Click Test, K200748/CW200003). Therefore, the reported performance is compared directly against the predicate where applicable.
Based on the "Clinical Performance of the Visby Test compared to the Click test performed by untrained operators" (Table 3) and "Performance of Visby Test Compared to Click Test - Clinical Specimens" (Table 4), the reported device performance is:
| Target | Performance Metric | Untrained Operators (Table 3) | Trained Operators (Table 4) |
|---|---|---|---|
| Chlamydia trachomatis (CT) | PPA (95% CI) | 100.0% (88.6%-100.0%) | 100.0% (96.8% - 100.0%) |
| NPA (95% CI) | 100% (94.9%-100.0%) | 98.7% (96.3% - 99.6%) | |
| Neisseria gonorrhoeae (NG) | PPA (95% CI) | 100.0% (83.9%-100.0%) | 100.0% (89.9% - 100.0%) |
| NPA (95% CI) | 100.0% (95.5%-100.0%) | 100.0% (98.8% - 100.0%) | |
| Trichomonas vaginalis (TV) | PPA (95% CI) | 100.0% (88.6%-100.0%) | 93.5% (87.1% - 96.8%) |
| NPA (95% CI) | 95.8% (88.3%-98.6%) | 98.4% (95.9% - 99.4%) |
Note: The acceptance criteria for the LoD comparison was explicitly stated: "The LoD values were determined to be equivalent if the lowest concentrations of organism with at least 19/20 detection rate from the two devices were within 3-fold of each other." As seen in Table 1, this criterion was met for all organisms.
2. Sample Size Used for the Test Set and Data Provenance
- LoD Comparison: Not explicitly stated as a "test set" in the traditional sense for PPA/NPA. For each LoD multiple, 20 Visby devices and 20 Click devices were tested. The data provenance appears to be laboratory-generated (spiked samples in negative pooled clinical vaginal sample).
- Reproducibility Study: Seven (7) panel members (negative, low positive, moderate positive for CT, NG, TV) were used. Each panel member was tested by 6 operators (2 per site) three times each day over 6 non-consecutive days, across 3 reagent lots. This results in $7 \times 6 \times 3 \times 6 = 756$ individual test results reported (though the table summarizes 108 tests per panel member for overall agreement, implying 108 replicates per panel across sites/operators/days). Data provenance is likely laboratory-contrived samples (cultured organisms in negative pooled clinical vaginal swab matrix).
- Multicenter Study with Untrained Operators:
- Sample Size: 102 samples initially selected; 1 was excluded, so 101 samples were included in the final analysis (30 CT positive, 20 NG positive, 30 TV positive, 33 negative based on comparator results, though the sum of these is 113, implying some overlap or specific positivity counts within the 101 total).
- Data Provenance: De-identified, frozen, self-collected vaginal swab specimens that are a subset of patient samples previously characterized in the clinical study for the Click Test. This indicates retrospective human clinical data from the U.S. (implied by FDA submission context, though not explicitly stated for this particular subset).
- Single Center Study with Trained Operators:
- Sample Size: 359 specimens initially; 1 excluded due to insufficient volume, 7 excluded due to retest issues. Final analysis included 351 specimens.
- Data Provenance: De-identified, frozen, archived self-collected vaginal swab specimens in Visby Medical Collection Media. These included specimens from the original clinical study of the Click Test and from two other previous Visby Medical studies. This indicates retrospective human clinical data, likely from the U.S.
- Contrived Specimens Spiked at Low Organism Concentrations:
- Sample Size: 80 contrived samples (20 CT positive, 20 NG positive, 20 TV positive, and 20 negative).
- Data Provenance: Laboratory-generated, spiked samples using individual clinical matrices.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Their Qualifications
The document mentions that for the Multicenter Study and Single Center Study:
- The samples were "previously characterized in the clinical study for the Click Test" and "based on comparator results from the clinic study for Click Test."
- "TV PIS (based on the original clinical study for Click Test) for all seven specimens were negative."
This strongly implies that the ground truth for these clinical test sets was established during the clinical studies for the predicate device (Click Test). The report for the current device (Visby Test) does not detail the specific number or qualifications of experts for establishing the ground truth for these retrospective samples, as that would have been documented in the original predicate device's submission. It refers to "comparator results," which typically means a highly sensitive and specific reference method (e.g., PCR with bidirectional sequencing, culture followed by definitive identification) performed and interpreted by qualified laboratory personnel, rather than a panel of clinical experts adjudicating cases for complex imaging or clinical diagnoses.
For the LoD and Contrived samples studies, the "ground truth" is known by design, as the samples are deliberately spiked with known concentrations of organisms.
4. Adjudication Method for the Test Set
The document does not describe a formal adjudication method (like 2+1 or 3+1) for establishing the ground truth in the context of clinical samples.
- For the multicenter and single-center clinical studies, the reference standard (ground truth) was based on "comparator results" from the original clinical studies of the predicate device. This typically implies a highly accurate lab-based method.
- For the reproducibility, LoD, and contrived sample studies, the ground truth was known by design (known presence/absence and concentration of spiked organisms).
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance
This device is an in vitro diagnostic (IVD) test that directly detects nucleic acids via PCR and provides a visually interpreted result. It is not an AI-based imaging analysis tool or a system that "assists human readers" in the way an MRMC study would typically evaluate for AI in radiology or pathology. The "operators" are interpreting the device's output (presence of a purple spot), not interpreting complex data that AI might enhance. Therefore, an MRMC comparative effectiveness study for human readers with and without AI assistance is not applicable and was not done.
The studies performed evaluate the performance of the device itself when operated by different user groups (trained vs. untrained) and compared to a predicate device.
6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) was done
The Visby Medical Sexual Health Test is a fully integrated, automated PCR diagnostic test that provides a visual result (purple spot). The results are then visually interpreted by an operator ("Operator visually interprets test results" - pg 6). While the internal PCR process is automated, the final "reading" is done by a human looking at the colorimetric output. Therefore, it's not a purely "algorithm-only" device without human intervention for result interpretation. However, the performance data presented (PPA/NPA) reflects the device's accuracy in producing the correct visual output, as subsequently interpreted by a human operator. The studies demonstrate the performance of the device system including human interpretation in different use environments.
7. The Type of Ground Truth Used
- Clinical Comparison Studies (Multicenter and Single Center): The ground truth was established by "comparator results" from the original clinical studies of the predicate device (Visby Medical Sexual Health Click Test). This typically refers to a highly accurate laboratory-based reference method, such as a laboratory-developed test (LDT) using PCR with sequencing confirmation, or culture followed by definitive identification, considered the gold standard for detecting these pathogens.
- LoD, Reproducibility, and Contrived Samples Studies: The ground truth was established by design, using cultured organisms spiked at known concentrations into negative clinical matrices.
8. The Sample Size for the Training Set
The document describes performance data for the Visby Medical Sexual Health Test, which is an "updated version of the Visby Medical Sexual Health Click test (Click Test, the predicate)." It explicitly states that "The Visby Test has no changes to the PCR primer and probe sequences, reagent formulations, detection method, or result interpretations."
Since this is an updated version of an existing device with no changes to the core diagnostic method (PCR primers, probes, detection), it's highly unlikely that a separate "training set" in the context of machine learning model development was used for this specific device. The document focuses on demonstrating equivalence to the predicate device. If the predicate device (Click Test) involved any form of algorithm development requiring a training set, that would have been part of its original 510(k) submission (K200748/CW200003). For the current submission, the studies are primarily verification and validation, showing that the new manufacturing process and simplified user interface do not negatively impact performance compared to the predicate.
Therefore, a sample size for a training set for the Visby Medical Sexual Health Test is not applicable / not provided in this document.
9. How the Ground Truth for the Training Set Was Established
As mentioned in point 8, a "training set" for an algorithm is not described or applicable to the type of update this device represents. If the predicate device (Click Test) had a training set, the ground truth for that would have been established according to its original clearance. This document focuses on demonstrating equivalence of the updated physical device.
§ 866.3393 Device to detect nucleic acids from non-viral microorganism(s) causing sexually transmitted infections and associated resistance marker(s).
(a)
Identification. A device to detect nucleic acids from non-viral microorganism(s) causing sexually transmitted infections and associated resistance marker(s) is an in vitro diagnostic device intended for the detection and identification of nucleic acids from non-viral microorganism(s) and their associated resistance markers in clinical specimens collected from patients suspected of sexually transmitted infections. The device is intended to aid in the diagnosis of non-viral sexually transmitted infections in conjunction with other clinical and laboratory data. These devices do not provide confirmation of antibiotic susceptibility since mechanisms of resistance may exist that are not detected by the device.(b)
Classification. Class II (special controls). The special controls for this device are:(1) The intended use for the labeling required under § 809.10 of this chapter must include a detailed description of targets the device detects, the results provided to the user, the clinical indications appropriate for test use, and the specific population(s) for which the device is intended.
(2) Any sample collection device used must be FDA-cleared, -approved, or -classified as 510(k) exempt (standalone or as part of a test system) for the collection of specimen types claimed by this device; alternatively, the sample collection device must be cleared in a premarket submission as a part of this device.
(3) The labeling required under § 809.10(b) of this chapter must include:
(i) A detailed device description, including reagents, instruments, ancillary materials, all control elements, and a detailed explanation of the methodology, including all pre-analytical methods for processing of specimens;
(ii) Detailed discussion of the performance characteristics of the device for all claimed specimen types based on analytical studies, including Limit of Detection, inclusivity, cross-reactivity, interfering substances, competitive inhibition, carryover/cross contamination, specimen stability, within lab precision, and reproducibility, as appropriate;
(iii) Detailed descriptions of the test procedure, the interpretation of test results for clinical specimens, and acceptance criteria for any quality control testing;
(iv) Limiting statements indicating that:
(A) A negative test result does not preclude the possibility of infection;
(B) The test results should be interpreted in conjunction with other clinical and laboratory data available to the clinician;
(C) Reliable results are dependent on adequate specimen collection, transport, storage, and processing. Failure to observe proper procedures in any one of these steps can lead to incorrect results; and
(D) If appropriate (
e.g., recommended by the Centers for Disease Control and Prevention, by current well-accepted clinical guidelines, or by published peer reviewed research), that the clinical performance is inferior in a specific clinical subpopulation or for a specific claimed specimen type; and(v) If the device is intended to detect antimicrobial resistance markers, limiting statements, as appropriate, indicating that:
(A) Negative results for claimed resistance markers do not indicate susceptibility of detected microorganisms, as resistance markers not measured by the assay or other potential mechanisms of antibiotic resistance may be present;
(B) Detection of resistance markers cannot be definitively linked to specific microorganisms and the source of a detected resistance marker may be an organism not detected by the assay, including colonizing flora;
(C) Detection of antibiotic resistance markers may not correlate with phenotypic gene expression; and
(D) Therapeutic failure or success cannot be determined based on the assay results, since nucleic acid may persist following appropriate antimicrobial therapy.
(4) Design verification and validation must include:
(i) Detailed device description documentation, including methodology from obtaining sample to result, design of primer/probe sequences, rationale for target sequence selection, and computational path from collected raw data to reported result (
e.g., how collected raw signals are converted into a reported result).(ii) Detailed documentation of analytical studies, including, Limit of Detection, inclusivity, cross-reactivity, microbial interference, interfering substances, competitive inhibition, carryover/cross contamination, specimen stability, within lab precision, and reproducibility, as appropriate.
(iii) Detailed documentation and performance results from a clinical study that includes prospective (sequential) samples for each claimed specimen type and, when determined to be appropriate by FDA, additional characterized clinical samples. The study must be performed on a study population consistent with the intended use population and compare the device performance to results obtained from FDA accepted comparator methods. Documentation from the clinical studies must include the clinical study protocol (including a predefined statistical analysis plan) study report, testing results, and results of all statistical analyses.
(iv) A detailed description of the impact of any software, including software applications and hardware-based devices that incorporate software, on the device's functions.