(88 days)
cobas® BKV is an in vitro nucleic acid amplification test for the quantitation of BKV) DNA in human EDTA plasma and urine stabilized in cobas® PCR media on the cobas® 6800/8800 Systems.
In EDTA plasma, cobas® BKV is intended for use as an aid in the management of BKV in transplant patients. In patients undergoing monitoring of BKV in EDTA plasma, serial DNA measurements can be used to indicate the need for potential treatment changes and to assess viral response to treatment.
In urine stabilized in cobas® PCR Media, cobas® BKV is intended for use as an aid in the management of BKV in transplant patients.
The results from cobas® BKV are intended to be read and analyzed by a qualified licensed healthcare professional in conjunction with clinical signs and symptoms and relevant laboratory findings. Test results must not be the sole basis for patient management decisions .
cobas® BKV is not intended for use as a screening test for blood products or human cells, tissues, and cellular and tissue-based products (HCT/Ps).
cobas® BKV (Figure 1) is based on fully automated sample preparation (nucleic acid extraction and purification) followed by PCR amplification and detection. The cobas® 6800/8800 Systems consist of the sample supply module, the transfer module, the processing module, and the analytic module. Automated data management is performed by the cobas® 6800/8800 software which assigns test results for all tests as either target not detected, BKV DNA detected ULoQ (upper limit of quantitation), or a value in the linear range LLoQ
This document describes the acceptance criteria and supporting studies for the cobas® BKV device for the quantitation of BK virus (BKV) DNA in human urine. The information is extracted from a 510(k) summary.
1. Table of Acceptance Criteria and Reported Device Performance
| Acceptance Criteria | Reported Device Performance (cobas® BKV in Urine) |
|---|---|
| Limit of Detection (LoD): 95% hit rate for BKV DNA. | 12.2 IU/mL (WHO International Standard, 95% confidence range: 9.2-18.3 IU/mL). Achieved ≥95% hit rate for subgroups Ia, Ic, and subtypes II, III, IV at 12.2 IU/mL. |
| Linear Range: Accuracy within ± 0.2 log10. | 7.41E+01 IU/mL to 7.41E+08 IU/mL. Maximum deviation of linear regression from better fitting non-linear regression was ≤ ± 0.2 log10 for all tested genotypes within the linear range. |
| Lower Limit of Quantitation (LLoQ): Mean deviation between observed and assigned log10 titer ≤ ± 0.3 log10 (based on upper 95% CI of worst performing lot). Total Analytical Error (TAE) ≤ 1 log10. | 200 IU/mL. Mean deviation between observed and assigned log10 titer was ≤ 0.3 log10. TAE was ≤ 0.44 for all lots and concentrations (Table 7). |
| Precision (Within-Laboratory): High precision across the concentration range. | Demonstrated high precision across a concentration range of 7.41E+02 IU/mL to 7.41E+05 IU/mL (Table 8, Table 9). Total %CV ranged from 7% (highest concentration) to 23% (lowest concentration). |
| Analytical Specificity: No interference from listed microorganisms; mean log10 titer of positive BKV samples with interfering organisms within ± 0.5 log10 of spike control. | None of the tested non-BKV pathogens (bacteria, yeast, viruses in Table 10) interfered. Mean log10 titer of positive BKV samples was within ± 0.5 log10 of the spike control. |
| Interfering Substances: No interference from listed endogenous substances and drug compounds, with mean log10 titer of positive BKV samples with interfering substances within ± 0.5 log10 of spike control. | All listed endogenous interferences and drug compounds (except talcum powder at >0.05%) did not interfere. Mean log10 titer of positive BKV samples was within ± 0.5 log10 of the spike control. |
| Cross Contamination: Zero cross-contamination rate with a low upper 95% confidence interval. | 0.0% (upper one-sided 95% CI 1.24%) with 240 replicates of negative samples. |
| Clinical Reproducibility: Acceptable reproducibility; 100% detection of 3 x LLoQ samples; 95% CI for difference between 2 measurements within ± 0.20 log10 copies/mL. | Acceptable clinical reproducibility. 100% of 3 x LLoQ samples detected. All estimated 95% CLs for the difference between 2 measurements from the same subject were within ± 0.20 log10 copies/mL. |
| Negative Percent Agreement (NPA) (Clinical): High negative percent agreement. | 100% (95% Exact CI of 94.1% to 100%) for 61 valid negative samples. |
| Clinical Concordance with LDT: High agreement at various thresholds and strong correlation. | Concordance analysis with comparator LDT showed high agreement (e.g., 93.9% at Target Not Detected threshold, 99.5% at LLoQ threshold). Deming linear regression showed a strong correlation with CI for intercept within ±0.5 log10 IU/mL. |
2. Sample Sizes Used for the Test Set and Data Provenance
The studies focused on analytical performance (Limit of Detection, Linearity, Precision, Specificity, Interference, Cross-contamination) and clinical performance (Reproducibility, Clinical Concordance).
- Limit of Detection (LoD):
- WHO International Standard: 63 replicates per concentration level (total 7 levels, x3 lots = 1323 replicates).
- Subgroups/Subtypes Verification: 63 replicates per concentration level for each genotype (total 5 genotypes, 3 levels each, x3 lots = 2835 replicates).
- Linear Range:
- Main linearity: 36 replicates per panel member (10 panel members, x3 lots = 1080 replicates).
- Genotype linearity: 12 replicates per level for each genotype (8 panel members each, 5 genotypes, x3 lots = 1440 replicates).
- Lower Limit of Quantitation (LLoQ): Data from the Linearity study at 100, 200, and 300 IU/mL concentrations.
- Precision (Within-Laboratory): 72 replicates for each of 5 dilution levels (x3 lots = 1080 replicates).
- Analytical Specificity: 3 replicates for each of the microorganisms listed in Table 10, both in BKV-negative and BKV-positive urine (number of microorganisms not explicitly totalled, but substantial).
- Interfering Substances: Replicates for each substance in presence and absence of BKV DNA (number of replicates not explicitly stated, but implies multiple for each substance in Table 11).
- Cross Contamination: 240 replicates of BKV-negative matrix samples, 225 replicates of high titer BKV DNA urine samples.
- Clinical Reproducibility: 270 tests per concentration (5 concentrations, total 1350 tests, not including controls).
- Clinical Performance / Concordance: 308 neat urine samples from 84 transplant subjects (for concordance analysis). 61 negative samples for NPA. 153 BKV positive urine samples from 55 transplant subjects (for correlation analysis).
Data Provenance: The document implies that the non-clinical studies were conducted internally or at authorized labs. The clinical performance evaluation was conducted at 3 testing sites, suggesting multi-site prospective data collection. The data samples were human EDTA plasma and urine. The origin of the samples (country) is not explicitly stated in the provided text. The clinical study used a retrospective cohort of samples from transplant patients.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications of Those Experts
The ground truth for the analytical studies was established based on known concentrations of BKV international standards or armored DNA. For clinical performance, the comparator was a "well-established laboratory developed nucleic acid test (LDT)".
The document does not mention the use of experts to establish ground truth for the test sets in the typical sense of human readers for image-based diagnostics. The "ground truth" for this diagnostic device study is based on the highly controlled properties of the spiked samples (known concentrations, genotypes) for analytical performance, and the results from a comparator LDT for clinical concordance.
4. Adjudication Method for the Test Set
Not applicable in the context of this in vitro diagnostic device, as the "ground truth" is based on quantitative measurements and known concentrations, not subjective expert assessment requiring adjudication.
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance
Not applicable. This device is an in vitro nucleic acid amplification test (NAAT) for quantitative measurement of BKV DNA. It is not an AI-assisted diagnostic device requiring human reader input or interpretation in the way an imaging diagnostic device would. Therefore, an MRMC comparative effectiveness study involving human readers and AI assistance is not relevant to this submission.
6. If a Standalone (i.e. algorithm only without human-in-the-loop performance) was done
Yes, the studies described are for the standalone performance of the cobas® BKV system, which is an automated molecular diagnostic assay. The system performs "fully automated sample preparation (nucleic acid extraction and purification) followed by PCR amplification and detection." The results are "assigned... by the cobas® 6800/8800 software." While "results are intended to be read and analyzed by a qualified licensed healthcare professional in conjunction with clinical signs and symptoms and relevant laboratory findings," the primary performance metrics are based on the direct output of the automated system.
7. The Type of Ground Truth Used (expert consensus, pathology, outcomes data, etc.)
- Analytical Studies: The ground truth for analytical studies (LoD, Linearity, Precision, Specificity, Interference) was established using known concentrations of BKV DNA, including the WHO International Standard (NIBSC 14/212), BKV armored DNA, and clinical specimens diluted to specific concentrations. Samples were spiked into BKV-negative urine.
- Clinical Studies: For clinical concordance, the ground truth was based on the results from a "well-established laboratory developed nucleic acid test (LDT) (comparator BKV LDT)" on actual clinical urine samples. DNA sequencing was also used in some cases to confirm BKV presence in discordant results.
8. The Sample Size for the Training Set
The document describes performance evaluation studies (validation and verification) rather than a machine learning model development process that typically involves distinct training and test sets.
Therefore, a separate "training set" sample size for a machine learning algorithm is not applicable in the context of this in vitro diagnostic device, which is based on established molecular biological techniques (PCR).
9. How the Ground Truth for the Training Set Was Established
As this is not an AI/ML-based device with a "training set," this question is not applicable. The ground truth for the evaluation of the device was established through known concentrations of viral standards and comparison to a comparator LDT, as described in point 7.
§ 866.3183 Quantitative viral nucleic acid test for transplant patient management.
(a)
Identification. A quantitative viral nucleic acid test for transplant patient management is identified as a device intended for prescription use in the detection of viral pathogens by measurement of viral DNA or RNA using specified specimen processing, amplification, and detection instrumentation. The test is intended for use as an aid in the management of transplant patients with active viral infection or at risk for developing viral infections. The test results are intended to be interpreted by qualified healthcare professionals in conjunction with other relevant clinical and laboratory findings.(b)
Classification. Class II (special controls). The special controls for this device are:(1) The labeling required under § 809.10(b) of this chapter must include:
(i) A prominent statement that the device is not intended for use as a donor screening test for the presence of viral nucleic acid in blood or blood products.
(ii) Limitations which must be updated to reflect current clinical practice. These limitations must include, but are not limited to, statements that indicate:
(A) Test results are to be interpreted by qualified licensed healthcare professionals in conjunction with clinical signs and symptoms and other relevant laboratory results; and
(B) Negative test results do not preclude viral infection or tissue invasive viral disease and that test results must not be the sole basis for patient management decisions.
(iii) A detailed explanation of the interpretation of results and acceptance criteria must be provided and include specific warnings regarding the potential for variability in viral load measurement when samples are measured by different devices. Warnings must include the following statement, where applicable: “Due to the potential for variability in [analyte] measurements across different [analyte] assays, it is recommended that the same device be used for the quantitation of [analyte] when managing individual patients.”
(iv) A detailed explanation of the principles of operation and procedures for assay performance.
(2) Design verification and validation must include the following:
(i) Detailed documentation of the device description, including all parts that make up the device, ancillary reagents required for use with the assay but not provided, an explanation of the methodology, design of the primer/probe sequences, rationale for the selected gene target, and specifications for amplicon size, guanine-cytosine content, and degree of nucleic acid sequence conservation. The design and nature of all primary, secondary and tertiary quantitation standards used for calibration must also be described.
(ii) A detailed description of the impact of any software, including software applications and hardware-based devices that incorporate software, on the device's functions;
(iii) Documentation and characterization (
e.g., determination of the identity, supplier, purity, and stability) of all critical reagents and protocols for maintaining product integrity throughout its labeled shelf-life.(iv) Stability data for reagents provided with the device and indicated specimen types, in addition to the basis for the stability acceptance criteria at all time points chosen across the spectrum of the device's indicated life cycle, which must include a time point at the end of shelf life.
(v) All stability protocols, including acceptance criteria.
(vi) Final lot release criteria along with documentation of an appropriate justification that lots released at the extremes of the specifications will meet the claimed analytical and clinical performance characteristics as well as the stability claims.
(vii) Risk analysis and documentation demonstrating how risk control measures are implemented to address device system hazards, such as Failure Mode Effects Analysis and/or Hazard Analysis. This documentation must include a detailed description of a protocol (including all procedures and methods) for the continuous monitoring, identification, and handling of genetic mutations and/or novel viral stains (
e.g., regular review of published literature and annual in silico analysis of target sequences to detect possible primer or probe mismatches). All results of this protocol, including any findings, must be documented.(viii) Analytical performance testing that includes:
(A) Detailed documentation of the following analytical performance studies: limit of detection, upper and lower limits of quantitation, inclusivity, precision, reproducibility, interference, cross reactivity, carry-over, quality control, specimen stability studies, and additional studies as applicable to specimen type and intended use for the device;
(B) Identification of the viral strains selected for use in analytical studies, which must be representative of clinically relevant circulating strains;
(C) Inclusivity study results obtained with a variety of viral genotypes as applicable to the specific assay target and supplemented by in silico analysis;
(D) Reproducibility studies that include the testing of three independent production lots;
(E) Documentation of calibration to a reference standard that FDA has determined is appropriate for the quantification of viral DNA or RNA (
e.g., a recognized consensus standard); and(F) Documentation of traceability performed each time a new lot of the standardized reference material to which the device is traceable is released, or when the field transitions to a new standardized reference material.
(ix) Clinical performance testing that includes:
(A) Detailed documentation from either a method comparison study with a comparator that FDA has determined is appropriate, or results from a prospective clinical study demonstrating clinical validity of the device;
(B) Data from patient samples, with an acceptable number of the virus-positive samples containing an analyte concentration near the lower limit of quantitation and any clinically relevant decision points. If an acceptable number of virus-positive samples containing an analyte concentration near the lower limit of quantitation and any clinically relevant decision cannot be obtained, contrived samples may be used to supplement sample numbers when appropriate, as determined by FDA;
(C) The method comparison study must include predefined maximum acceptable differences between the test and comparator method across all primary outcome measures in the clinical study protocol; and
(D) The final release test results for each lot used in the clinical study.