(82 days)
The Joint Biological Agent Identification and Diagnostic System (JBAIDS) Influenza A Subtyping Kit is intended for the in vitro qualitative detection and differentiation of seasonal Influenza A/H1, seasonal Influenza A/H3, and 2009 H1N1Influenza viral nucleic acids isolated and purified from nasopharyngeal swab (NPS) and nasopharyngeal wash (NPW) specimens from human patients with signs and symptoms of respiratory infection, in conjunction with clinical and epidemiological risk factors. The JBAIDS Influenza A Subtyping Kit contains reverse transcriptase real-time polymerase chain reaction (rRT-PCR) assays for use on the JBAIDS instruments. The Flu A H1. Flu A H3. and Flu A H1 2009 assays of the JBAIDS Influenza A Subtyping Kit target a region of the hemagglutinin (HA) gene of the respective Influenza A virus. The Flu A Sw assay of the JBAIDS Influenza A Subtyping Kit targets a region of the nucleocapsid protein (NP) gene of the 2009 H1N1 Influenza virus, as well as some other Influenza A viruses of swine lineage. This kit is not intended to detect Influenza B or Influenza C viruses.
A negative result for all assays in the JBAIDS Influenza A Subtyping Kit is a presumptive negative result for Influenza A. These results should be confirmed using the JBAIDS Influenza A & B Detection Kit.
Test results are to be used in conjunction with other clinical and epidemiological information. Negative results do not preclude influenza virus infection and should not be used as the sole basis for treatment or other patient management decisions.
Due to low seasonal prevalence, performance characteristics for detection of seasonal Influenza A/H1 were established primarily with retrospective and contrived clinical specimens.
All users, analysts, and any person reporting diagnostic results from use of this device should be trained to perform and interpret the results from this procedure by JBAIDS instructors or designees prior to use. Use of this device is limited to designated Department of Defense (DoD) laboratories equipped with the JBAIDS instruments.
If infection with a novel influenza A virus is suspected based on current clinical and epidemiological screening criteria recommended by public health authorities, specimens should be collected with appropriate infection control precautions for novel virulent influenza viruses and sent to state or local health departments for testing. Viral culture should not be attempted in these cases unless a biosafety laboratory (BSL) 3+ facility is available to receive and culture specimens.
The JBAIDS Influenza A Subtyping Kit is a real time RT-PCR test kit, which, when used with the JBAIDS instrument and software, allows the qualitative in vitro detection and identification of influenza A subtypes H1, H3, and H1 2009 (swine lineage) viral RNA. The assays have been optimized as freeze-dried assays with primer and fluorescent-probe sets for the detection of influenza A/H1, A/H3, and A/2009 H1 viral RNA. In particular, the Flu A H1, Flu A H3, and Flu A H1 2009 assays target distinct regions of the hemagglutinin gene specific to those subtypes, and the Flu A Sw assay targets a region of the nucleocapsid protein gene as a secondary target for the influenza A/2009 H1(swine lineage) virus. The tests are performed using the JBAIDS instrument and software.
Here's an analysis of the acceptance criteria and study details for the JBAIDS Influenza A Subtyping Kit based on the provided text:
Preamble Regarding Acceptance Criteria:
The document provided, a 510(k) Summary, details performance data but does not explicitly state pre-defined "acceptance criteria" in the format of pass/fail thresholds for the clinical study. Instead, it presents the achieved performance metrics (PPA and NPA with 95% Confidence Intervals) from the clinical and analytical studies, implicitly demonstrating that these achieved results were deemed sufficient for substantial equivalence. For the purpose of this response, I will present the reported performance as the "acceptance criteria met," recognizing that the specific numerical targets for acceptance are not explicitly listed in this summary.
1. Table of Acceptance Criteria and Reported Device Performance
As described above, explicit acceptance criteria (i.e., specific numerical targets prior to testing) were not provided in the 510(k) summary. However, the reported performance from the clinical studies served as the basis for the device's clearance. The table below summarizes the reported clinical performance.
| Influenza A Strain | Sample Type | Performance Metric | Reported Performance (PPA/NPA) | 95% Confidence Interval |
|---|---|---|---|---|
| 2009 H1N1 | NPW | PPA | 100.0% (66/66) | 94.6-100% |
| NPW | NPA | 99.3% (414/417) | 97.9-99.9% | |
| NPS | PPA | 100.0% (34/34) | 89.7-100% | |
| NPS | NPA | 99.6% (277/278) | 98.0-100% | |
| Seasonal H3 | NPW | PPA | 100.0% (33/33) | 89.4-100% |
| NPW | NPA | 100.0% (450/450) | 99.2-100% | |
| NPS | PPA | 100.0% (26/26) | 86.8-100% | |
| NPS | NPA | 100.0% (286/286) | 98.7-100% | |
| Seasonal H1 | NPW | PPA | Not Detected (0/0) | - |
| (Clinical Study) | NPW | NPA | 99.8% (482/483) | 98.9-100% |
| NPS | PPA | Not Detected (0/0) | - | |
| NPS | NPA | 100.0% (312/312) | 98.8-100% | |
| Seasonal H1 | NPS | PPA | 100.0% (29/29) | 88.1-100% |
| (Archived Samples) | NPS | NPA | 100.0% (21/21) | 83.4-100% |
| Seasonal H1 | NPW | PPA | 100% (54/54) | 93.4-100% |
| (Surrogate Samples) | NPW | NPA | 100.0% (8/8) | 63.1-100% |
| NPS | PPA | 100% (59/59) | 93.9-100% | |
| NPS | NPA | 100.0% (7/7) | 59.0-100% |
2. Sample Size Used for the Test Set and Data Provenance
-
Clinical Performance (Prospective Study):
- Sample Size: 795 valid specimens (312 NPS, 483 NPW) were analyzed. 6% (44/795) required retesting due to invalid/inconclusive/unsubtypeable initial results.
- Data Provenance: Prospective, collected from 5 geographically separated military clinical sites in the U.S. during the 2010-2011 influenza season (December 2010 to April 2011).
-
Testing of Preselected Archived Samples (for Seasonal Influenza A/H1):
- Sample Size: 51 NPS specimens (30 known positive seasonal Influenza A/H1, 21 influenza-negative).
- Data Provenance: Retrospective, pre-selected archived clinical NPS specimens obtained and confirmed at two different clinical study sites in the U.S.
-
Testing of Surrogate Clinical Specimens (for Seasonal Influenza A/H1):
- Sample Size: 136 individual influenza-negative clinical specimens (68 NPS, 68 NPW) were spiked with known concentrations of seasonal Influenza A/H1 virus. Valid results obtained for 128 (62 NPW, 66 NPS).
- Data Provenance: Contrived clinical samples generated from residual influenza-negative NPS and NPW samples. Sent to two different clinical trial sites in the U.S.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Their Qualifications
The document does not explicitly state the number or qualifications of individual experts who established the ground truth. However, the ground truth for the clinical and archived samples was established using a reference method:
- "The reference method was the CDC rRT-PCT Flu Panel influenza A and influenza B assays, followed by subtype-specific PCR and bi-directional sequencing of amplicons."
- "the presence of Influenza A/H1 viral RNA was confirmed using 'validation' PCR assays. The validation PCR assays were identical to the comparator assays that were used for the prospective clinical evaluation study."
This implies that the "experts" were the personnel performing and interpreting the CDC assays and sequencing results, which would typically be highly trained laboratory professionals and molecular biologists proficient in these techniques, likely adhering to CDC protocols. No specific number of such experts is given.
4. Adjudication Method for the Test Set
The document states: "JBAIDS Software analyzes the fluorescence amplification curves and reports results as positive, negative, or uncertain. A failure of the Positive or Negative Control will result in the entire run being called invalid. Retesting is required to resolve uncertain or invalid results."
For the clinical study, 6% of specimens (44/795) required retesting. Specifically, "Invalid" (32/44), "Inconclusive" (1/44), and "Unsubtypeable" (1/44). "Forty (40) out of 44 samples resolved upon a 1st retest and the remaining 4 samples required a re-extraction and retest and resolved." This indicates a retesting and re-extraction protocol for initial indeterminate or invalid results rather than a multi-expert adjudication panel for interpretive discrepancies. The ground truth itself (CDC rRT-PCR + sequencing) would have its own internal validation/adjudication processes, but this is not detailed for external review.
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
No multi-reader, multi-case (MRMC) comparative effectiveness study comparing human readers with and without AI assistance was performed. This device is a diagnostic kit (real-time RT-PCR assay) that provides automated qualitative results, not an imaging AI diagnostic aid for human interpretation. Therefore, the concept of "human readers improve with AI vs without AI assistance" does not directly apply to this type of device.
6. Standalone Performance Study (Algorithm Only Without Human-in-the-Loop Performance)
Yes, the studies described (clinical performance, archived samples, surrogate samples) represent standalone performance of the algorithm/device. The "JBAIDS Software analyzes the fluorescence amplification curves and reports results as positive, negative, or uncertain." While human operators perform the physical tests, the interpretation of the raw PCR data into a final qualitative result (positive/negative/uncertain) is automated by the device's software (algorithm). The clinical performance data presented (PPA and NPA) are a direct measure of this standalone performance against a defined ground truth.
7. Type of Ground Truth Used
The ground truth used was:
- Expert Consensus/Reference Method: For prospective clinical specimens and archived samples, the reference method was the CDC rRT-PCR Flu Panel influenza A and influenza B assays, followed by subtype-specific PCR and bi-directional sequencing of amplicons. This is a highly robust and accepted laboratory method for viral identification and subtyping.
- Known Spiked Concentration: For surrogate clinical specimens, the ground truth for positive samples was established by spiking influenza-negative clinical samples with known concentrations of seasonal Influenza A/H1 virus. Un-spiked negative samples also served as ground truth negatives.
8. Sample Size for the Training Set
The document does not provide details on a specific "training set" sample size or data for the development of the device's algorithm, as would typically be described for machine learning or AI models. This device is a PCR-based assay, where the "algorithm" is primarily the pre-defined thresholds and analysis logic within the JBAIDS software to interpret PCR amplification curves. Performance characteristics are primarily established through analytical validation (LoD, inclusivity, exclusivity) and clinical validation with independent test sets, rather than an explicit training-validation split in the context of an adaptive or learning algorithm. The design of the primers and probes, and the associated software interpretation, would have been iteratively developed and optimized, but a formal "training set" as understood in AI/ML is not mentioned.
9. How the Ground Truth for the Training Set Was Established
As noted above, a formal "training set" for an AI/ML algorithm is not described for this diagnostic kit. The underlying principles of the PCR assay (primer/probe design, reaction conditions) are established through extensive analytical studies to ensure specificity and sensitivity. The "ground truth" for developing the analytical performance characteristics (like LoD, exclusivity, inclusivity) would involve testing well-characterized viral strains and non-target organisms with known concentrations, typically obtained from reference collections. This process establishes the analytical sensitivity and specificity that the device is designed to achieve when interpreting a sample.
§ 866.3332 Reagents for detection of specific novel influenza A viruses.
(a)
Identification. Reagents for detection of specific novel influenza A viruses are devices that are intended for use in a nucleic acid amplification test to directly detect specific virus RNA in human respiratory specimens or viral cultures. Detection of specific virus RNA aids in the diagnosis of influenza caused by specific novel influenza A viruses in patients with clinical risk of infection with these viruses, and also aids in the presumptive laboratory identification of specific novel influenza A viruses to provide epidemiological information on influenza. These reagents include primers, probes, and specific influenza A virus controls.(b)
Classification. Class II (special controls). The special controls are:(1) FDA's guidance document entitled “Class II Special Controls Guidance Document: Reagents for Detection of Specific Novel Influenza A Viruses.” See § 866.1(e) for information on obtaining this document.
(2) The distribution of these devices is limited to laboratories with experienced personnel who have training in standardized molecular testing procedures and expertise in viral diagnosis, and appropriate biosafety equipment and containment.