(144 days)
The Acucy™ Influenza A&B Test for the rapid qualitative detection of influenza A and B is composed of a rapid chromatographic immunoassay for the direct and qualitative detection of influenza A and B viral nucleoprotein antigens from nasal and nasopharyngeal swabs of symptomatic patients that is automatically analyzed on the Acucy Reader. The Acucy Influenza A&B Test is a differentiated test, such that influenza A viral antigens can be distinguished from influenza B viral antigens from a single processed sample using a single Test Cassette. The test is intended for use with the Acucy System as an aid in the diagnosis of influenza A and B viral infections. The test is not intended for the detection of influenza C viruses. Negative test results are presumptive and should be confirmed by viral culture or an FDA-cleared influenza A and B molecular assay. Negative test results do not preclude influenza viral infection and should not be used as the sole basis for treatment or other patient management decisions.
The Acucy™ Influenza A&B Test is a lateral flow immunochromatographic assay in the sandwich immunoassay format. The Acucy Influenza A&B Test consists of a Test Cassette that detects and differentiates influenza A and influenza B viral antigens from a patient sample. The test sample, a nasal swab or nasopharyngeal swab, is processed to extract nucleoproteins by mixing the swab in Acucy Influenza A&B Extraction Buffer. The mixture is then added to the sample well of the Test Cassette. From there, the sample migrates along the membrane surface. If influenza A or B viral antigens are present, they form a complex with mouse monoclonal antibodies to influenza A and/or B nucleoproteins conjugated to colloidal gold. The complex is then bound by a rat anti-influenza A and/or mouse anti-influenza B antibody coated on the nitrocellulose membrane. The Acucy Reader is an optoelectronic instrument that uses a reflectance-based measurement method to evaluate the line signal intensities in the results window of the Test Cassette. The Reader scans the Test Cassette and measures the absorbance intensity by processing the results using method-specific algorithms. The Acucy Reader displays the test results POS (+), NEG (-), or INVALID on the screen. The results can also be automatically printed on the Acucy Printer if this option is selected.
The document describes the performance of the Acucy™ Influenza A&B Test with the Acucy™ System, which is a rapid in-vitro diagnostic test. Here's a breakdown of the acceptance criteria and the study that proves the device meets them:
1. Table of Acceptance Criteria and Reported Device Performance
The document doesn't explicitly list "acceptance criteria" for sensitivity and specificity in a separate table. Instead, it presents the achieved performance, which implicitly serves as the successful outcome of the clinical study. The performance is compared against a Composite Reference.
Here's a table summarizing the reported device performance for the combined nasal and nasopharyngeal swab samples from the clinical study, which are the primary results for overall performance:
| Metric | Influenza A Performance | Influenza B Performance |
|---|---|---|
| Sensitivity | 96.4% (95% CI: 93.1% - 98.2%) | 82.3% (95% CI: 75.6% - 87.4%) |
| Specificity | 96.0% (95% CI: 94.4% - 97.2%) | 98.1% (95% CI: 96.9% - 98.8%) |
Other "Acceptance Criteria" implicitly met through other studies:
| Study Category | Acceptance Criteria (Implicit from Results) | Reported Device Performance |
|---|---|---|
| Reproducibility | High agreement across sites, operators, and days for various concentrations. | Overall Percent Agreement for Influenza A samples (High Negative, Low Positive, Moderate Positive, True Negative) ranged from 98.9% to 100%. Overall Percent Agreement for Influenza B samples (High Negative, Low Positive, Moderate Positive, True Negative) ranged from 98.9% to 100%. |
| Limit of Detection (LoD) | Consistently positive results >95% of the time at specified concentrations. | LoD for Influenza A (H1N1pdm09): 1.41 x 10^1 TCID50/mL. LoD for Influenza A (H3N2): 7.06 x 10^1 TCID50/mL. LoD for Influenza B (Victoria): 2.35 x 10^1 TCID50/mL. LoD for Influenza B (Yamagata): 3.40 x 10^1 TCID50/mL. |
| Analytical Reactivity | All tested influenza A strains yield positive A and negative B results; all influenza B strains yield positive B and negative A results. | All 28 tested influenza A and B strains at or near LoD met this criterion. |
| Analytical Specificity / Cross-Reactivity | No cross-reactivity with tested organisms; no interference with influenza A or B detection from microorganisms or human DNA. | All 41 tested bacterial, viral, fungal organisms and human DNA showed no cross-reactivity or interference. |
| Interfering Substances | No interference observed with common respiratory substances. | No interference observed for any of the 20+ tested substances at specified concentrations. |
| Performance Near Cutoff | Untrained users can accurately interpret and perform the test at and below the LoD. | Agreement for low positive and high negative samples ranged from 96.83% to 100% across multiple sites and operators. |
2. Sample Size Used for the Test Set and Data Provenance
- Sample Size for Test Set:
- Clinical Performance Study: 1003 evaluable nasal or nasopharyngeal swab samples were included in the primary analysis. A total of 1053 subjects were enrolled initially.
- Near Cutoff Study: A panel of 84 samples was tested at each of the three CLIA-waived sites, totaling 252 tests.
- Reproducibility Study: Data for each sample type (e.g., Flu A High Negative) involved 30 replicates per site for 3 sites, totaling 90 replicates per sample type.
- Data Provenance:
- Country of Origin: The clinical study was conducted at sixteen investigational sites across the U.S. ("across the U.S.").
- Retrospective or Prospective: The primary clinical study was prospective, conducted during the 2017-2018 influenza season. The "Assay Cutoff" section mentions a clinical dataset comprised of 1252 "prospectively and retrospectively collected samples" was used for ROC analysis, indicating a mix for cutoff optimization, but the main performance evaluation uses prospective data. The Near Cutoff Study was also prospective, conducted during a "normal testing day" over "non-consecutive days."
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications of Those Experts
The document does not specify the number or qualifications of experts for establishing ground truth for the clinical test set. Instead, it states that the ground truth was established by a "Composite Reference" consisting of:
- Two FDA-cleared molecular influenza A&B assays
- Cell culture
A sample was considered positive or negative if two or three of these comparative reference methods agreed. This implies an objective, laboratory-based ground truth rather than a panel of human expert review.
4. Adjudication Method for the Test Set
The adjudication method for the clinical test set was a Composite Reference approach.
- "A sample was considered positive for influenza A or influenza B by the Composite Reference if two or three of the comparative reference methods gave a positive result."
- "A sample was considered negative for influenza A or influenza B by the composite reference if two or three of the comparative reference methods gave a negative result."
This is a form of 2-out-of-3 or 3-out-of-3 majority vote from the reference methods, ensuring robust ground truth.
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study Was Done, and the Effect Size of How Much Human Readers Improve with AI vs. Without AI Assistance
This document describes an in vitro diagnostic device (a rapid test and its reading system), not an AI-assisted diagnostic imaging device for human readers. Therefore, an MRMC comparative effectiveness study involving human readers and AI assistance is not applicable and was not performed. The "Acucy Reader" is an optoelectronic instrument that automatically processes the lateral flow test, so human interpretation of the test line is not the primary mechanism of reading.
6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) Was Done
Yes, the primary clinical performance evaluation and all analytical studies (LoD, reactivity, specificity, interfering substances) represent standalone performance of the device (Acucy™ Influenza A&B Test with the Acucy™ System). The Acucy Reader is an automated system that "scans the Test Cassette and measures the absorbance intensity by processing the results using method-specific algorithms" and "displays the test results POS (+), NEG (-), or INVALID on the screen." This is purely algorithm/device driven, without human interpretation of the test lines themselves.
7. The Type of Ground Truth Used (expert consensus, pathology, outcomes data, etc.)
The ground truth used for the clinical performance evaluation was a Composite Reference based on laboratory methods:
- Two FDA-cleared molecular influenza A&B assays
- Cell culture
This is a robust and objective ground truth commonly used for infectious disease diagnostics.
8. The Sample Size for the Training Set
The document describes a 510(k) submission for a diagnostic test kit involving a lateral flow immunoassay and an optical reader. This is a traditional IVD device, not an AI/ML model that typically requires a large "training set" in the machine learning sense.
However, the "Assay Cutoff" section mentions:
- Initial determination using "contrived influenza A samples, influenza B samples, and negative samples prepared in clinical nasal matrix," tested in "replicates of 60 with two lots of reagents (a total of 360 test results)." This internal testing likely contributed to the initial algorithm development or parameter setting.
- "To validate the primary cutoff values, a clinical dataset comprised of 1252 prospectively and retrospectively collected samples was tested with the Acucy Influenza A&B Test, and Receiver Operator Characteristic (ROC) analysis was performed to determine the optimal values for sensitivity and specificity." This dataset of 1252 samples was used for optimizing and validating the assay cutoffs, which is analogous to a development/validation set in an ML context, though not a "training set" in the iterative learning sense.
9. How the Ground Truth for the Training Set Was Established
For the 1252 samples used for ROC analysis and cutoff adjustment:
- The document implies that these samples were "clinical samples," and it's highly probable their true status (positive/negative for Flu A/B) would have been determined by similar reference methods (molecular assays, cell culture) as used for the main clinical study, or other highly accurate laboratory methods.
- For the "contrived samples" used for initial cutoff determination, the ground truth was known by design (i.e., whether the sample was spiked with influenza virus and at what concentration).
§ 866.3328 Influenza virus antigen detection test system.
(a)
Identification. An influenza virus antigen detection test system is a device intended for the qualitative detection of influenza viral antigens directly from clinical specimens in patients with signs and symptoms of respiratory infection. The test aids in the diagnosis of influenza infection and provides epidemiological information on influenza. Due to the propensity of the virus to mutate, new strains emerge over time which may potentially affect the performance of these devices. Because influenza is highly contagious and may lead to an acute respiratory tract infection causing severe illness and even death, the accuracy of these devices has serious public health implications.(b)
Classification. Class II (special controls). The special controls for this device are:(1) The device's sensitivity and specificity performance characteristics or positive percent agreement and negative percent agreement, for each specimen type claimed in the intended use of the device, must meet one of the following two minimum clinical performance criteria:
(i) For devices evaluated as compared to an FDA-cleared nucleic acid based-test or other currently appropriate and FDA accepted comparator method other than correctly performed viral culture method:
(A) The positive percent agreement estimate for the device when testing for influenza A and influenza B must be at the point estimate of at least 80 percent with a lower bound of the 95 percent confidence interval that is greater than or equal to 70 percent.
(B) The negative percent agreement estimate for the device when testing for influenza A and influenza B must be at the point estimate of at least 95 percent with a lower bound of the 95 percent confidence interval that is greater than or equal to 90 percent.
(ii) For devices evaluated as compared to correctly performed viral culture method as the comparator method:
(A) The sensitivity estimate for the device when testing for influenza A must be at the point estimate of at least 90 percent with a lower bound of the 95 percent confidence interval that is greater than or equal to 80 percent. The sensitivity estimate for the device when testing for influenza B must be at the point estimate of at least 80 percent with a lower bound of the 95 percent confidence interval that is greater than or equal to 70 percent.
(B) The specificity estimate for the device when testing for influenza A and influenza B must be at the point estimate of at least 95 percent with a lower bound of the 95 percent confidence interval that is greater than or equal to 90 percent.
(2) When performing testing to demonstrate the device meets the requirements in paragraph (b)(1) of this section, a currently appropriate and FDA accepted comparator method must be used to establish assay performance in clinical studies.
(3) Annual analytical reactivity testing of the device must be performed with contemporary influenza strains. This annual analytical reactivity testing must meet the following criteria:
(i) The appropriate strains to be tested will be identified by FDA in consultation with the Centers for Disease Control and Prevention (CDC) and sourced from CDC or an FDA-designated source. If the annual strains are not available from CDC, FDA will identify an alternative source for obtaining the requisite strains.
(ii) The testing must be conducted according to a standardized protocol considered and determined by FDA to be acceptable and appropriate.
(iii) By July 31 of each calendar year, the results of the last 3 years of annual analytical reactivity testing must be included as part of the device's labeling. If a device has not been on the market long enough for 3 years of annual analytical reactivity testing to have been conducted since the device received marketing authorization from FDA, then the results of every annual analytical reactivity testing since the device received marketing authorization from FDA must be included. The results must be presented as part of the device's labeling in a tabular format, which includes the detailed information for each virus tested as described in the certificate of authentication, either by:
(A) Placing the results directly in the device's § 809.10(b) of this chapter compliant labeling that physically accompanies the device in a separate section of the labeling where the analytical reactivity testing data can be found; or
(B) In the device's label or in other labeling that physically accompanies the device, prominently providing a hyperlink to the manufacturer's public Web site where the analytical reactivity testing data can be found. The manufacturer's home page, as well as the primary part of the manufacturer's Web site that discusses the device, must provide a prominently placed hyperlink to the Web page containing this information and must allow unrestricted viewing access.
(4) If one of the actions listed at section 564(b)(1)(A)-(D) of the Federal Food, Drug, and Cosmetic Act occurs with respect to an influenza viral strain, or if the Secretary of Health and Human Services (HHS) determines, under section 319(a) of the Public Health Service Act, that a disease or disorder presents a public health emergency, or that a public health emergency otherwise exists, with respect to an influenza viral strain:
(i) Within 30 days from the date that FDA notifies manufacturers that characterized viral samples are available for test evaluation, the manufacturer must have testing performed on the device with those viral samples in accordance with a standardized protocol considered and determined by FDA to be acceptable and appropriate. The procedure and location of testing may depend on the nature of the emerging virus.
(ii) Within 60 days from the date that FDA notifies manufacturers that characterized viral samples are available for test evaluation and continuing until 3 years from that date, the results of the influenza emergency analytical reactivity testing, including the detailed information for the virus tested as described in the certificate of authentication, must be included as part of the device's labeling in a tabular format, either by:
(A) Placing the results directly in the device's § 809.10(b) of this chapter compliant labeling that physically accompanies the device in a separate section of the labeling where analytical reactivity testing data can be found, but separate from the annual analytical reactivity testing results; or
(B) In a section of the device's label or in other labeling that physically accompanies the device, prominently providing a hyperlink to the manufacturer's public Web site where the analytical reactivity testing data can be found. The manufacturer's home page, as well as the primary part of the manufacturer's Web site that discusses the device, must provide a prominently placed hyperlink to the Web page containing this information and must allow unrestricted viewing access.