(177 days)
The PANBIO West Nile Virus IgG Indirect ELISA is for the qualitative presumptive detection of IgG antibodies to West Nile virus in serum. In conjunction with the PANBIO West Nile Virus IgM Capture ELISA, this test is intended as an aid in the clinical laboratory diagnosis of West Nile virus infection in patients with clinical symptoms consistent with encephalitis / meningitis. Positive results must be confirmed by plaque reduction neutralization test (PRNT), or by using the current Centers for Disease Control and Prevention (CDC) guidelines for diagnosis of this disease.
The PANBIO West Nile Virus IgG Indirect ELISA is for the qualitative detection of IgG antibodies to West Nile virus in serum. In conjunction with the PANBIO West Nile Virus IgM Capture ELISA, this test is intended as an aid in the presumptive laboratory diagnosis of West Nile virus infection in patients with clinical symptoms consistent with encephalitis / meningitis.
Acceptance Criteria and Device Performance for PANBIO West Nile Virus IgG Indirect ELISA
1. Table of Acceptance Criteria and Reported Device Performance
The acceptance criteria for this device are not explicitly stated as numerical targets in the provided text (e.g., "sensitivity must be > X%"). Rather, the studies aim to demonstrate adequate performance through descriptive statistical measures (sensitivity, specificity, positive and negative presumptive agreement) and reproducibility. The performance is reported with 95% Confidence Intervals. The cross-reactivity study aims to show the device's analytical specificity.
| Performance Metric | Acceptance Criteria (Implicit) | Reported Device Performance |
|---|---|---|
| Study Site 1 (Louisiana, USA) | ||
| Negative Presumptive Agreement (Endemic Normal) | Demonstrate high agreement with endemic normal specimens not known to have flavivirus related illness. | 90.5% (181/200) with 95% CI: 85.6 - 94.2%. Adjusted specificity: 95.0% (191/201) with 95% CI: 91.0 - 97.6%. |
| Serological Sensitivity (PRNT Confirmed WNV) | Demonstrate high sensitivity for WNV PRNT confirmed positive specimens. | 79.0% (79/100) with 95% CI: 69.7 – 86.5%. Adjusted sensitivity: 88.9% (88/99) with 95% CI: 81.0 – 94.3%. |
| Study Site 2 (Utah, USA) | ||
| Positive Presumptive Agreement (Encephalitis/Meningitis Patients, IgG IFA Pos) | Demonstrate agreement with IFA positive samples from symptomatic patients. | 81.3% (26/32) with 95% CI: 63.6 – 92.8%. |
| Negative Presumptive Agreement (Encephalitis/Meningitis Patients, IgG IFA Neg) | Demonstrate agreement with IFA negative samples from symptomatic patients. | 100.0% (2/2) with 95% CI: 15.8 – 100.0%. |
| Positive Presumptive Agreement (WNV IFA Positive) | Demonstrate agreement with all WNV positive samples confirmed by IFA. | 88.0% (286/325) with 95% CI: 84.5 - 91.5%. |
| Negative Presumptive Agreement (WNV IFA Negative) | Demonstrate agreement with all WNV negative samples confirmed by IFA. | 88.1% (140/159) with 95% CI: 83.0 - 93.1%. |
| Study Site 3 (Ohio, USA) | ||
| Negative Presumptive Agreement (Endemic Normal) | Demonstrate high agreement with endemic normal specimens not known to have arbovirus infection and negative by IFA. | 92.3% (180/195) with 95% CI: 87.6 - 95.6%. |
| Clinical Sensitivity (PRNT) (Encephalitis/Meningitis Patients, PRNT & IgG IFA Pos) | Demonstrate sensitivity for symptomatic patients confirmed by PRNT and IFA. | 80.4% (41/51) with 95% CI: 66.9 - 90.2%. |
| Positive Presumptive Agreement (Encephalitis/Meningitis Patients, IgG IFA Pos) | Demonstrate agreement with symptomatic patients confirmed by IFA. | 76.3% (29/38) with 95% CI: 59.8 - 88.6%. |
| Reproducibility | Demonstrate acceptable precision (low variability) across runs, days, and sites for reactive and negative samples. | Reactive Sample: Total CV 4.2% (SD 0.24). Negative Sample: Total CV 38.8% (SD 0.07). (Values for other samples ranged from 9.0% to 41.0% total CV). |
| Cross-Reactivity | Demonstrate low cross-reactivity with antibodies to other diseases, especially other flaviviruses. | Dengue: 14/15 positive or equivocal (high cross-reactivity). St. Louis encephalitis: 28/35 positive or equivocal (high cross-reactivity). Japanese encephalitis: 3/3 positive (high cross-reactivity). La Crosse encephalitis: 2/26 positive (low cross-reactivity). California encephalitis: 1/11 positive or equivocal (low cross-reactivity). Eastern Equine encephalitis: 0/1 positive. Varicella-Zoster, Cytomegalovirus, Epstein-Barr, Enterovirus, Rheumatoid Factor, Anti-Nuclear Antibody: Showed varying levels of observed reactivity, which was generally low to moderate for most non-flavivirus agents. Ross River virus: 11/39 positive or equivocal (flavivirus positive confirmed by Western blot for 11/11 reactive samples). Barmah Forest virus: 13/36 positive or equivocal (flavivirus positive confirmed by Western blot for 9/13 reactive samples). |
2. Sample Sizes and Data Provenance for Test Sets
-
Study Site 1 (Louisiana, USA):
- Sample Size: 300 retrospective sera.
- Data Provenance: Louisiana, USA. Retrospective samples collected in 2002-2003.
-
Study Site 2 (Utah, USA):
- Sample Size: 325 retrospective sera.
- Data Provenance: Utah, USA. Retrospective samples.
-
Study Site 3 (Ohio, USA):
- Sample Size: 284 retrospective sera.
- Data Provenance: Ohio, USA. Retrospective samples collected in 2002.
-
Reproducibility Study:
- Sample Size: 8 sera, tested 3 times each on 3 different assays (total 72 tests per parameter).
- Data Provenance: One Australian study site and two study sites in the USA.
-
Cross-Reactivity Study:
- Sample Size: 314 specimens in total, with varying numbers per disease state.
- Data Provenance: Multiple sites including PANBIO (Site 5), a state health laboratory in Louisiana (Site 1), a private reference laboratory in Utah (Site 2), a hospital laboratory in Ohio (Site 3), and a private research laboratory in Maryland (Site 6). The exact timing of collection for these cross-reactive samples is not specified beyond being "from patients with confirmed diseases other than WNV."
3. Number of Experts and Qualifications for Ground Truth
The document does not explicitly state the "number of experts" or their specific "qualifications" (e.g., "radiologist with 10 years of experience") for establishing the ground truth. However, the ground truth was established by:
- Plaque Reduction Neutralization Test (PRNT): This is a gold standard serological test, implying expert virological laboratory practices and interpretation.
- West Nile Virus IgG IFA (Immunofluorescence Assay): This is another standard serological method, likely performed and interpreted by trained laboratory personnel or experts.
- Clinical and serological characterization: This suggests a combination of clinical diagnosis and laboratory test results, presumably by medical professionals and laboratory specialists.
- Western blot analysis: Used for confirmation of flavivirus positivity in cross-reactivity samples, indicating expert serological techniques.
4. Adjudication Method for the Test Set
The document does not describe a formal "adjudication method" involving multiple human readers or a consensus process for discrepant results in the context of device performance evaluation (e.g., 2+1, 3+1 for image interpretation). Instead, the comparison is directly between the PANBIO ELISA results and the established "gold standard" or "characterization" of the specimens (PRNT, IgG IFA, or clinical characterization).
Equivocal results from the PANBIO ELISA were generally "not retested" due to sample unavailability or because the cut-off was modified post-clinical trials, indicating that a formal re-adjudication process for equivocal outcomes was not systematically applied within these studies.
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
No Multi-Reader Multi-Case (MRMC) comparative effectiveness study was done. The device is an in-vitro diagnostic (IVD) assay designed for laboratory testing, not an imaging device requiring human-in-the-loop interpretation. Therefore, the concept of human readers improving with or without AI assistance is not applicable to this device.
6. Standalone Performance
Yes, a standalone performance study was done. The entire evaluation presented (sensitivity, specificity, presumptive agreements, reproducibility, and cross-reactivity) describes the performance of the PANBIO West Nile Virus IgG Indirect ELISA algorithm/assay itself, without human-in-the-loop assistance in its operation or primary interpretation. The results are based on how the assay performs when processing samples and generating qualitative results (Positive, Equivocal, Negative).
7. Type of Ground Truth Used
The ground truth used for the test sets included:
- Expert Consensus/Reference Methods: Primarily, Plaque Reduction Neutralization Test (PRNT) and West Nile Virus IgG Immunofluorescence Assay (IFA) were used as reference methods to characterize samples as positive or negative for West Nile Virus IgG. These are established laboratory methods considered reliable.
- Clinical Characterization: Samples from patients with "clinical symptoms consistent with encephalitis/meningitis" were also used, implying correlation with patient presentation alongside serological markers.
- Disease State Confirmation: For the cross-reactivity study, samples were characterized by their "disease state" (e.g., Dengue virus, St. Louis encephalitis, Cytomegalovirus) and for some flaviviruses, confirmed by Western blot analysis.
8. Sample Size for the Training Set
The document does not explicitly mention a "training set" or "validation set" in the context of machine learning. This is an IVD assay, and the "training" of the assay's cut-off or parameters would typically involve internal development studies and optimization rather than a distinct "training set" like in AI/ML contexts. The performance characteristics described are based on testing against retrospectively collected "test sets" or panels of samples.
9. How the Ground Truth for the Training Set was Established
Since a distinct "training set" for an AI/ML algorithm is not described, the method for establishing its ground truth is not applicable in this context. The assay's performance evaluation relies on comparing its output to established reference methods (PRNT, IFA, Western blot) and clinical characterization using the "test sets" described above. The statement that "cut-off was modified following clinical trials" in Study Site 2 hints at an iterative process of optimizing the assay's interpretive criteria based on initial performance data.
§ 866.3940 West Nile virus serological reagents.
(a)
Identification. West Nile virus serological reagents are devices that consist of antigens and antisera for the detection of anti-West Nile virus IgM antibodies, in human serum, from individuals who have signs and symptoms consistent with viral meningitis/encephalitis. The detection aids in the clinical laboratory diagnosis of viral meningitis/encephalitis caused by West Nile virus.(b)
Classification. Class II (special controls). The special control is FDA's guidance entitled “Class II Special Controls Guidance Document: Serological Reagents for the Laboratory Diagnosis of West Nile Virus.” See § 866.1(e) for the availability of this guidance document.