(90 days)
The Focus Technologies West Nile Virus IgM Capture ELISA is intended for qualitatively detecting IgM antibodies to West Nile virus in human serum. In conjunction with the Focus Technologies West Nile Virus ELISA IgG, the test is indicated for testing persons having symptoms of meningioencephalitis, as an aid in the presumptive laboratory diagnosis of West Nile virus infection. Positive results must be tested using the background subtraction method (either on the initial test or on a repeat test). Positive results must be confirmed by neutralization test, or by using the current CDC guidelines for diagnosing West Nile encephalitis. This test is not intended for self-testing, and this test is not FDA cleared nor approved for testing blood or plasma donors. Assay performance characteristics have not been established for automated instruments.
Indirect Enzyme-linked immunosorbent assay for qualitatively detecting human serum IgM antibodies to West Nile virus.
Here's a summary of the acceptance criteria and the studies that demonstrate the device meets those criteria, based on the provided text:
Acceptance Criteria and Device Performance
The provided document does not explicitly state "acceptance criteria" in a separate, clear table with pre-defined thresholds. However, the performance characteristics studies implicitly define the expected performance benchmarks based on agreement with reference assays and clinical classifications. For the purpose of this response, I will synthesize the performance results as the "reported device performance" and infer "acceptance criteria" from the documented agreements and sensitivities/specificities deemed acceptable for regulatory submission.
Note on Background Subtract: The device's performance is consistently evaluated both "without Background Subtract" and "with Background Subtract." The Intended Use explicitly states that "Positive results must be tested using the background subtraction method," suggesting that the performance with background subtract is the clinically relevant and accepted performance. Therefore, the table below will primarily focus on the "with Background Subtract" results as the crucial performance metrics, but acknowledge the "without" where significant differences exist.
| Acceptance Criterion (Inferred from Study Results and Predicate Device Performance) | Reported Device Performance (with Background Subtract) |
|---|---|
| Clinical Sensitivity (Encephalitis/Meningitis Patients: Confirmed WNV) | 93.2% (41/44) (95% CI 78.3-97.5%) |
| Agreement with Presumptive CDC IgM ELISA (Presumed Positive WNV) | 100% (2/2) (95% CI 15.8-100%) |
| Agreement with Presumptive CDC IgM ELISA (Presumed Negative WNV) | 100% (250/250) (95% CI 98.6-100%) |
| Serological Sensitivity (WNV PRNT Positive) | 100% (70/70) (95% CI 94.9-100%) |
| Negative Agreement with Presumptive WNV IFA (WNV IFA Negative) | 98.1% (101/103) (95% CI 93.2-99.8%) |
| Serological Sensitivity (Suspected Encephalitis/Meningitis, Confirmed WNV) | 100% (1/1) (95% CI NA) |
| Negative Agreement with Presumptive CDC IgM ELISA (Suspected Encephalitis/Meningitis, Presumptive Negative) | 100% (49/49) (95% CI 92.7-100%) |
| Positive Agreement with Presumptive CDC IgM ELISA (Non-Flavivirus Test Samples, CDC Positive) | 66.7% (2/3) (95% CI 9.4-99.2%). (Note: This rate is for non-flavivirus test samples, not primary WNV suspected cases; the sample size is very small. Without background subtract, it was 33.3%). |
| Negative Agreement with Presumptive CDC IgM ELISA (Non-Flavivirus Test Samples, CDC Negative) | 100% (469/469) (95% CI 99.2-100%) |
| Cross-Reactivity with other Flaviviruses/Pathogens | Varies significantly by pathogen. For example, Dengue virus (secondary infections) showed 40.0% positivity with background subtract. St. Louis Encephalitis not tested with background subtract, but was 53.8% positive without. Other pathogens showed 0% positivity with background subtract (e.g., Herpes simplex, Epstein-Barr, Cytomegalovirus, Borrelia burgdorferi, Rheumatoid factor, Anti-nuclear antibodies, Polio virus). |
| Reproducibility (Inter-Lab, Inter-assay, Intra-assay) | Coefficients of Variation (CVs) were generally low for positive samples and higher for negative/equivocal samples (e.g., for BS22 (positive), Inter-Lab %CV 2.1, Inter-assay %CV 7.6, Intra-assay %CV 3.4 with background subtract). |
| Specificity (2-mercaptoethanol treatment) | 100% (15/15) of WNV IgM/IgG positive samples became IgM negative after 2-ME treatment, indicating IgM specificity. |
| Freeze-Thaw Stability | No changes in interpretation were observed for positive or negative samples across up to 5 freeze-thaw cycles. |
Study Details:
-
Sample size used for the test set and the data provenance (e.g. country of origin of the data, retrospective or prospective)
- Study Site 1 (Encephalitis/Meningitis Patients):
- Sample Size: 300 patients. 44 confirmed positive (WNV encephalitis/meningitis symptoms, CDC IgM ELISA positive and WNV PRNT positive), 256 presumptive results (250 presumed negative, 2 presumed positive, 4 excluded).
- Data Provenance: Sera were sequentially submitted to a state department of health laboratory located in the northeastern U.S., archived, and masked. Prospective collection from clinical suspicion.
- Study Site 2 & 4 (WNV PRNT Positives):
- Sample Size: 75 retrospective samples (70 ultimately tested with background subtract).
- Data Provenance: Retrospective samples from a clinical laboratory (mid-western U.S.) with no clinical information, pre-screened positive by Focus and confirmed by WNV PRNT.
- Study Site 3 (West Nile IFA Negatives):
- Sample Size: 103 samples.
- Data Provenance: Reactive samples that were West Nile IFA negative, from a clinical laboratory located in the southwestern U.S. (Retrospective implied by "assessed reactive samples").
- Study Site 4 (Suspected Encephalitis/Meningitis Patients):
- Sample Size: 50 samples.
- Data Provenance: Archived and masked sera from a U.S. federal government laboratory. One confirmed positive by WNV PRNT, 49 presumptively negative by CDC ELISA for arboviruses.
- Study Site 4 (Non-Flavivirus Test Samples):
- Sample Size: 476 samples (4 samples were indeterminant with CDC IgM ELISA and excluded from final calculations).
- Data Provenance: Prospectively collected from North America during August 2003. Submitted to a clinical laboratory in Southern California for non-flavivirus tests.
- Cross-Reactivity Study (Study Site 1 & 4):
- Sample Size: 75 samples sero-positive to other potentially cross-reactive pathogens.
- Data Provenance: Retrospective and masked sera. DOH (northeastern U.S.) tested St. Louis encephalitis positives, Focus (Study Site 4) tested others.
- Specificity (2-ME treatment, Study Site 4):
- Sample Size: 15 sera.
- Data Provenance: Sera positive for both WNV IgM and IgG.
- Freeze-Thaw Study (Study Site 4):
- Sample Size: 8 sera (5 positive, 3 negative).
- Data Provenance: N/A (likely in-house selection).
- Reproducibility Studies (Inter-lot, Inter/Intra-assay, Inter-laboratory):
- Sample Size: Varies by study: 5 samples for inter-lot, 7 for inter/intra-assay (63 data points), 7 for inter-laboratory. Masked duplicates were used.
- Data Provenance: Study Site 4 (Focus), Study Site 5 (mid-western U.S. clinical lab), Study Site 6 (northern California university lab), Study Site 1 (northeastern U.S. state DOH lab), Study Site 2 (mid-western U.S. clinical lab assumed different from site 5 for variety).
- Study Site 1 (Encephalitis/Meningitis Patients):
-
Number of experts used to establish the ground truth for the test set and the qualifications of those experts (e.g. radiologist with 10 years of experience)
The document does not specify the "number of experts" or their "qualifications" involved in establishing the ground truth. Ground truth was primarily established using reference assays such as:- CDC West Nile Virus IgM ELISAs
- West Nile Virus Plaque Reduction Neutralization Test (WNV PRNT)
- West Nile IFA
- Clinical criteria for encephalitis/meningitis (fever, altered mental status, CSF pleocytosis, etc.)
These reference methods are established laboratory techniques, implying their performance is well-understood and accepted, but the exact human expert involvement in interpreting these for ground truth is not detailed.
-
Adjudication method (e.g. 2+1, 3+1, none) for the test set
The document does not describe an explicit adjudication method (like 2+1 or 3+1 consensus). Ground truth was determined by comparing the device's results to a predefined reference standard (e.g., CDC IgM ELISA, WNV PRNT, WNV IFA, or clinical criteria), which implicitly serves as the adjudication or definitive determination. -
If a multi reader multi case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance
This is an in vitro diagnostic (IVD) device, specifically an ELISA assay. The concept of "human readers" for an "AI" assistance is not applicable in the context of this type of diagnostic test. The results of an ELISA are typically read by a spectrophotometer and interpreted based on a cutoff value, rather than a human "reader" making subjective interpretations that could be "assisted" by AI. Therefore, an MRMC comparative effectiveness study involving human readers with/without AI assistance was not done. -
If a standalone (i.e. algorithm only without human-in-the-loop performance) was done
Yes, the studies presented are standalone performance evaluations of the Focus Technologies West Nile Virus IgM Capture ELISA. The device (an immunoassay kit) itself is the "algorithm" in this context, providing a quantitative output that is then interpreted against a cutoff. The performance metrics (sensitivity, specificity, agreement) directly reflect the device's ability to classify samples independently of human interpretational variability beyond reading the spectrophotometer. The "background subtract procedure" is an integral part of the assay's interpretive algorithm to mitigate false positives, as described in the "Test Principle" section. -
The type of ground truth used (expert consensus, pathology, outcomes data, etc.)
The ground truth was established using reference laboratory assays (CDC WNV IgM ELISA, WNV PRNT, WNV IFA) and clinical diagnostic criteria for encephalitis/meningitis (which implicitly involves expert clinical judgment and outcomes). In some cases, previous positive screening results by other methods were also used to enrich the sample set. There is no mention of pathology (histopathological examination) or long-term outcomes data being used directly as ground truth for all studies, although the "confirmed" positive cases were patients with symptoms of meningioencephalitis. -
The sample size for the training set
The document does not specify a separate "training set" for the device. As an ELISA kit, it is a chemical/biological assay rather than a machine learning algorithm that requires explicit training data in the same sense. The performance characteristics were evaluated on various test sets as described above, but these are typically considered validation sets for an established assay methodology, rather than a "training set" that a machine learning model would use to learn parameters. -
How the ground truth for the training set was established
Since no explicit "training set" is mentioned in the context of a machine learning algorithm, this question is not directly applicable. If one considers the development and optimization of the ELISA assay itself, the ground truth would have been established during the research and development phase through various experiments using known positive and negative samples, similar to the reference assays used in the validation studies (e.g., using WNV PRNT confirmed samples, CDC ELISAs). However, the document does not detail this R&D process.
§ 866.3940 West Nile virus serological reagents.
(a)
Identification. West Nile virus serological reagents are devices that consist of antigens and antisera for the detection of anti-West Nile virus IgM antibodies, in human serum, from individuals who have signs and symptoms consistent with viral meningitis/encephalitis. The detection aids in the clinical laboratory diagnosis of viral meningitis/encephalitis caused by West Nile virus.(b)
Classification. Class II (special controls). The special control is FDA's guidance entitled “Class II Special Controls Guidance Document: Serological Reagents for the Laboratory Diagnosis of West Nile Virus.” See § 866.1(e) for the availability of this guidance document.