(92 days)
The BioStar® OIA® SHIGATOX assay is an Optical Immunoassay (OIA) test for the qualitative, rapid detection of the presence of Shiga toxins in human diarrheal fecal specimens, broth cultures, fecal specimens in Cary Blair Transport Media, or swab sampling of colonies from a culture plate. This test is intended for in vitro diaqnostic use as an aid in the diagnosis of infection by Shiga toxin-producing Escherichia coli (STEC) both O157 and all non - O157 Shiga toxin-producing strains.
The OIA SHIGATOX test involves the qualitative detection of Shiga toxins 1 and 2 (Stx1 and Stx 2) produced by certain strains of Escherichia coli and other organisms. The Optical ImmunoAssay technology enables the direct visual detection of a physical change in the optical thickness of molecular thin films. This change is the result of antigen - antibody binding on an optical surface (silicon wafer). After a specimen potentially containing Shiga toxin is mixed with conjugates and placed directly on the optical surface, the immobilized surface antibodies capture the antigen/conjugate complex. After washing, the substrate is added, increasing the thickness (mass enhancement) of the molecular thin film. This change in thickness alters the reflected light path, and this alteration is visually perceived as a color change. Slight changes in the optical thickness produce a distinct visible color change. A positive result appears as a purple spot on the gold background. When antigen is not present in the specimen, no binding takes place. Therefore, the optical thickness remains unchanged, and the surface retains the original gold color indicating a negative result.
More specifically, the BioStar OIA SHIGATOX device is based on a novel thin film optical detection technology that relies on the interaction of white light with thin films to create a destructive interference phenomenon. Characteristic of this phenomenon is the generation of a reflective surface that changes color as a function of the change in optical thickness (refractive index x thickness) of the films on the surface of the device. To take advantage of this phenomenon for monitoring biological binding events, the optical surface with a special background color is coated with a capture reagent specific to the analyte of interest. In the OIA SHIGATOX device, the biological capture film is a combination of affinity-purified polyclonal antibodies to Shiga toxins 1 and 2 (Stx 1 and Stx 2). Samples suspected of containing either or both of the toxins are mixed with cocktail containing polyclonal antibodies to Stx 1 and Stx 2 that have been covalently conjugated to horseradish peroxidase (HRP). Once a sample containing toxins or either toxin is applied to the surface, the immune complex of toxin(s) and the anti-toxin-HRP conjugate(s) are bound to the surface antibodies. Following a wash step, a precipitating substrate for HRP is added, and a thin film generated by the immobilized immune complex is enhanced by the precipitation of the HRP product. Once washed and dried, a simple color change relative to the gold background color is observed as an indication of the presence of Stx 1 or Stx 2 in the original specimen.
The OIA SHIGATOX device produces a qualitative result for the presence or absence of Shiga toxin as the device output. Input to the device is the simple addition of an aliquot of fecal material (direct or in transport media) or broth culture to the reagents contained in the kit. Fecal samples are routinely collected and no special collection requirements exist. Test devices within the kit are single use devices, and disposal instructions are provided in the Package Insert. The kit contains all components necessary for analysis of the range of samples approved for use in this product, with the exception of a timer.
Here's a summary of the acceptance criteria and study details for the BioStar® OIA® SHIGATOX device based on the provided text:
Acceptance Criteria and Reported Device Performance
The acceptance criteria for each study type are implicitly defined by the reported performance, which consistently achieved high accuracy. For the analytical studies, the key criteria were demonstrating sensitivity and specificity, and the reproducibility studies aimed for 100% agreement. For clinical studies, the percentage agreements with comparator methods (EIA, SMAC, CTA, or reference OIA) were the primary metrics.
| Metric / Test Type | Acceptance Criteria (Implicit from Results) | Reported Device Performance (OIA SHIGATOX) |
|---|---|---|
| Analytical Sensitivity (LOD) | Lowest toxin concentration producing at least 50% positive results. | Stx 1: 1 ng/mL (in both buffer and liquid stool) |
| Stx 2: 0.5 ng/mL (in antigen diluent); 1 ng/mL (in liquid stool) | ||
| Analytical Strain Recognition | Detect all Shiga toxin-producing strains. | 100% agreement (70/70 clinical isolates and Shigella dysenteriae) |
| Analytical Specificity (Cross-Reactivity) | Expected negative without toxin spike, expected positive with toxin spike. | All members of the cross-reactivity panel produced expected negative results without toxin spike and expected positive results with toxin spike. Demonstrated no cross-reactivity with various bacteria, fungi, and parasites, nor with commercial Rotavirus EIA positive stools. |
| Interfering Substances | No false positive or false negative results in presence of interferents. | None of the tested substances (Barium Sulfate, Bovine Mucin, Kaopectate®, Pepto Bismol®, Imodium®, Whole Blood) caused false positives or false negatives in antigen diluent or in liquid/semi-solid stool, up to specified concentrations. |
| Reproducibility | 100% inter-site and intra-site reproducibility. | Overall Score for the Study: 100% (486/486 samples across 3 clinical sites, 3 POL sites, and 3 days). Achieved 100% agreement for all operators, all days, within runs, between runs, and between sites. |
| Clinical Sensitivity/Specificity (Colony Sweep) | 100% agreement with previous specimen result. | 100% agreement (21/21 colony sweeps) with previous specimen results. |
| Clinical Performance (Direct Stool vs. EIA - Prospective) | High agreement with commercial EIA; positive samples confirmed by CTA reference. | Positive Agreement: 100% (95% CI: 73.5 - 100%) |
| Negative Agreement: 98.1% (95% CI: 95.6 - 99.4%) | ||
| Overall Percent Agreement: 98.2% (95% CI: 95.8 - 99.4%) | ||
| 1/5 OIA positive/EIA negative samples confirmed by CTA. | ||
| Clinical Performance (Direct Stool vs. EIA - Frozen) | High agreement with commercial EIA. | Positive Agreement: 87.5% (95% CI: 67.6 – 97.3%) |
| Negative Agreement: 97.4% (95% CI: 86.2 – 99.9%) | ||
| Overall Percent Agreement: 93.6% (95% CI: 84.3 – 98.2%) | ||
| Clinical Performance (Broth Culture vs. EIA - Fresh) | High agreement with commercial EIA; positive samples confirmed by CTA reference. | Positive Agreement: 100% (95% CI: 73.5 - 100%) |
| Negative Agreement: 99.6% (95% CI: 97.9 - 100%) | ||
| Overall Percent Agreement: 99.6% (95% CI: 98.0 - 100%) | ||
| 1 OIA positive/EIA negative sample confirmed by CTA. | ||
| Clinical Performance (Broth Culture vs. EIA - Frozen) | High agreement with commercial EIA. | Positive Agreement: 100% (95% CI: 84.6-100%) |
| Negative Agreement: 95.6% (95% CI: 81.7-99.9%) | ||
| Overall Percent Agreement: 98% (95% CI: 89.4 - 100%) | ||
| Clinical Performance (Direct Stool vs. SMAC - Fresh) | High agreement, acknowledging SMAC limitations. | Positive Agreement: 90% (95% CI: 55.5 – 99.8%) |
| Negative Agreement: 96.9% (95% CI: 94.0 – 98.7%) | ||
| Overall Percent Agreement: 96.7% (95% CI: 93.7 – 98.5%) | ||
| 4/8 OIA+/SMAC- samples confirmed by CTA. | ||
| Clinical Performance (Direct Stool vs. SMAC - Frozen) | High agreement, acknowledging SMAC limitations. | Positive Agreement: 100% (95% CI: 66.4-100%) |
| Negative Agreement: 75.5% (95% CI: 61.7 - 86.2%) | ||
| Overall Percent Agreement: 79% (95% CI: 66.8 - 88.3%) | ||
| All 13 OIA+/SMAC- samples were previously positive for STEC. | ||
| Clinical Performance (Direct Stool vs. CTA) | High detection rate of CTA positives. | OIA SHIGATOX detected 12/13 CTA positive direct stool samples. EIA detected 11/13. |
| Clinical Performance (Broth Culture vs. CTA) | 100% detection rate of CTA positives. | OIA SHIGATOX detected 12/12 CTA positive broth aliquots. EIA also detected 12/12. |
| Clinical Performance (1 Hour Cary Blair vs. Direct OIA) | High agreement with direct OIA reference. | Positive Agreement: 88% (95%CI: 71 - 96.5%) |
| Negative Agreement: 100% (95%CI: 94.6 - 100%) | ||
| Overall Percent Agreement: 96% (95%CI: 89.9 - 98.9 %) | ||
| Clinical Performance (24 Hour Cary Blair vs. Direct OIA) | High agreement with direct OIA reference. | Positive Agreement: 91% (95%CI: 75 - 98%) |
| Negative Agreement: 100% (95%CI: 94.6 - 100%) | ||
| Overall Percent Agreement: 97% (95%CI: 91.3 – 99.4 %) | ||
| Clinical Performance (GN Broth vs. MAC Broth - Direct Fecal) | High agreement with MAC broth reference. | Positive Agreement: 100% (95%Cl: 86.8 - 100%) |
| Negative Agreement: 95% (95%Cl: 84.5 - 99.4%) | ||
| Overall Percent Agreement: 97.1% (95%CI: 90.1 - 99.7%) | ||
| Clinical Performance (GN Broth from Cary Blair vs. Direct MAC Broth) | High agreement with MAC broth reference. | Positive Agreement: 100% (95%CI: 86.8 - 100%) |
| Negative Agreement: 98% (95%CI: 87.4 - 99.9%) | ||
| Overall Percent Agreement: 98.5% (95%CI: 92.1 - 100%) | ||
| Clinical Performance (MAC Broth from Cary Blair vs. Direct MAC Broth) | High agreement with MAC broth reference. | Positive Agreement: 93% (95%CI: 76.5 – 99.1%) |
| Negative Agreement: 93% (95%CI: 80.9 – 98.5%) | ||
| Overall Percent Agreement: 93% (95%CI: 84.3 – 97.7%) |
Study Details:
-
Sample sizes used for the test set and data provenance:
- Analytical Sensitivity: No specific sample size for a "test set" in the traditional sense, but tested in triplicate for LOD determination (5 samples at each concentration analyzed on 2 lots of devices).
- Analytical Strain Recognition: 70 clinical isolates (49 from a Department of Public Health, 21 from a university laboratory) + Shigella dysenteriae (ATCC 13313). Data provenance appears to be from US public health and university laboratories.
- Analytical Specificity (Cross-Reactivity): 43 different organisms and Rotavirus-positive stools.
- Reproducibility Studies: 9 sample types (Negative, Low Stx 1, Low Stx 2, Moderate Stx 1, Moderate Stx 2, various combinations). Each sample tested in triplicate by each of 6 operators over 3 consecutive days for a total of 486 tests (9 samples * 3 replicates * 6 operators * 3 days). Data provenance: Not explicitly stated beyond "clinical trial sites" and "Physician Office Laboratories (POL) sites." Samples were prepared by spiking stools from a healthy individual.
- Clinical Specimen Testing (Unmodified Device):
- Colony Sweep: 21 positive samples (from 22 frozen fecal specimens, one failed to grow). Data provenance: Not explicitly stated, samples were "previously found to contain Shiga toxin producing E. coli."
- Direct Stool (Prospective): 272 prospective diarrheal fecal specimens. Data provenance: Collected from three clinical trial sites in the Eastern and Western regions of the United States.
- Direct Stool (Frozen): 62 additional frozen specimens. Data provenance: Not explicitly stated, but "two of the clinical sites" performed this study.
- Broth Culture (Fresh): 269 prospective diarrheal fecal specimens (from original 272, three failed to grow). Data provenance: Same as Direct Stool (Prospective).
- Broth Culture (Frozen): 50 frozen specimens (from 62, 12 were not tested or failed to grow). Data provenance: Same as Direct Stool (Frozen).
- SMAC Culture Comparison (Fresh Stool): 269 direct stool samples. Data provenance: Same as Direct Stool (Prospective).
- SMAC Culture Comparison (Frozen Stool): 62 frozen samples. Data provenance: "two of the clinical sites."
- CTA Comparison: 19 specimens (direct stool aliquot and broth culture aliquot) from the clinical study (one excluded as inconclusive).
- Clinical Specimen Testing (Modified Device - New Sample Types):
- Cary Blair & Broth Inoculation Validation: 98 frozen fecal specimens. Data provenance: Collected from 2 laboratory sites (University of Utah & Primary Children's, Fairfax Hospital).
- 1 and 24 Hour Cary Blair (vs. Direct OIA): 98 frozen fecal specimens.
- GN/MAC Broth Culture (Direct Fecal vs. MAC Broth OIA): 98 fecal specimens.
- GN Broth from Cary Blair (vs. Direct MAC Broth OIA): 98 fecal specimens.
- MAC Broth from Cary Blair (vs. Direct MAC Broth OIA): 98 fecal specimens.
-
Number of experts used to establish the ground truth for the test set and their qualifications:
-
Analytical Strain Recognition: Ground truth based on "previously analyzed for the presence of Shiga toxin genes and serotyped" by the Department of Public Health and a university laboratory. No specific number or qualifications of experts are given, but implies laboratory expertise in molecular and serological typing.
-
Clinical Studies (Unmodified Device):
- Comparator Methods: A commercial EIA test, SMAC culture, and Cytotoxicity Testing Assay (CTA) were used as comparators or ground truth methods.
- CTA Confirmation: "All positive results from either immunoassay method were confirmed by cytotoxicity testing, CTA." CTA itself is a reference method for Shiga toxin activity. No specific number or qualifications of "experts" involved in CTA interpretation are provided, but it's an established laboratory method.
- Clinical sites were staffed by MT ASCP (Medical Technologists certified by the American Society for Clinical Pathology).
-
Reproducibility Studies: Expected results were based on the known spiked concentrations of toxins.
-
Clinical Studies (Modified Device): Direct fecal testing in the OIA method or direct fecal inoculation of MAC broth were used as reference methods.
-
-
Adjudication method for the test set:
- For the prospective clinical studies, "All positive results from either immunoassay method were confirmed by cytotoxicity testing, CTA." This implies a form of adjudication where discrepant positive immunoassay results were further investigated by CTA, serving as a higher-level reference.
- For reproducibility studies, there was no adjudication method mentioned, as results were compared to known spiked concentrations.
-
If a multi reader multi case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance:
- This is not an AI-assisted device. The device is a rapid diagnostic test (Optical Immunoassay) for direct detection of Shiga toxins, generating a visible color change for interpretation. Therefore, an MRMC comparative effectiveness study comparing human readers with and without AI assistance is not applicable.
- The "Reproducibility Studies" did involve multiple operators (6 operators across 6 sites) reading 27 blinded and randomized samples in triplicate over 3 days, demonstrating consistency in human interpretation of the device's visual output. This is a multi-reader, multi-case study for reproducibility, but not for AI-assisted human improvement.
-
If a standalone (i.e. algorithm only without human-in-the loop performance) was done:
- Yes, the performance data presented (e.g., sensitivity, specificity, agreement percentages) is the standalone performance of the OIA SHIGATOX device. The device produces a "qualitative result for the presence or absence of Shiga toxin as the device output" based on a visual color change. While a human interprets this visual change, the performance metrics reported represent the device's ability to correctly identify the toxin compared to reference methods, independent of human variability in the interpretation step. The high reproducibility across multiple readers attests to the clarity and consistency of this visual output for human interpretation.
-
The type of ground truth used (expert concensus, pathology, outcomes data, etc):
- Analytical Strain Recognition: Shiga toxin gene presence and serotyping (presumably by molecular and serological methods).
- Clinical Studies (Unmodified Device):
- Primary Comparator: Commercial EIA test.
- Confirmatory Reference: Cytotoxicity Testing Assay (CTA).
- Additional Comparator: SMAC culture (acknowledged for its limitations as a direct comparison method for all STEC).
- Clinical Studies (Modified Device):
- Primary Reference: Direct fecal testing in the OIA method (for Cary Blair comparisons) or direct fecal inoculation of MAC broth (for broth culture comparisons). These are essentially using the device itself or a similar culture method as the internal reference for evaluating new sample types with the device.
-
The sample size for the training set:
- The document does not explicitly describe a "training set" in the context of a machine learning algorithm. This is a medical device (Optical Immunoassay), not an AI/ML algorithm. The development process involved various analytical studies and clinical trials to establish performance, but these are not referred to as "training sets." The 70 clinical isolates used for Analytical Strain Recognition and the samples used for analytical sensitivity and specificity contribute to establishing the device's operational characteristics, which is analogous to a development set in traditional terms.
-
How the ground truth for the training set was established:
- As noted above, there isn't a "training set" in the context of an AI/ML device. However, the ground truth for the various analytical and clinical studies were established using methods such as:
- Known spiked concentrations of purified Stx 1 and Stx 2 toxins (for analytical sensitivity).
- Clinical isolates "previously analyzed for the presence of Shiga toxin genes and serotyped" (for analytical strain recognition).
- Reference tests like commercial EIA, SMAC culture, and crucially, Cytotoxicity Testing Assay (CTA) for clinical samples (for general device performance validation).
- As noted above, there isn't a "training set" in the context of an AI/ML device. However, the ground truth for the various analytical and clinical studies were established using methods such as:
§ 866.3255
Escherichia coli serological reagents.(a)
Identification. Escherichia coli serological reagents are devices that consist of antigens and antisera used in serological tests to identifyEscherichia coli from cultured isolates derived from clinical specimens. Additionally, some of these reagents consist ofEscherichia coli antisera conjugated with a fluorescent dye used to identifyEscherichia coli directly from clinical specimens or cultured isolates derived from clinical specimens. The identification aids in the diagnosis of diseases caused by this bacterium belonging to the genusEscherichia, and provides epidemiological information on diseases caused by this microorganism. AlthoughEscherichia coli constitutes the greater part of the microorganisms found in the intestinal tract in humans and is usually nonpathogenic, those strains which are pathogenic may cause urinary tract infections or epidemic diarrheal disease, especially in children.(b)
Classification. Class I (general controls). The device is exempt from the premarket notification procedures in subpart E of part 807 of this chapter subject to the limitations in § 866.9.