(170 days)
The ARCHITECT Syphilis TP assay is a chemiluminescent microparticle immunoassay (CMIA) for the qualitative detection of antibodies (IgG and IgM) directed against Treponema pallidum (TP) in human serum and plasma. The ARCHITECT Syphilis TP assay is intended to be used as an initial diagnostic test or in conjunction with a nontreponemal laboratory test and clinical findings to aid in the diagnosis of syphilis infection.
Warning: The ARCHITECT Syphilis TP assay is not intended for use in screening blood, plasma, or tissue donors. The effectiveness of the ARCHITECT Syphilis TP assay for use in screening blood, plasma, or tissue donors has not been established.
The ARCHITECT Syphilis TP Calibrator is for the ARCHITECT iSystem when used for the qualitative detection of antibody to Treponema pallidum (TP) in human serum and plasma.
The ARCHITECT Syphilis TP Controls are for the estimation of test precision of systematic analytical deviations of the ARCHITECT iSystem when used for the qualitative detection of antibody to Treponema pallidum (TP) in human serum and plasma.
The ARCHITECT Syphilis TP assay is a two-step immunoassay for the qualitative detection of antibodies (IgG and IgM) directed against TP in human serum or plasma using CMIA technology with flexible assay protocols, referred to as Chemiflex.
- Sample, assay diluent, and recombinant TP antigen (TpN15, TpN17 and TpN47) coated microparticles are combined. Anti-TP antibodies present in the sample bind to the TP coated microparticles.
- After washing, anti-human IgG and IgM acridinium-labeled conjugate is added to create a reaction mixture.
- Following another wash cycle, Pre-Trigger and Trigger Solutions are added to the reaction mixture.
- The resulting chemiluminescent reaction is measured as relative light units (RLUs). There is a direct relationship between the amount of anti-TP antibodies in the sample and the RLUs detected by the ARCHITECT iSystem optics.
The presence or absence of anti-TP antibodies in the specimen is determined by comparing the chemiluminescent signal in the reaction to the cutoff signal determined from an active calibration. If the chemiluminescent signal in the reaction is greater than or equal to the cutoff signal, the specimen is considered reactive for anti-TP antibodies.
Here's a breakdown of the acceptance criteria and study information for the ARCHITECT Syphilis TP Reagent Kit:
1. Acceptance Criteria and Reported Device Performance
The document does not explicitly state pre-defined acceptance criteria for positive and negative percent agreement. Instead, it presents the calculated percent agreements observed in the clinical study. The FDA's substantial equivalence decision implies that these observed performance metrics were deemed acceptable when compared to the predicate device.
| Performance Metric | Acceptance Criteria (Implied by FDA Acceptance) | Reported Device Performance (ARCHITECT Syphilis TP Assay) |
|---|---|---|
| Overall Positive Percent Agreement (Prospectively-Collected Intended Use Population) | Expected high agreement with composite reference for positive cases. | 96.2% (153/159) with 95% CI: 92.0% to 98.3% |
| Overall Negative Percent Agreement (Prospectively-Collected Intended Use Population) | Expected high agreement with composite reference for negative cases. | 99.0% (976/986) with 95% CI: 98.1% to 99.4% |
| Positive Percent Agreement (Routine Syphilis Category) | Expected high agreement for this subpopulation. | 97.3% (36/37) with 95% CI: 86.2-99.5% |
| Negative Percent Agreement (Routine Syphilis Category) | Expected high agreement for this subpopulation. | 99.5% (403/405) with 95% CI: 98.2-99.9% |
| Positive Percent Agreement (HIV Positive Category) | Expected high agreement for this subpopulation. | 95.9% (117/122) with 95% CI: 90.8-98.2% |
| Negative Percent Agreement (HIV Positive Category) | Expected high agreement for this subpopulation. | 97.5% (270/277) with 95% CI: 94.9-98.8% |
| Positive Percent Agreement (Pre-Selected Positive Specimens) | Expected high agreement for this enriched population. | 98.9% (376/380) with 95% CI: 97.3% to 99.6% |
| Negative Percent Agreement (Pre-Selected Positive Specimens) | Expected high agreement for this enriched population. | 92.3% (24/26) with 95% CI: 75.9% to 97.9% |
| Positive Percent Agreement (Presumed Positive Category) | Expected high agreement for this subpopulation. | 98.9% (356/360) with 95% CI: 97.2-99.6% |
| Negative Percent Agreement (Presumed Positive Category) | Expected high agreement for this subpopulation. | 92.3% (24/26) with 95% CI: 75.9-97.9% |
| Positive Percent Agreement (Reactive Pregnant Category) | Expected 100% agreement for this subpopulation. | 100.0% (20/20) with 95% CI: 83.9-100.0% |
| Negative Percent Agreement (Prospectively-Collected Pregnant Females) | Expected high agreement for this subpopulation. | 99.7% (303/304) with 95% CI: 98.2-99.9% |
Note: NA (not applicable) for some categories indicates that there were no positive or negative cases, respectively, in that specific subpopulation to calculate the agreement.
The document also describes several non-clinical performance acceptance criteria through studies such as:
- Precision: Evaluated within-run, within-laboratory, within-day, between-site, and between-lot variability. Specific SD and %CV values are reported for controls and panels. The implication is that these values demonstrate acceptable precision.
- Interference: Demonstrated that the assay is not susceptible to interference from various substances at specified levels (e.g., conjugated bilirubin ≤ 20 mg/dL, hemoglobin ≤ 500 mg/dL).
- Analytical Specificity: Evaluation against various medical conditions and disease states. While some cross-reactivity was noted for a few specimens in specific categories (e.g., CMV IgG, HIV), the overall performance and context are considered acceptable, with a specific limitation noted for yaws, pinta, and bejel.
- Tube Type Matrix Comparison: Acceptable mean/median S/CO differences and distribution of S/CO differences for different tube types compared to serum.
- Sample On-Board Stability: Up to 3 hours storage on the ARCHITECT i2000SR System.
- Sample Stability: Up to 7 days at 2-8°C, up to 72 hours at room temperature, up to 6 freeze/thaw cycles, and up to 30 days frozen (-10°C or colder).
- High Dose Hook Effect: Determined not to be susceptible.
- Within-Assay Sample Carryover: Determined not to be susceptible.
2. Sample Sizes Used for the Test Set and Data Provenance
Test Set Sample Sizes:
- Clinical Study Total Specimens: 2220
- Prospectively-Collected Intended Use Population: 1145 specimens
- Routine Syphilis Testing: 442
- Pregnant Females: 304
- HIV Positive: 399
- Pre-Selected Positive for TP Antibodies: 406 specimens
- Apparently Healthy Individuals: 480 specimens
- Medically Diagnosed (Primary, Secondary, Latent Syphilis): 179 specimens
- Spiked Pregnant Female Specimens: 10 specimens
- Prospectively-Collected Intended Use Population: 1145 specimens
Data Provenance:
- Country of Origin: United States. Specimens for the prospectively-collected intended use population were sourced or collected from various locations: Baltimore, Maryland; Colton, California; Fort Lauderdale, Florida; Hyannis, Massachusetts; Los Angeles, California; Miami, Florida; San Antonio, Texas; and Temple, Texas.
- Retrospective/Prospective:
- Prospectively-collected: 1145 specimens from the intended use population were collected prospectively.
- Pre-selected: 406 specimens were "pre-selected positive" based on previous laboratory testing, suggesting a retrospective selection criterion based on existing positive results.
- Medically diagnosed and Apparently healthy: These populations represent specific cohorts selected for the study.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications
This document describes a diagnostic device for syphilis, not an imaging diagnostic AI. Therefore, the "ground truth" is a laboratory-based composite reference, not expert-read interpretations of images.
The ground truth for the clinical studies was established using comparator assays, not human experts interpreting data for a "ground truth" consensus in the way a radiologist would for an imaging study. The comparator results were determined using a "2 out of 3 rule" from the following assays:
- Treponemal Chemiluminescent Immunoassay (TP-CLIA)
- Rapid Plasma Reagin (RPR) - a nontreponemal assay
- Treponema Pallidum Particle Agglutination (TP-PA) - a second treponemal assay
For the "medically diagnosed" category, the diagnosis was made by a licensed physician based on clinical information and serological testing results (e.g., VDRL, RPR, TP-PA). While a physician's input is a form of expert opinion, it's not described as a multi-expert consensus process specifically for establishing ground truth for the algorithm's direct input.
4. Adjudication Method for the Test Set
The adjudication method for the clinical test set (specifically for the final comparator result) was a "2 out of 3 rule" involving the three comparator assays: TP-CLIA, RPR, and TP-PA. If at least two of these assays were concordant (e.g., both positive or both negative), that determined the final comparator result.
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
No, a Multi-Reader Multi-Case (MRMC) comparative effectiveness study was not done. This device is an in-vitro diagnostic assay, not an AI-powered imaging or interpretation system designed to assist human readers. Therefore, the concept of "human readers improving with AI vs without AI assistance" does not apply here.
6. Standalone Performance Study
Yes, a standalone performance study was done. The entire document describes the performance of the ARCHITECT Syphilis TP assay as a standalone diagnostic device. Its performance metrics (positive percent agreement, negative percent agreement, precision, specificity, etc.) are all measures of the algorithm's (assay's) performance without human-in-the-loop intervention for result interpretation. The assay outputs a qualitative "Reactive" or "Nonreactive" result based on its internal cutoff calculation (S/CO).
7. Type of Ground Truth Used
The primary type of ground truth used was a composite reference standard derived from the results of three established laboratory assays: a treponemal chemiluminescent immunoassay (TP-CLIA), a nontreponemal assay (Rapid Plasma Reagin [RPR]), and a second treponemal assay (Treponema Pallidum Particle Agglutination [TP-PA]), adjudicated by a "2 out of 3 rule."
Additionally, for some categories, clinical context and medical diagnosis by a licensed physician were used to characterize the patient population (e.g., "medically diagnosed individuals with primary, secondary, or latent syphilis").
8. Sample Size for the Training Set
The document does not explicitly state a separate "training set" sample size for the ARCHITECT Syphilis TP assay. For in-vitro diagnostic devices like this, the "development" or "training" of the assay (e.g., selection of antigens, optimization of reagents, determination of cutoff) is often an iterative process using internal studies and samples, rather than a distinct, large "training set" in the machine learning sense.
The "internal study" for selecting and validating the cutoff is mentioned, indicating some data was used for this purpose, but the size is not specified. The clinical studies described are primarily for validation and performance assessment (akin to a test set).
9. How the Ground Truth for the Training Set was Established
Since a distinct "training set" in the machine learning context is not explicitly described, the method for establishing its ground truth is also not detailed. However, based on the general development of such assays, the "ground truth" for optimizing the assay's components and cutoff during development would likely involve:
- Known positive and negative samples: Samples from individuals with confirmed syphilis infections (e.g., by culture, Western blot, or a panel of established treponemal and non-treponemal tests) and samples from healthy individuals.
- Reference standards: Internal or international reference standards for Treponema pallidum antibodies.
The "internal study" for cutoff selection and validation would have used such characterized samples to ensure the assay's sensitivity and specificity met developmental targets before moving to large-scale clinical validation.
§ 866.3830
Treponema pallidum treponemal test reagents.(a)
Identification. Treponema pallidum treponemal test reagents are devices that consist of the antigens, antisera and all control reagents (standardized reagents with which test results are compared) which are derived from treponemal sources and that are used in the fluorescent treponemal antibody absorption test (FTA-ABS), theTreponema pallidum immobilization test (T.P.I.), and other treponemal tests used to identify antibodies toTreponema pallidum directly from infecting treponemal organisms in serum. The identification aids in the diagnosis of syphilis caused by bacteria belonging to the genusTreponema and provides epidemiological information on syphilis.(b)
Classification. Class II (performance standards).