(64 days)
The Emit® II Plus Cocaine Metabolite Assay is a homogeneous drugs-of-abuse enzyme immunoassay with a 150 ng/mL or 300 ng/mL cutoff (SAMSHA initial test cutoff level). The assay is intended for use in the qualitative and semiquantitative analyses of benzoylecgonine (cocaine metabolite) in human urine. Emit ® II Plus assays are designed for use with a number of chemistry analyzers.
The Emit® II Plus Cocaine Metabolite Assay provides only a preliminary analytical test result. A more specific alternative chemical method must be used to obtain a confirmed analytical result. Gas chromatography/mass spectrometry (GC/MS) is the preferred confirmatory method. Other chemical confirmation methods are available. Clinical consideration and professional judgement should be applied to any drug-of-abuse test result, particularly when preliminary positive results are used.
The Emit® II Plus Cocaine Metabolite Assay is a homogeneous enzyme immunoassay with a 150 ng/mL or 300 ng/mL cutoff. The assay is intended for use in the qualitative and semiquantitative analyses of benzoylecognine (cocaine metabolite) in human urine. The Emit® II Plus Cocaine Metabolite Assay and has been found to be equivalent to the predicate device: Emit® II Cocaine Metabolite Assay with regard to intended use, assay sample, and overall performance characteristics.
The provided document describes the Emit® II Plus Cocaine Metabolite Assay, a homogeneous enzyme immunoassay for detecting benzoylecognine (cocaine metabolite) in human urine. The study presented aims to demonstrate the substantial equivalence of this new assay to a predicate device, the Emit® II Cocaine Metabolite Assay, and its correlation with Gas chromatography/mass spectrometry (GC/MS).
Here's an analysis of the acceptance criteria and the study, organized according to your requested points:
1. Table of Acceptance Criteria and Reported Device Performance
The document does not explicitly state pre-defined acceptance criteria in a quantitative manner (e.g., "sensitivity must be > X%, specificity > Y%"). Instead, it describes performance in terms of "excellent correlation" and "acceptable precision." For the purpose of this table, I will infer the implicit criteria from the reported results and the comparison to the reference methods.
| Acceptance Criterion (Inferred) | Reported Device Performance |
|---|---|
| Correlation with GC/MS (150 ng/mL cutoff): High agreement with the gold standard for confirmatory results. | 100% agreement. (Reported as "excellent correlation to GC/MS (reference method) for the optional 150 ng/mL cutoff level. The percent agreement between these methods was 100%.") |
| Correlation with Predicate Device (300 ng/mL cutoff): High agreement with the existing legally marketed device. | 98% agreement. (Reported as "excellent correlation between Emit® II Cocaine Metabolite Assay and The Emit® II Cocaine Metabolite Assay (comparative method) at the 300 ng/mL cutoff level.") |
| Note: Three discordant samples were borderline positive with the Emit® II Plus, within the precision limit of the predicate. | |
| Qualitative Accuracy (150 ng/mL cutoff): Ability to correctly distinguish positive/negative at specified spike levels. | Negative: Known levels of benzoylecognine ≤ 112.5 ng/mL (minus 25% of 150 ng/mL cutoff) were distinguished as negative. |
| Positive: Spiked levels ≥ 187 ng/mL (plus 25% of 150 ng/mL cutoff, up to 3000 ng/mL) were routinely distinguished as positive. | |
| Qualitative Accuracy (300 ng/mL cutoff): Ability to correctly distinguish positive/negative at specified spike levels. | Negative: Known levels of benzoylecognine ≤ 225 ng/mL (minus 25% of 300 ng/mL cutoff) were distinguished as negative. |
| Positive: Spiked levels ≥ 375 ng/mL (plus 25% of 300 ng/mL cutoff, up to 3000 ng/mL) were routinely distinguished as positive. | |
| Semiquantitative Recovery (150/300 ng/mL): Accuracy of reported concentrations within a specified range. | Recovered within 20% of the nominal value for known spiked concentrations between 45 ng/mL and 900 ng/mL. |
| Precision (Qualitative mode): Low variability in repeated measurements. | Acceptable. |
- Within-run %CV for controls and cutoffs (rates) ranged from 0.4% to 0.5%.
- Total precision %CV for controls and cutoffs (rates) ranged from 0.5% to 0.6%. |
| Precision (Semiquantitative mode): Low variability in repeated measurements of concentrations. | Acceptable. - Within-run %CV for controls and cutoffs (concentrations) ranged from 3.7% to 10.9%.
- Total precision %CV for controls and cutoffs (concentrations) ranged from 5.1% to 14.9%. |
| Substantial Equivalence: Overall performance comparable to the predicate device. | The manufacturer concluded the device is "substantially equivalent to the Emit® II Cocaine Metabolite Assay with regard to intended use, assay sample, and overall performance characteristics." This was affirmed by the FDA's 510(k) clearance letter. |
2. Sample Size Used for the Test Set and Data Provenance
-
Sample Size for Test Set:
- For the correlation with GC/MS (150 ng/mL cutoff): The document does not specify the exact number of samples used for the full 100% agreement study. It only mentions that "All positive samples and a portion of negative samples (n=20), as assessed by the Emit® II Plus Cocaine Assay, were analyzed by GC/MS for confirmatory (positive samples) and specificity (negative samples) purposes." This implies at least 20 negative samples plus an unspecified number of positive samples were tested against GC/MS. The number of samples for the 100% agreement claim is not clearly stated.
- For the correlation with the predicate device (300 ng/mL cutoff): The sample size is not explicitly stated, beyond the mention of "three samples were discordant."
- For spiked sample recovery (qualitative and semiquantitative): The number of samples (spiked urine) and replicates is not specified.
- For precision studies: The number of samples/replicates for the precision studies is not specified, only the resulting %CVs.
-
Data Provenance: The document does not provide information on the country of origin of the data or whether the study was retrospective or prospective. Given the nature of in-vitro diagnostic assays for drug testing, it's typically prospective testing of collected urine samples, but this is not explicitly stated.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications of those Experts
Not applicable. This device is an in-vitro diagnostic (IVD) assay for chemical analysis. The "ground truth" is established by a reference chemical method (GC/MS) or by known concentrations in spiked samples, not by expert interpretation of images or clinical findings.
4. Adjudication Method for the Test Set
Not applicable. Adjudication methods (like 2+1, 3+1) are typically used for studies involving human interpretation (e.g., radiologists, pathologists) where discrepancies need to be resolved. For an IVD device, the "adjudication" of results is based on comparison to the reference chemical method (GC/MS) or expected values for spiked samples. Discordant results are typically investigated to understand the reason (e.g., borderline concentration, interference).
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance
Not applicable. This is an in-vitro diagnostic device for chemical analysis. It does not involve human readers interpreting cases or AI assistance in the context of diagnostic imaging.
6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) was done
Yes, the studies described are standalone performance evaluations of the Emit® II Plus Cocaine Metabolite Assay. The device (assay) itself produces quantitative or qualitative results, which are then compared to reference methods. While human operators perform the testing, the results are derived directly from the assay's chemical reactions and detection system, without an "algorithm" in the AI sense or a human performing interpretation of the assay's output that would require a human-in-the-loop study design.
7. The Type of Ground Truth Used
The ground truth used in the studies includes:
- Chemical Reference Method: Gas chromatography/mass spectrometry (GC/MS) for confirmation of drug metabolite presence and concentration in real urine samples. GC/MS is considered the gold standard for drug quantification.
- Known Spiked Concentrations: Urine samples spiked with known, precise concentrations of benzoylecognine for evaluating qualitative accuracy (distinguishing positive/negative at specific cutoffs) and semiquantitative recovery.
8. The Sample Size for the Training Set
The document does not mention a "training set" in the context of machine learning or AI. This is a traditional in-vitro diagnostic assay. The development of such assays involves formulation, optimization, and characterization, but not typically a "training set" in the AI sense.
9. How the Ground Truth for the Training Set Was Established
Not applicable, as there is no "training set" for an AI algorithm described for this device. The assay itself relies on established biochemical principles and reagents, not on learning from data in the way an AI model does.
§ 862.3250 Cocaine and cocaine metabolite test system.
(a)
Identification. A cocaine and cocaine metabolite test system is a device intended to measure cocaine and a cocaine metabolite (benzoylecgonine) in serum, plasma, and urine. Measurements obtained by this device are used in the diagnosis and treatment of cocaine use or overdose.(b)
Classification. Class II (special controls). A cocaine and cocaine metabolite test system is not exempt if it is intended for any use other than employment or insurance testing or is intended for Federal drug testing programs. The device is exempt from the premarket notification procedures in subpart E of part 807 of this chapter subject to the limitations in § 862.9, provided the test system is intended for employment and insurance testing and includes a statement in the labeling that the device is intended solely for use in employment and insurance testing, and does not include devices intended for Federal drug testing programs (e.g., programs run by the Substance Abuse and Mental Health Services Administration (SAMHSA), the Department of Transportation (DOT), and the U.S. military).