(47 days)
First Sign™ Drug of Abuse Tests are immunochromatographic assays for the qualitative determination of Amphetamine (d-amphetamine), Cocaine (Benzoylecgonine), and Marijuana (11-nor-A9-THC-9-COOH ) in human urine at cut-off concentrations of 1000 ng/mL, and 50 ng/mL, respectively. The tests are available in a Cup format and a Dip Card format.
The tests provide only preliminary test results. A more specific alternative chemical method must be used in order to obtain a confirmed analytical result. GC/MS is the preferred confirmatory method. Clinical consideration and professional iudgment should be exercised with any drug of abuse test result, particularly when the preliminary result is possive. For in vitro diagnostic use only. The tests are intended for over-the-counter and for prescription use.
First Sign™ Drug of Abuse Tests are immunochromatographic assays. Each assay test is a lateral flow system for the qualitative detection of cocaine, amphetamine, and marijuana in human urine. The products are single-use in vitro diagnostic devices, which come in the formats of DipCards or Cups. Each test kit contains a Test Device (in one of the two formats), a package insert and a urine cup for sample collection. Each test device is sealed with a desiccant in an aluminum pouch.
This document describes the performance characteristics of the First Sign™ Drug of Abuse Dip Card Test and First Sign™ Drug of Abuse Cup Test for Amphetamine, Cocaine, and Marijuana.
Here's an analysis based on your request:
1. Table of Acceptance Criteria and Reported Device Performance
The document does not explicitly state "acceptance criteria" as a set of predefined thresholds. However, the performance is demonstrated through various studies. The primary measure of performance is the ability of the device to correctly identify positive and negative samples relative to a defined cut-off, as confirmed by GC/MS.
For the precision studies, the acceptance criterion implicitly appears to be 100% agreement for samples significantly below (-50%, -75%, -100% cut-off) and significantly above (+50%, +75%, +100% cut-off) the cut-off. For samples near the cut-off, some discordance is expected.
For the comparison studies (against GC/MS), the performance is presented through the number of correct classifications. For the lay-user study, the performance is given as the "percentage of correct results (%)".
Below is a summary of the reported device performance for samples at the cut-off and at +25% and -25% of the cut-off, as these are the most critical regions for accuracy.
Amphetamine (AMP) - Cut-off: 1000 ng/mL (all values are percentage of samples correctly classified)
| Device Type | Assay | % Correct at -25% Cut-off (750 ng/mL) | % Correct at Cut-off (1000 ng/mL) | % Correct at +25% Cut-off (1250 ng/mL) |
|---|---|---|---|---|
| Dip Card (Precision) | AMP | 100% Negative (150/150) | 94% Positive (140/150) | 100% Positive (150/150) |
| Cup (Precision) | AMP | 100% Negative (150/150) | 96% Positive (144/150) | 100% Positive (150/150) |
| Dip Card (Comparison) | AMP | Varies by viewer (e.g., Viewer A: 19/20 negative) | Varies by viewer (e.g., Viewer A: 4/14 negative, 10/14 positive) | Varies by viewer (e.g., Viewer A: 26/26 positive, 4/4 negative) |
| Cup (Comparison) | AMP | Varies by viewer (e.g., Viewer A: 19/20 negative) | Varies by viewer (e.g., Viewer A: 1/14 negative, 13/14 positive) | Varies by viewer (e.g., Viewer A: 26/26 positive, 1/1 negative) |
| Lay-User (Dip Card) | AMP | 95% Negative (19/20) | Not explicitly reported at cut-off | 90% Positive (18/20) |
| Lay-User (Cup) | AMP | 90% Negative (18/20) | Not explicitly reported at cut-off | 95% Positive (19/20) |
Cocaine (COC) - Cut-off: 300 ng/mL
| Device Type | Assay | % Correct at -25% Cut-off (225 ng/mL) | % Correct at Cut-off (300 ng/mL) | % Correct at +25% Cut-off (375 ng/mL) |
|---|---|---|---|---|
| Dip Card (Precision) | COC | 100% Negative (150/150) | 95% Positive (142/150) | 100% Positive (150/150) |
| Cup (Precision) | COC | 100% Negative (150/150) | 96% Positive (145/150) | 100% Positive (150/150) |
| Dip Card (Comparison) | COC | Varies by viewer (e.g., Viewer A: 19/20 negative) | Varies by viewer (e.g., Viewer A: 4/14 negative, 10/14 positive) | Varies by viewer (e.g., Viewer A: 26/26 positive, 4/4 negative) |
| Cup (Comparison) | COC | Varies by viewer (e.g., Viewer A: 19/20 negative) | Varies by viewer (e.g., Viewer A: 2/14 negative, 12/14 positive) | Varies by viewer (e.g., Viewer A: 26/26 positive, 2/2 negative) |
| Lay-User (Dip Card) | COC | 85% Negative (17/20) | Not explicitly reported at cut-off | 95% Positive (19/20) |
| Lay-User (Cup) | COC | 90% Negative (18/20) | Not explicitly reported at cut-off | 100% Positive (20/20) |
Marijuana (THC) - Cut-off: 50 ng/mL
| Device Type | Assay | % Correct at -25% Cut-off (37.5 ng/mL) | % Correct at Cut-off (50 ng/mL) | % Correct at +25% Cut-off (62.5 ng/mL) |
|---|---|---|---|---|
| Dip Card (Precision) | THC | 100% Negative (150/150) | 95% Positive (144/150) | 100% Positive (150/150) |
| Cup (Precision) | THC | 100% Negative (150/150) | 96% Positive (146/150) | 100% Positive (150/150) |
| Dip Card (Comparison) | THC | Varies by viewer (e.g., Viewer A: 18/20 negative) | Not applicable, samples categorized as "Near Cutoff Positive" | 100% Positive (26/26) |
| Cup (Comparison) | THC | Varies by viewer (e.g., Viewer A: 19/20 negative) | Not applicable, samples categorized as "Near Cutoff Positive" | 100% Positive (26/26) |
| Lay-User (Dip Card) | THC | 95% Negative (19/20) | Not explicitly reported at cut-off | 100% Positive (20/20) |
| Lay-User (Cup) | THC | 95% Negative (19/20) | Not explicitly reported at cut-off | 95% Positive (19/20) |
Note: For the precision and lay-user studies, performance at the exact cut-off received mixed results (some negative, some positive), indicating the inherent variability near the detection limit. For the comparison studies, samples were categorized into "Near Cutoff Negative" (between -50% and cut-off) and "Near Cutoff Positive" (between cut-off and +50%), not specifically at the cut-off. Performance percentages for comparison studies are not directly provided in the text and would require calculation from the tables.
2. Sample Size and Data Provenance for Test Set
-
Precision Studies:
- For each drug (AMP, COC, THC) and each format (Dip Card, Cup), 9 different concentrations were tested: -100%, -75%, -50%, -25%, Cut-off, +25%, +50%, +75%, +100% of the cut-off.
- Each concentration was tested two runs per day for 25 days, using three different lots of the device. This means for each drug/format/concentration, there were 50 tests per lot, totaling 150 tests per concentration. (e.g., 50-/0+ means 50 negative results, 0 positive results).
- Total tests for precision: 3 drugs * 2 formats * 9 concentrations * 150 tests/concentration = 8100 tests.
- Data Provenance: "Samples were prepared by spiking drug in negative samples." "Each drug concentration was confirmed by GC/MS." This suggests prospective, laboratory-controlled data using spiked urine samples. The country of origin is not specified but implied to be the location of the manufacturing company (W.H.P.M., Inc., Irwindale, CA).
-
Cut-off Verification Study:
- A total of 150 samples equally distributed at -50% cut-off; -25% cut-off; cut-off; +25% cut-off; +50% cut-off were tested.
- This implies 5 concentrations * 30 samples/concentration = 150 samples.
- Data Provenance: Not explicitly stated but likely lab-prepared spiked samples similar to precision studies.
-
Comparison Studies (Method Comparison):
- 80 clinical samples (40 negative and 40 positive) were used for each drug (AMP, COC, THC) and each format (Cup, Dip Card).
- Total samples for comparison studies: 3 drugs * 2 formats * 80 samples/drug/format = 480 clinical samples.
- Data Provenance: "Un-altered clinical samples." This suggests retrospective or prospectively collected clinical samples, but without further detail on how they were collected. The country of origin is not specified but implied to be the location of the study (in-house).
-
Lay-user Study:
- 280 lay persons tested the Amphetamine devices.
- 280 lay persons tested the Cocaine devices.
- 280 lay persons tested the Marijuana devices.
- Each participant was given 1 blind labeled sample. These samples were prepared at 7 different concentrations: negative, +/-25%, +/-50%, +/-75%, +/-100% of the cut-off. This suggests that for each drug, 7 concentrations * 20 samples/concentration = 140 samples were prepared. Given 280 lay persons, it implies each person tested one sample, and samples were duplicated for the number of participants.
- Data Provenance: "Urine samples were prepared at the following concentrations; negative, +/-75%, +/-25% of the cutoff by spiking drugs into drug free-pooled urine specimens. The concentrations of the samples were confirmed by GC/MS." This is prospective, laboratory-prepared spiked urine samples, presented as blind-labeled to the lay users. Country of origin not specified, but study performed "at three intended user sites".
3. Number of Experts and Qualifications for Ground Truth
- Precision, Cut-off, Interference, Specificity, and Effect of Urine Specific Gravity and pH Studies: The ground truth was established by GC/MS (Gas Chromatography/Mass Spectrometry). GC/MS is a laboratory analytical method and does not involve human experts in establishing the "ground truth" concentration of the drugs. These studies were performed by laboratory personnel.
- Comparison Studies (Method Comparison): The ground truth for the 480 clinical samples was also established by GC/MS. These were "compared to GC/MS results," meaning GC/MS served as the reference method (ground truth). The tests themselves were run by "three different laboratory assistants." Their qualifications beyond being "laboratory assistants" are not specified.
- Lay-user Study: The ground truth for the spiked samples was established by GC/MS.
4. Adjudication Method for the Test Set
- Precision, Cut-off, Interference, Specificity, and Effect of Urine Specific Gravity and pH Studies: The data presented are direct results from the devices (negative/positive) for specific concentrations as confirmed by GC/MS. There is no mention of an adjudication method as the results are quantitative and compared against a defined cut-off.
- Comparison Studies: The tests were run by "three different laboratory assistants" (Viewers A, B, C). Each viewer independently provided results (Positive/Negative). The tables present results for each viewer separately. There is no explicit adjudication method (like rule-based consensus, e.g., 2+1, 3+1). Discordant results are noted for individual viewers without a resolution process described for a single, unified "device result."
- Lay-user Study: Each lay person provided individual results. The data aggregates these individual results. There is no adjudication method described for resolving differences between lay users.
5. Multi Reader Multi Case (MRMC) Comparative Effectiveness Study
No. The document presents neither an MRMC study design nor a comparative effectiveness study evaluating human readers with and without AI assistance. The studies involve (1) laboratory personnel interpreting the device, and (2) lay users interpreting the device. There is no AI component mentioned.
6. Standalone (Algorithm Only) Performance
Yes, in essence, the "Analytical Performance" section (Precision, Interference, Specificity, Cut-off, Stability, Effect of Urine Specific Gravity and pH) and the "Comparison Studies" with laboratory assistants can be considered standalone performance of the device without explicit human-in-the-loop assistance influencing the device's detection capabilities. The human element here is about reading and interpreting the visual line on the test, not assisting an underlying algorithm. The device itself is an immunochromatographic assay, which is a chemical/biological-based test, not an AI algorithm.
7. Type of Ground Truth Used
The type of ground truth used across all reported studies (Precision, Cut-off, Comparison, Lay-user) is GC/MS (Gas Chromatography/Mass Spectrometry). This is a laboratory-based, highly accurate chemical method considered the gold standard for drug confirmation in urine. All samples (spiked or clinical) had their drug concentrations confirmed using GC/MS.
8. Sample Size for the Training Set
This document describes a medical device which is an immunochromatographic assay, not an AI/ML-based device. Therefore, there is no "training set" in the context of machine learning algorithms. The performance is based on the chemical reactions and design of the lateral flow assay.
9. How the Ground Truth for the Training Set Was Established
As there is no "training set" for an AI/ML algorithm mentioned, this question is not applicable. The device operates based on antigen-antibody immunochemistry, not learned patterns from data.
§ 862.3250 Cocaine and cocaine metabolite test system.
(a)
Identification. A cocaine and cocaine metabolite test system is a device intended to measure cocaine and a cocaine metabolite (benzoylecgonine) in serum, plasma, and urine. Measurements obtained by this device are used in the diagnosis and treatment of cocaine use or overdose.(b)
Classification. Class II (special controls). A cocaine and cocaine metabolite test system is not exempt if it is intended for any use other than employment or insurance testing or is intended for Federal drug testing programs. The device is exempt from the premarket notification procedures in subpart E of part 807 of this chapter subject to the limitations in § 862.9, provided the test system is intended for employment and insurance testing and includes a statement in the labeling that the device is intended solely for use in employment and insurance testing, and does not include devices intended for Federal drug testing programs (e.g., programs run by the Substance Abuse and Mental Health Services Administration (SAMHSA), the Department of Transportation (DOT), and the U.S. military).