(31 days)
First Sign™ Drug of Abuse Tests are immunochromatographic assays for the qualitative determination of Methadone, Phencyclidine, and Oxycodone in human urine at cut-off concentrations of 300 ng/mL, 25 ng/mL, and 100 ng/mL, respectively. The tests are available in a Cup format and a Dip Card format.
The tests may yield preliminary positive results even when prescription drugs Methadone and Oxycodone are ingested, at prescribed doses; it is not intended to distinguish between prescription use or abuse of this drug. There are no uniformly recognized cutoff concentration levels for Methadone and Oxycodone in urine. The tests provide only preliminary test results. A more specific alternative chemical method must be used in order to obtain a confirmed analytical result. Gas Chromatography/Mass Spectrometry is the preferred confirmatory method. Clinical consideration and professional judgment should be exercised with any drug of abuse test result, particularly when the preliminary result is positive.
For in vitro diagnostic use only. The tests are intended for over-the-counter and for prescription use.
First Sign™ Drug of Abuse Tests are immunochromatographic assays. Each assay test is a lateral flow system for the qualitative detection of Methadone, Phencyclidine, and Oxycodone in human urine. The products are single-use in vitro diagnostic devices, which come in the formats of DipCards or Cups. Each test kit contains a Test Device (in one of the two formats), a package insert and a urine cup for sample collection. Each test device is sealed with a desiccant in an aluminum pouch.
The provided text describes the performance characteristics and studies for the "First Sign® Drug of Abuse Cup Test" and "First Sign® Drug of Abuse Dip Card Test" for Methadone, Phencyclidine, and Oxycodone.
The acceptance criteria for each drug are implicitly defined by the reported performance, specifically in the precision, cut-off verification, and comparison studies. For instance, in the precision study, at -100% to -25% of the cut-off, all results were expected to be negative, and at +25% to +100% of the cut-off, all results were expected to be positive, with some allowance for variation at the exact cut-off concentration.
Here's a breakdown of the requested information:
1. A table of acceptance criteria and the reported device performance
The document does not explicitly state acceptance criteria in a separate table. However, the performance tables demonstrate the device's adherence to the expected behavior around the cut-off concentrations. The "Precision" section's implied criteria are 100% correct negatives for concentrations below -25% of the cut-off, 100% correct positives for concentrations above +25% of the cut-off, and a high percentage of correct results at +/-25% and at the cut-off values. The "Cut-off" section verifies the stated cut-off values.
Table: Acceptance Criteria (Implied from Precision and Cut-off Studies) and Reported Device Performance
| Drug | Concentration Range | Implied Acceptance: Expected Result | Reported Performance (Precision Study: Examples from Lot 1/3) |
|---|---|---|---|
| Methadone (Cut-off: 300 ng/mL) | -100% to -25% Cut-off (0-225 ng/mL) | Negative | 50-/0+ (100% Negative) |
| Cut-off (300 ng/mL) | Mix of Positive/Negative | 3-/47+ or 1-/49+ (mostly Positive) | |
| +25% to +100% Cut-off (375-600 ng/mL) | Positive | 50+/0- (100% Positive) | |
| Phencyclidine (Cut-off: 25 ng/mL) | -100% to -25% Cut-off (0-18.75 ng/mL) | Negative | 50-/0+ (100% Negative) |
| Cut-off (25 ng/mL) | Mix of Positive/Negative | 3-/47+ or 2-/48+ (mostly Positive) | |
| +25% to +100% Cut-off (31.25-50 ng/mL) | Positive | 50+/0- (100% Positive) | |
| Oxycodone (Cut-off: 100 ng/mL) | -100% to -25% Cut-off (0-75 ng/mL) | Negative | 50-/0+ (100% Negative) |
| Cut-off (100 ng/mL) | Mix of Positive/Negative | 4-/46+ or 3-/47+ (mostly Positive) | |
| +25% to +100% Cut-off (125-200 ng/mL) | Positive | 50+/0- (100% Positive) |
2. Sample sized used for the test set and the data provenance (e.g. country of origin of the data, retrospective or prospective)
- Precision Study:
- Sample Size: For each drug and each format (Dip Card/Cup), 8 concentrations were tested across 3 lots, with samples run 2 times per day for 25 days by 3 different operators. This is not a simple count of unique clinical samples. It uses spiked samples. Each individual concentration point (e.g., -100% cut-off) had 50 measurements (2 tests/day * 25 days by each of 3 lots, but the values are given as 50-/0+ or 50+/0- for each lot, suggesting 50 replicates per lot at each concentration).
- Data Provenance: The samples were "prepared by spiking drug in negative samples." The document does not specify the country of origin, but the testing was conducted "in-house." These are prospective, laboratory-prepared samples.
- Cut-off Verification Study:
- Sample Size: 150 samples ("equally distributed at concentrations of -50% cut-off; cut-off; +25% cut-off; +50% cut-off"). These were tested using three different lots of each device by three different operators.
- Data Provenance: Similar to precision study, these were laboratory-prepared samples ("spiked drug in negative samples"). No country of origin is specified.
- Comparison Studies (Clinical Samples):
- Sample Size: 80 "unaltered clinical samples" for each drug (40 negative and 40 positive). This means 80 samples for Methadone, 80 for Phencyclidine, and 80 for Oxycodone, for both Dip Card and Cup formats (though the tables suggest the same samples were likely used for both formats for a given viewer).
- Data Provenance: "in-house" studies, using "unaltered clinical samples." The origin of these clinical samples is not specified (e.g., country, retrospective/prospective).
- Lay-user Study:
- Sample Size: 280 lay persons for Methadone, 280 for Phencyclidine, and 280 for Oxycodone devices. Each participant received 1 blind-labeled sample. The samples themselves were prepared at 7 different concentrations, with 20 samples per concentration. This means a total of 140 samples for each drug (7 concentrations * 20 samples/concentration) were tested by the lay-users.
- Data Provenance: "Urine samples were prepared... by spiking drugs into drug free-pooled urine specimens." The study was performed "at three intended user sites." No country of origin is specified. These are prospective, laboratory-prepared samples.
3. Number of experts used to establish the ground truth for the test set and the qualifications of those experts (e.g. radiologist with 10 years of experience)
- Precision, Cut-off, and Lay-user Studies (Spiked Samples): The ground truth was established by the precise spiking of drugs into negative urine samples at known concentrations, confirmed by GC/MS (Gas Chromatography/Mass Spectrometry). This is an analytical chemistry method, and implicitly, the expertise lies in the laboratory personnel conducting these precise preparations and GC/MS confirmations. No further expert qualifications are provided.
- Comparison Studies (Clinical Samples): The ground truth was established by GC/MS results for each of the 80 unaltered clinical samples per drug. No details on the number or qualifications of the GC/MS experts are provided, but GC/MS is the "preferred confirmatory method" for drug of abuse tests.
4. Adjudication method (e.g. 2+1, 3+1, none) for the test set
- Precision Study: "All sample aliquots were blind-labeled and randomized by the person who prepared samples and did not take part in the sample testing." Tests were performed by "three different operators." The results are aggregated, but no specific adjudication method among operators is explicitly described. It appears each operator's results were simply recorded.
- Cut-off Verification Study: Tested by "three different operators." Similar to precision study, results are aggregated, no specific adjudication is mentioned.
- Comparison Studies: "three different laboratory assistants" ("Viewer A, B, C") tested the samples. The tables show individual viewer results, followed by a list of "Discordant Results." This implies that there wasn't a formal adjudication method (like 2+1 or 3+1) in place to yield a single "device" result, but rather a comparison of each viewer's interpretation against the GC/MS ground truth.
- Lay-user Study: Each "participant was provided with the package insert, 1 blind labeled sample and a device." The results appear to be individual lay person interpretations compared to the GC/MS ground truth, with no inter-lay user adjudication.
5. If a multi reader multi case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance
- No, an MRMC comparative effectiveness study involving AI assistance was not done.
- The studies involved human readers (laboratory assistants, lay-users) interpreting the device results, and these devices are immunochromatographic assays (lateral flow tests), not AI software. Therefore, there is no "AI vs without AI assistance" component to these studies.
6. If a standalone (i.e. algorithm only without human-in-the loop performance) was done
- No, a standalone algorithm-only performance study was not done. The devices are physical tests interpreted by humans (either trained laboratory personnel or lay-users). They do not involve an algorithm separate from human interpretation.
7. The type of ground truth used (expert concensus, pathology, outcomes data, etc)
- For all studies (Precision, Cut-off, Comparison, Lay-user), the ground truth was established by Gas Chromatography/Mass Spectrometry (GC/MS) of the spiked or clinical urine samples. GC/MS is a highly accurate analytical method considered the gold standard for confirming drug concentrations.
8. The sample size for the training set
- The document implies that these studies (precision, cut-off, comparison, lay-user) represent the validation of the device. It does not mention a separate "training set" in the context of machine learning or algorithm development, as this medical device is an immunoassay, not an AI/ML product. The development process for such a test would involve internal R&D and optimization, but the specific term "training set" as used in AI/ML is not applicable here.
9. How the ground truth for the training set was established
- As a traditional immunoassay device, there is no "training set" in the AI/ML sense. The ground truth for the analytical and clinical performance studies was established using GC/MS of urine samples with known (spiked) or confirmed (clinical) drug concentrations.
§ 862.3620 Methadone test system.
(a)
Identification. A methadone test system is a device intended to measure methadone, an addictive narcotic pain-relieving drug, in serum and urine. Measurements obtained by this device are used in the diagnosis and treatment of methadone use or overdose and to determine compliance with regulations in methadone maintenance treatment.(b)
Classification. Class II (special controls). A methadone test system is not exempt if it is intended for any use other than employment or insurance testing or is intended for Federal drug testing programs. The device is exempt from the premarket notification procedures in subpart E of part 807 of this chapter subject to the limitations in § 862.9, provided the test system is intended for employment and insurance testing and includes a statement in the labeling that the device is intended solely for use in employment and insurance testing, and does not include devices intended for Federal drug testing programs (e.g., programs run by the Substance Abuse and Mental Health Services Administration (SAMHSA), the Department of Transportation (DOT), and the U.S. military).