(235 days)
The ATTEST Drug Screen Cup and the ATTEST Drug Screen Dip Card are rapid lateral flow immunoassays for the qualitative detection of 6-Acetylmorphine, d-Ampletamine, Benzoylecgonine, EDDP, d/1-Methadone, d-Methamphetamine, d/Methylenedioxymethamphetamine, Nortriptyline, Oxazepam, Oxycodone, Phencyclidine, d-Propoxyphene, Secobarbital and THC-COOH in human urine. The test cut-off concentrations and the compounds the tests are calibrated to are as follows:
| Analyte | Calibrator | Cutoff (ng/mL) |
|---|---|---|
| 6-Acetylmorphine | 6-monoacetylmorphine | 10 |
| Amphetamine | d-Amphetamine | 500 |
| Amphetamine | d-Amphetamine | 1,000 |
| Barbiturates | Secobarbital/Pentobarbital | 300 |
| Benzodiazepines | Oxazepam | 300 |
| Buprenorphine | Buprenorphine | 10 |
| EDDP | 2-ethylidene-1,5-dimethyl-3-3- diphenylpyrrolidine | 300 |
| Cocaine | Benzoylecgonine | 150 |
| Cocaine | Benzoylecgonine | 300 |
| Ecstasy | d,l-Methylenedioxymethamphetamine | 500 |
| Methamphetamine | d-Methamphetamine | 500 |
| Methamphetamine | d-Methamphetamine | 1,000 |
| Marijuana | 11-nor-Δ9-THC-9-COOH | 50 |
| Methadone | d/l-Methadone | 300 |
| Opiates | Morphine | 300 |
| Opiates | Morphine | 2,000 |
| Oxycodone | Oxycodone | 100 |
| Phencyclidine | Phencyclidine | 25 |
| Propoxyphene | Propoxyphene | 300 |
| Tricyclic Antidepressants | Nortriptyline | 1,000 |
The single or multi-test panels can consist of the above listed analytes in any combination, up to a maximum of 16 analytes, with and without on-board adulteration/specimen validity tests (SVT) in the cup format. The drug screen tests are intended for prescription use only.
The tests provide only a preliminary result. A more specific alternative chemical method should be used in order to obtain a confirmed presumptive positive result if the donor doesn't admit use or anytime required by testing procedures. Gas Chromatography / Mass Spectrometry (GC/MS), Liquid Chromatography / Mass Spectrometry (LC/MS) and their tandem mass-spectrometer versions are the preferred confirmatory methods. Careful consideration and judgment should be applied to any drugs of abuse screen test result, particularly when evaluating preliminary positive results.
For professional use, the devices consist of:
- a. 10 or 25 test cups or dip cards with or without adulteration/specimen validity tests
- b. Package insert
- c. Procedure Card
The provided text describes the performance characteristics of the ATTEST Drug Screen Cup and ATTEST Drug Screen Dip Card, which are rapid lateral flow immunoassays for the qualitative detection of various drugs in human urine.
Here's an analysis of the acceptance criteria and the study proving the device meets them:
1. Table of Acceptance Criteria and Reported Device Performance
The acceptance criteria for qualitative drug screening devices are typically based on agreement with a gold-standard confirmatory method (like GC/MS, LC/MS) at and around the cut-off concentrations. The concept of "agreement" translates to sensitivity and specificity for qualitative tests. While explicit numeric acceptance criteria (e.g., "must achieve >95% positive agreement") aren't directly stated as "acceptance criteria" in a separate table, they are implied by the performance statistics presented in the "Method Comparison/Accuracy Studies" section.
The reported device performance is presented in the two large tables under "Method Comparison/Accuracy Studies" (Pages 14-16), showing the agreement of the ATTEST devices (Cup and Dip Card) with GC/MS or LC/MS results.
Let's interpret the tables to deduce the implicit acceptance criteria and the reported performance. The "Agreement" column in these tables represents the percentage of agreement between the device's qualitative result (Negative or Positive) and the quantitative result from the gold-standard method, particularly around the cutoff and at significant concentrations above and below it.
For each drug assay (e.g., 6-AM/10, AMP/500, etc.), the data shows the number of samples that fell into different concentration ranges relative to the cutoff (e.g., "Drug-free", "-50% C/O to +50% C/O"). The "Agreement" percentage is then provided for both Negative and Positive results.
Implicit Acceptance Criteria (Deduced from Reported Performance):
| Drug Test/Cutoff (ng/mL) | Acceptance Criteria (Implicit) | Reported Device Performance (Agreement %) - ATTEST Cup & Dip Card |
|---|---|---|
| General | High agreement (ideally 100% or very close to it) for drug-free samples and samples with concentrations significantly above the cutoff. Very high agreement is also expected for samples significantly below the cutoff that should test negative, and samples significantly above the cutoff that should test positive. | Achieved for all assays as detailed below. |
| Specific to Each Assay | Negative Agreement: Expected to be high for drug-free samples and samples definitively below the cutoff. | |
| Positive Agreement: Expected to be high for samples definitively above the cutoff. | ||
| 6-AM/10 | Expected high agreement for Negatives (samples = 10 ng/mL). | Neg: 100%, Pos: 100% |
| AMP/500 | Expected high agreement for Negatives (samples = 500 ng/mL). | Neg: 97.7%, Pos: 100% |
| AMP/1000 | Expected high agreement for Negatives (samples = 1000 ng/mL). | Neg: 100%, Pos: 100% |
| BAR/300 | Expected high agreement for Negatives (samples = 300 ng/mL). | Neg: 95.2%, Pos: 100% |
| BUP/10 | Expected high agreement for Negatives (samples = 10 ng/mL). | Neg: 95.5%, Pos: 100% |
| BZO/300 | Expected high agreement for Negatives (samples = 300 ng/mL). | Neg: 93.2%, Pos: 100% (This is the lowest negative agreement) |
| COC/150 | Expected high agreement for Negatives (samples = 150 ng/mL). | Neg: 97.70%, Pos: 100% |
| COC/300 | Expected high agreement for Negatives (samples = 300 ng/mL). | Neg: 100%, Pos: 100% |
| EDDP/300 | Expected high agreement for Negatives (samples = 300 ng/mL). | Neg: 93.2%, Pos: 100% (Similar to BZO/300) |
| MDMA/500 | Expected high agreement for Negatives (samples = 500 ng/mL). | Neg: 95.5%, Pos: 100% |
| MET/500 | Expected high agreement for Negatives (samples = 500 ng/mL). | Neg: 93.2%, Pos: 100% (Similar to BZO/300 & EDDP/300) |
| MET/1000 | Expected high agreement for Negatives (samples = 1000 ng/mL). | Neg: 100%, Pos: 100% |
| MTD/300 | Expected high agreement for Negatives (samples = 300 ng/mL). | Neg: 95.5%, Pos: 100% |
| OPI/300 | Expected high agreement for Negatives (samples = 300 ng/mL). | Neg: 97.72%, Pos: 100% |
| OPI/2000 | Expected high agreement for Negatives (samples = 2000 ng/mL). | Neg: 93.2%, Pos: 100% (Similar to BZO/300, EDDP/300, MET/500) |
| OXY/100 | Expected high agreement for Negatives (samples = 100 ng/mL). | Neg: 93.2%, Pos: 100% (Similar to OPI/2000) |
| PCP/25 | Expected high agreement for Negatives (samples = 25 ng/mL). | Neg: 97.7%, Pos: 100% |
| PPX/300 | Expected high agreement for Negatives (samples = 300 ng/mL). | Neg: 95.3%, Pos: 100% |
| TCA/1000 | Expected high agreement for Negatives (samples = 1000 ng/mL). | Neg: 95.5%, Pos: 100% |
| THC/50 | Expected high agreement for Negatives (samples = 50 ng/mL). | Neg: 97.7%, Pos: 100% |
Note: The performance percentages across the two formats (Cup and Dip Card) are presented identically in the provided combined tables for all assays, suggesting the performance data is pooled or very similar for both. The discordant results tables (pages 15-17) provide a more granular view for specific instances where the device result differed from the GC/MS or LC/MS result, primarily for samples near the cutoff. For example, for AMP/500, a 97.7% negative agreement means 2.3% of negative samples were discordant (likely false negative or near-cutoff negative that read positive by device). For Amp/500, 3 "Neg" results were in the "-50% C/O to +25% C/O to +50% C/O, >+50% C/O).
* The "Summary of Discordant Results" tables (pages 15-17) list specific instances of samples where the device's qualitative result did not match the quantitative GC/MS/LC/MS result (e.g., "Positive" device result for a sample with a drug concentration slightly below the cutoff, or "Negative" for a sample slightly above). These are qualitative examples, not the full dataset.
- Data Provenance: The studies used clinical urine specimens "previously quantitated for the target drugs of abuse by gold-standard methods (GC/MS, LC/MS or equivalent)." The text does not specify the country of origin of the data or whether the samples were retrospective or prospective collections. However, given the nature of a 510(k) submission for a device to be marketed in the US, it is generally assumed that the clinical studies are relevant to the US population.
3. Number of Experts Used to Establish Ground Truth for the Test Set and Qualifications
This type of device (lateral flow immunoassay for drug screening) does not rely on human experts for establishing ground truth for the test set. The ground truth is established by analytical methods (Gas Chromatography / Mass Spectrometry (GC/MS), Liquid Chromatography / Mass Spectrometry (LC/MS) and their tandem mass-spectrometer versions), which are considered the preferred confirmatory methods for drug testing. Therefore, there were no human "experts" in the sense of radiologists interpreting images to establish medical ground truth.
4. Adjudication Method for the Test Set
Not applicable. Since the ground truth is established by objective analytical methods (GC/MS, LC/MS), there's no need for an adjudication process involving human readers or experts. The assay results are compared directly against the quantitative chemical analysis.
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was Done
No, an MRMC comparative effectiveness study was not done. The device is a rapid diagnostic test for drug detection, not an AI-powered image analysis tool for human interpretation. The study is a comparison of the device's performance against a gold-standard chemical analysis method, not a study of human readers. Therefore, there is no effect size related to human readers improving with AI assistance.
6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) was done
Yes, the study described is essentially a standalone (algorithm/device-only) performance evaluation. The device's qualitative results are directly compared to the quantitative results from GC/MS/LC/MS, without human interpretation as an intermediate step. The "intended professional users" mentioned in Section M.a.iv (Detection Limit/sensitivity) are those who read the test result (presence/absence of a line), not those who interpret complex data as an AI model would.
7. The Type of Ground Truth Used
The ground truth used for the test set was analytical confirmation by gold-standard chemical methods: Gas Chromatography / Mass Spectrometry (GC/MS), Liquid Chromatography / Mass Spectrometry (LC/MS), or equivalent. This is explicitly stated in "Method Comparison/Accuracy Studies" (Page 14).
8. The Sample Size for the Training Set
The document describes a 510(k) submission for a diagnostic device. Unlike AI/ML models, these traditional diagnostic devices do not typically have a "training set" in the sense of a data set used to train an algorithm. The device's formulation and antibodies are developed through a research and development process, and its performance is then validated (tested) using various samples.
The document mentions "The precision, reproducibility and sensitivity of the ATTEST Drug Screen Cup and the ATTEST Drug Screen Dip Card were evaluated at Advin Biotech and at three (3) external testing sites. Data obtained at all testing sites across the intended use populations indicate >99% correlation at +/-50% of each assay cutoff." (Page 8-9). This suggests internal development and testing, but not explicitly a "training set" as understood in machine learning. The stability and specificity studies also involve testing with numerous samples.
9. How the Ground Truth for the Training Set was Established
As there is no "training set" for this type of device in the AI sense, this question is not directly applicable. The device's underlying chemistry and design are based on established immunochromatographic principles. The process involves:
- Standardization: The assays are harmonized with DHHS/SAMHSA screening cutoffs (Page 8-9).
- Analytical Specificity: This is established by testing the device with contrived solutions of chemically related/structurally similar compounds and other potential interferents (Pages 9-13). This involves verifying that the device reacts appropriately to the target analyte and does not cross-react significantly with other substances.
- Precision/Reproducibility: Evaluated through repeated testing at various concentrations and across different sites (Page 8).
These evaluations contribute to the overall confidence in the device's performance and design, akin to "establishing ground truth" for its analytical properties, but it's fundamentally different from training an AI model on a labeled dataset.
§ 862.3650 Opiate test system.
(a)
Identification. An opiate test system is a device intended to measure any of the addictive narcotic pain-relieving opiate drugs in blood, serum, urine, gastric contents, and saliva. An opiate is any natural or synthetic drug that has morphine-like pharmocological actions. The opiates include drugs such as morphine, morphine glucoronide, heroin, codeine, nalorphine, and meperedine. Measurements obtained by this device are used in the diagnosis and treatment of opiate use or overdose and in monitoring the levels of opiate administration to ensure appropriate therapy.(b)
Classification. Class II (special controls). An opiate test system is not exempt if it is intended for any use other than employment or insurance testing or is intended for Federal drug testing programs. The device is exempt from the premarket notification procedures in subpart E of part 807 of this chapter subject to the limitations in § 862.9, provided the test system is intended for employment and insurance testing and includes a statement in the labeling that the device is intended solely for use in employment and insurance testing, and does not include devices intended for Federal drug testing programs (e.g., programs run by the Substance Abuse and Mental Health Services Administration (SAMHSA), the Department of Transportation (DOT), and the U.S. military).