(486 days)
reSET is intended to provide cognitive behavioral therapy, as an adjunct to a contingency management system, for patients 18 years of age and older who are currently enrolled in outpatient treatment under the supervision of a clinician. reSET is indicated as a 12 week (90 days) prescription-only treatment for patients with substance use disorder (SUD), who are not currently on opioid replacement therapy, who do not abuse alcohol solely, or who do not abuse opioids as their primary substance of abuse. It is intended to:
- increase abstinence from a patient's substances of abuse during treatment, and ●
- increase retention in the outpatient treatment program.
reSET™ is a digital therapy comprised of a patient application and clinician dashboard intended to deliver cognitive behavioral therapy (CBT) to patients with SUD to increase abstinence from substance use and increase retention in outpatient therapy programs. CBT is a psychosocial intervention that aims to change a patient's thinking and behavior, and it has been studied in psychiatric disorders such as major depressive disorder (Psychiatr Clin North Am. 2010; 33: 537-55). reSET is based on a specialized version of CBT known as the community reinforcement approach (CRA). which was originally developed for alcohol dependence and cocaine use (Behav Res Ther. 1973; 11:91-104; Exp. Clin. Psychopharmacol 2000; 3:205–2). The community reinforcement incorporates a range of therapeutic modalities including CBT to make substance-free lifestyle rewarding, skill building to promote behavioral change, and contingency management to reward and incentivize abstinence and replace the satisfaction obtained from substance abuse. CBT and CRA are considered valid models for substance abuse therapy and other psychiatric disorders.
reSET consists of several therapy lessons (modules) that are intended to teach the user the following skills to aid in the treatment of substance use disorder:
- Identifying situations and triggers that make substance use more likely ●
- . Avoiding substance use.
- . Coping with thoughts about substance use,
- . Recognizing negative thinking and identifying techniques to move to positive thinking
- Making decisions about substance use
- Taking responsibility for choices made and evaluating the consequences of those choices ●
Each therapy lesson is comprised of a cognitive behavioral therapy component and skill building exercises related to the above areas. The content of the therapy lessons is delivered primarily via text, and may include videos, animations and graphics. All text and video within reSET is narrated in English.
Following most therapy lessons, the patient undergoes fluency learning, a method of questioning that intended to promote learning and improve both short-term and long-term retention of material. Within the fluency learning section of reSET, patients are asked between 4-10 multiple choice and fill-in-the-blank questions about the key concepts presented in the lesson. An example of screenshots from the device is provided in the figure below.
When a question is answered incorrectly, the patient is presented with the correct response and the question is recycled back into the queue and asked again. In order to successfully complete the fluency learning section, patients must answer each question correctly three times, providing a repetitive component that reinforces concept mastery. If the patient completes the lesson and demonstrates proficiency, they can "spin the wheel" for virtual rewards that may be incorporated into a contingency management program at the treating physician's clinic.
reSET is comprised of 62 lessons, including one on-screen User Guide that explains how to use the reSET app and 61 therapy lessons. When a patient uses reSET for the very first time, he or she must step through the User Guide session to ensure they understand how to use the app. The 61 therapy lessons are split into 31 core therapy lessons and 30 supplemental therapy lessons. The therapy lessons include categories related to life skills, treatment, mood matters, social connections, sexual health, and hepatitis C and HIV.
The therapy lessons in the core therapy lesson group are focused on building basic cognitive behavioral and relapse prevention skills (e.g., functional analysis of drug use and selfmanagement planning, drug refusal skills). The therapy lessons in the supplemental group cover a range of topics that can be relevant for patients with SUD such as managing relationships, building communication skills, and time management. They also provide more in-depth training on HIV, hepatitis and STI prevention as well as support for those patients living with HIV and Hepatitis C.
Once the initial User Guide lesson has been completed, the patient gains access to the next core therapy lesson. In the core therapy lessons, a patient can only advance to the next lesson after successfully completing the prior lesson. At any time, patients can choose to review a completed lesson. Once a patient successfully completes all the core therapy lessons, they gain access to all the supplemental therapy lessons. These lessons do not have a set order of completion, and patients can choose lessons that are relevant to managing their disease or as recommended by their clinician.
reSET recommends that patients should complete 4 lessons per week. Each lesson is intended to take between 10-20 minutes to complete. Therapy lesson lengths vary, as do the number of fluency assessment questions the patient must take at the end of a lesson. Some therapy lessons have optional worksheets for the patient to complete that are intended to help the patient understand the key concepts taught in the therapy lesson.
The reSET application allows patients to track their own progress on the device's therapy modules. The device additionally has a Patient Self Report interface that allows for patients to track their cravings and substance use. This information is available to the patient's treating physician through the device.
The clinician dashboard for reSET displays the patient's progress. The clinician can view which therapy lessons the patient has completed, as well as view patient-reported substance use, cravings and triggers. The reSET app automatically pushes an assessment every four days to ask if the patient has taken any drugs or alcohol in the past 4 days, and if so, on which days. The patient is also asked to report whether they have had any cravings for drugs and alcohol and if so, to rate the intensity of the cravings. This reSET initiated, self-report data will display on the clinician's dashboard. Patients may also log their drug and alcohol use at any time into reSET along with tracking their cravings and distinct triggers. These use, craving and trigger data are presented to the clinician. The clinician can also enter in-clinic data inputs such as urine drug screens and appointment attendance.
reSET is intended be used in conjunction with a contingency management incentives system. reSET provides virtual "rewards" when the patient completes a lesson successfully as well as when their urine drug screen, or other objective test, is negative for substances. Clinics mav convert the virtual "rewards" into tangible rewards according to their own procedures.
The reSET device is a software-based medical device, and its acceptance criteria and clinical performance are based on a randomized controlled trial. Here's a breakdown of the requested information:
1. A table of acceptance criteria and the reported device performance
The acceptance criteria for reSET, a computerized behavioral therapy device for psychiatric disorders, were implicitly defined by the statistically significant improvements shown in the clinical trial. The device was deemed acceptable if it demonstrated a benefit over "treatment as usual" (TAU) in its primary outcome measures, especially in cohorts relevant to its indicated use.
| Acceptance Criterion (Implicit) | Reported Device Performance (rTAU + reSET vs. TAU) | Met? |
|---|---|---|
| Increased Abstinence at Weeks 9-12 (Overall) | Cohort 1 (All Comers): 29.7% vs. 16.0% (p=0.0076, Odds Ratio=2.22) | |
| Cohort 2 (Excluding Primary Opioids): 40.3% vs. 17.6% (p=0.0004, Odds Ratio=3.17) | ||
| Cohort 3 (Excluding All Opioids): 38.5% vs. 17.5% (p=0.0034, Odds Ratio=2.95) | Yes | |
| Increased Retention in Outpatient Therapy (Time to Drop-out) | Cohort 1 (All Comers): 27.8% dropouts vs. 36.5% dropouts (p=0.0316) | |
| Cohort 2 (Excluding Primary Opioids): 23.8% dropouts vs. 36.8% dropouts (p=0.0042) | ||
| Cohort 3 (Excluding All Opioids): 25.0% dropouts vs. 38.6% dropouts (p=0.0113) | Yes | |
| No increase in device-related adverse events | 11.5% in TAU arm vs. 14.5% in rTAU + reSET arm (p=0.3563). None adjudicated as device-related. | Yes |
| Software meets regulatory standards | Software verification and validation testing demonstrated that the device met its design, implementation, and cybersecurity requirements, consistent with a "Moderate" level of software concern. Hazard analysis also performed. | Yes |
| Labeling meets regulatory requirements | Physician and patient labeling includes instructions for use, images, compatible devices, warnings about standalone use, substitute for medication, and summary of clinical testing (for physicians). | Yes |
| Based on a validated behavioral therapy model | reSET is based on a specialized version of CBT known as the community reinforcement approach (CRA), which has been studied for alcohol dependence and cocaine use and is considered a valid model for substance abuse therapy. Clinical data further validated its implementation. | Yes |
2. Sample size used for the test set and the data provenance
- Test Set (Clinical Study):
- Cohort 1 (All Comers): 507 participants (252 in TAU, 255 in rTAU + reSET)
- Cohort 2 (Excluding Primary Opioids): 399 participants (193 in TAU, 206 in rTAU + reSET)
- Cohort 3 (Excluding All Opioids): 305 participants (153 in TAU, 152 in rTAU + reSET)
- Data Provenance: The study was a multi-site, randomized clinical trial (National Institute of Drug Abuse CTN0044). The text does not explicitly state the country of origin, but "National Institute of Drug Abuse" strongly suggests the United States. The study was prospective as it was a randomized controlled trial designed to characterize reSET's probable benefits and risks.
3. Number of experts used to establish the ground truth for the test set and the qualifications of those experts
The ground truth for the clinical study was established primarily through self-report and objective measures like urine drug screens.
- Self-Report: Patient self-reported substance use via Timeline Follow-Back (TLFB).
- Objective Measures: Urine drug screens and breath alcohol tests.
- Expert involvement: While clinicians were supervising patients and there was a "clinician dashboard," the "ground truth" for abstinence and retention was based on a combination of patient self-report and verifiable clinical tests (urine drug screens). The text does not specify a panel of experts explicitly establishing "ground truth" through a consensus process for individual patient outcomes in the test set. Clinical investigators would have interpreted the urine drug screens based on established protocols.
4. Adjudication method (e.g., 2+1, 3+1, none) for the test set
For abstinence, a combination of self-report and urine drug screens was used. The adjudication method for discrepancies between self-report and urine drug screens was defined in "Table 1: Urine drug screen interpretations." For example, a positive self-report plus a negative urine drug screen resulted in a "positive" interpretation (meaning non-abstinent). A negative self-report plus a positive urine drug screen also resulted in a "positive" interpretation. If both were negative, it was "negative" (abstinent). There was no explicit mention of an expert consensus or multi-reader adjudication method in the traditional sense for image-based diagnostic AI, as this is a behavioral therapy device.
5. If a multi-reader multi-case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance
No, an MRMC comparative effectiveness study was not done. The reSET device is a direct patient-facing therapeutic intervention, not a diagnostic aid for human readers. Its effectiveness was evaluated by comparing patient outcomes (abstinence and retention) in two groups: one receiving rTAU + reSET and the other receiving TAU.
6. If a standalone (i.e., algorithm only without human-in-the-loop performance) was done
The study was not a "standalone" algorithm performance study in the typical sense for diagnostic AI. The reSET device is designed to be an adjunct to clinician-supervised outpatient treatment. It is a digital therapy tool that patients use, and its performance is evaluated in that human-in-the-loop context (i.e., patient using the app, clinician supervising). The "rTAU + reSET" arm represents the device's performance within its intended use alongside human clinical care. The "control arm" (TAU) did not receive the device.
7. The type of ground truth used (expert consensus, pathology, outcomes data, etc.)
The ground truth used was outcomes data, specifically:
- Abstinence: A combination of patient self-report (Timeline Follow-Back) and objective urine drug screens.
- Retention: Time-to-event data (time until last face-to-face contact with the treatment program).
8. The sample size for the training set
The provided text describes a clinical trial (CTN0044) for the evaluation of the reSET device. It does not mention a separate "training set" in the context of machine learning model development. For behavioral therapy, the "training" data typically refers to the aggregated knowledge and clinical evidence that informed the design and content of the therapeutic modules themselves, rather than a specific dataset used to train a statistical model within the device. The device's content is based on established CBT and CRA principles.
9. How the ground truth for the training set was established
As there was no "training set" in the machine learning sense described for this specific clinical trial, there's no ground truth established in that manner. The "ground truth" for the content of reSET (i.e., why certain therapeutic modules and exercises are included) is based on the established evidence and clinical literature supporting Cognitive Behavioral Therapy (CBT) and Community Reinforcement Approach (CRA) as effective interventions for substance use disorder. This underlying scientific and clinical understanding forms the basis for the device's therapeutic content.
§ 882.5801 Computerized behavioral therapy device for psychiatric disorders.
(a)
Identification. A computerized behavioral therapy device for psychiatric disorders is a prescription only device intended to provide a computerized version of condition-specific behavioral therapy as an adjunct to clinician supervised outpatient treatment to patients with psychiatric conditions. The digital therapy is intended to provide patients access to therapy tools used during treatment sessions to improve recognized treatment outcomes.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Clinical data must be provided to fulfill the following:
(i) Describe a validated model of behavioral therapy for the psychiatric disorder; and
(ii) Validate the model of behavioral therapy as implemented by the device.
(2) Software must be described in detail in the software requirements specification (SRS) and software design specification (SDS). Software verification, validation, and hazard analysis must be performed. Software documentation must demonstrate that the device effectively implements the behavioral therapy model.
(3) The following labeling must be provided:
(i) Patient and physician labeling must include instructions for use, including images that demonstrate how to interact with the device.
(ii) Patient and physician labeling must list compatible devices.
(iii) Patient and physician labeling must include a warning that the device is not intended for use as a standalone therapy.
(iv) Patient and physician labeling must include a warning that the device does not represent a substitution for the patient's medication.
(v) Physician labeling must include a summary of the clinical testing with the device.