(116 days)
ImPACT is intended for use as a computer-based neurocognitive test battery to aid in the assessment and management of concussion.
ImPACT is a neurocognitive test battery that provides healthcare professionals with objective measures of neurocognitive functioning as an assessment aid and in the management of concussion in individuals ages 12-80.
ImPACT® (Immediate Post-Concussion Assessment and Cognitive Testing) is a computer-based neurocognitive test battery that allows healthcare professionals to conduct a series of tests on individuals to gather data related to the neurocognitive functioning of the test subject. This test battery measures various aspects of neurocognitive functioning including reaction time, memory, attention, spatial processing speed, and records symptoms of a test subject. ImPACT Version 4 is similar to the paper-and-pencil neuropsychological tests that have long been used by psychologists to evaluate cognition, and memory related to a wide variety of disabilities.
The device is not intended to provide a direct diagnosis or a return-to-activity recommendation, it does not directly manage or provide any treatment recommendations, and any interpretation of the results should be made only by qualified healthcare professional. The neurocognitive assessment represents only one aspect of assisting healthcare professionals in evaluating and managing individuals with cognitive function impairment related to TBI (concussion).
Here's a breakdown of the acceptance criteria and the study information for the ImPACT Version 4 device, based on the provided text:
1. Table of Acceptance Criteria and Reported Device Performance
The FDA submission primarily focuses on demonstrating substantial equivalence to a predicate device (ImPACT Version 3.3.0) rather than listing specific, quantitative acceptance criteria for each functional aspect. The document indicates that all software verification and validation tests met the required acceptance criteria, but these criteria are described generally rather than with specific metrics.
However, based on the summary of performance testing and clinical data, the implicit acceptance criteria relate to:
- Software Functionality: The software performs as intended, modifications did not affect existing functionality.
- Safety and Effectiveness: The device modifications do not raise new questions of safety and effectiveness.
- Test-Retest Reliability (for extended age range): Cognitive performance should remain stable over time.
- Construct Validity (for extended age range): ImPACT Version 4 scores should correlate with established neuropsychological tests.
- Normative Database Quality: The normative data should be accurately established for the specified age ranges, differentiating between input device types.
| Acceptance Criteria (Inferred from testing) | Reported Device Performance |
|---|---|
| Software functionality met design specifications. | All software verification activities (code reviews, design reviews, walkthroughs, software V&V testing, regression testing) met "required acceptance criteria." Modifications did not affect existing functionality. |
| Device modifications do not affect safety or effectiveness. | "The differences between the two devices... do not affect the safety or effectiveness of ImPACT Version 4 for its intended use and do not raise new questions of safety and effectiveness, which was demonstrated through risk management and performance testing." Risk management concluded all individual risk is acceptable and the new device has "virtually the same safety characteristics... and same risk profile" as the predicate. |
| Test-retest reliability is established (for ages 60-80). | For a subset of 93 individuals (ages 60-80), only a "small percentage" (0-1% for composite scores, 0-2% for factor scores) of scores showed "reliable or 'significant' change" over an average of 16.04 days. This "suggest[s] the cognitive performance of test takers at baseline remained stable over a one-month period." |
| Construct validity correlated with established tests (for ages 60-80). | ImPACT Verbal Memory Composite scores correlated significantly (P24). |
* For construct validity, the reference standards were scores from other validated neuropsychological tests administered by trained neuropsychologists.
5. If a Multi Reader Multi Case (MRMC) Comparative Effectiveness Study Was Done, If So, What Was the Effect Size of How Much Human Readers Improve with AI vs Without AI Assistance
- No, the document does not describe a multi-reader multi-case (MRMC) comparative effectiveness study evaluating how human readers (healthcare professionals) improve with ImPACT Version 4 assistance versus without it.
- ImPACT is presented as an "aid" in assessment and management, providing objective measures, but the study design did not assess the human reader's diagnostic performance with and without the device.
6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) Was Done
- Yes, the performance testing described for ImPACT Version 4 is in a standalone capacity. The device itself (the neurocognitive test battery) collects and processes data, and its outputs (scores, validity indicators) are what were evaluated for reliability and validity against other tests or over time. The "performance" being assessed is that of the computerized test generating consistent and correlated results, not a human using the device to make a decision.
7. The Type of Ground Truth Used
- For the normative database: The "ground truth" for subject inclusion was based on self-reported health status (no recent concussion, no neurological issues, no ADHD/LD) and, for the older age group, a basic cognitive screen (MMSE score > 24). The purpose was to establish norms for a healthy population.
- For test-retest reliability: The "ground truth" was the expectation that cognitive performance in a stable individual should not significantly change over a short period.
- For construct validity: The "ground truth" was established by scores from other widely utilized and previously validated traditional neuropsychological tests (HVLT, BVMT-R, SDMT).
8. The Sample Size for the Training Set
- The document does not explicitly mention a "training set" in the context of machine learning model development. ImPACT is described as a "computer-based neurocognitive test battery" and its modifications primarily involve updating the normative database and output scores, not training a new predictive algorithm.
- The "normative database" could be considered analogous to a reference dataset that the device's scoring relies upon.
- For ages 12-59: The normative database was constructed from 71,815 subjects.
- For ages 60-80: The normative database was constructed from 554 subjects.
9. How the Ground Truth for the Training Set Was Established
- As mentioned above, there isn't a "training set" for a machine learning model described.
- For the normative databases, which serve as the reference for interpreting individual test results, the "ground truth" for inclusion was:
- For 12-59 age range: De-identified data selected from a large company database. Subjects had to meet criteria such as no concussion in the past 6 months, no other neurological issues, no ADHD/LD diagnosis. These were self-reported or existing medical history.
- For 60-80 age range: Prospective data collection where subjects met strict inclusion criteria: ages 60-80, primary English speaking, not in a skilled nursing facility, not suffering from/being treated for a concussion, no known impairing physical/neurological/behavioral/psychological conditions, corrected hearing/vision within normal limits, and a Mini-Mental State Examination (MMSE) score of 24 or greater. This was established through screening and clinical assessment at 8 different sites.
§ 882.1471 Computerized cognitive assessment aid for concussion.
(a)
Identification. The computerized cognitive assessment aid for concussion is a prescription device that uses an individual's score(s) on a battery of cognitive tasks to provide an indication of the current level of cognitive function in response to concussion. The computerized cognitive assessment aid for concussion is used only as an assessment aid in the management of concussion to determine cognitive function for patients after a potential concussive event where other diagnostic tools are available and does not identify the presence or absence of concussion. It is not intended as a stand-alone diagnostic device.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Software, including any proprietary algorithm(s) used by the device to arrive at its interpretation of the patient's cognitive function, must be described in detail in the software requirements specification (SRS) and software design specification (SDS). Software verification, validation, and hazard analysis must be performed.
(2) Clinical performance data must be provided that demonstrates how the device functions as an interpretation of the current level of cognitive function in an individual that has recently received an injury that causes concern about a possible concussion. The testing must:
(i) Evaluate device output and clinical interpretation.
(ii) Evaluate device test-retest reliability of the device output.
(iii) Evaluate construct validity of the device cognitive assessments.
(iv) Describe the construction of the normative database, which includes the following:
(A) How the clinical workup was completed to establish a “normal” population, including the establishment of inclusion and exclusion criteria.
(B) Statistical methods and model assumptions used.
(3) The labeling must include:
(i) A summary of any clinical testing conducted to demonstrate how the device functions as an interpretation of the current level of cognitive function in a patient that has recently received an injury that causes concern about a possible concussion. The summary of testing must include the following:
(A) Device output and clinical interpretation.
(B) Device test-retest reliability of the device output.
(C) Construct validity of the device cognitive assessments.
(D) A description of the normative database, which includes the following:
(
1 ) How the clinical workup was completed to establish a “normal” population, including the establishment of inclusion and exclusion criteria.(
2 ) How normal values will be reported to the user.(
3 ) Representative screen shots and reports that will be generated to provide the user results and normative data.(
4 ) Statistical methods and model assumptions used.(
5 ) Whether or not the normative database was adjusted due to differences in age and gender.(ii) A warning that the device should only be used by health care professionals who are trained in concussion management.
(iii) A warning that the device does not identify the presence or absence of concussion or other clinical diagnoses.
(iv) A warning that the device is not a stand-alone diagnostic.
(v) Any instructions technicians must convey to patients regarding the administration of the test and collection of cognitive test data.