Search Results

ImPACT is a neurocognitive test battery that provides healthcare professionals with objective measure of neurocognitive functioning as an assessment aid and in the management of concussion in individuals ages 12-80.

Device Description

ImPACT® (Immediate Post-Concussion Assessment and Cognitive Testing) is a computer-based neurocognitive test battery that allows healthcare professionals to conduct a series of tests on individuals to gather data related to the neurocognitive functioning of the test subject. This test battery measures various aspects of neurocognitive functioning including reaction time, memory, attention, spatial processing speed, and records symptoms of a test subject. ImPACT Version 4 is similar to the paper-and-pencil neuropsychological tests that have long been used by psychologists to evaluate cognition, and memory related to a wide variety of disabilities.

The device is not intended to provide a direct diagnosis or a return-to-activity recommendation, it does not directly manage or provide any treatment recommendations, and any interpretation of the results should be made only by qualified healthcare professional. The neurocognitive assessment represents only one aspect of assisting healthcare professionals in evaluating and managing individuals with cognitive function impairment related to TBI (concussion).

AI/ML Overview

Here's a breakdown of the acceptance criteria and the study proving the device meets them, based on the provided text:

Acceptance Criteria and Reported Device Performance

The document primarily focuses on demonstrating substantial equivalence to a predicate device (ImPACT Version 4, K202485) rather than explicitly listing quantitative acceptance criteria for each neurocognitive measure. However, the core of the performance demonstration rests on the establishment of a normative database for iPad use and test-retest reliability consistent with previous versions.

Implied Acceptance Criteria and Performance:

Acceptance Criteria (Implied)	Reported Device Performance
1. Normative Database for iPad: The device must establish a reliable and representative normative database for performance on an iPad, allowing for accurate interpretation of test results compared to a healthy population.	A prospective clinical investigation was conducted to collect data for standardization and to construct the normative database for tests performed on an iPad.
The normative sample included 1495 subjects ages 12-59 (670 males, 825 females). Data was collected prospectively from 4 different sites across the US in 2022 and 2023. These sites were approved by two ethics boards (Advarra IRB Services and St. Joseph's University in Philadelphia).
Data collection occurred in a mixed environment (supervised and unsupervised testing) to approximate real-world conditions. All subjects met inclusion criteria consistent with previous normative data creation (age 12-59, primary English speaking, no concussion in past 6 months, no known physical/neurological/behavioral/psychological impairment affecting test, corrected hearing/vision impairments, signed IRB consent).
2. Test-retest Reliability (iPad): The neurocognitive measures on the iPad must demonstrate high consistency over time, meaning a subject's scores should be similar if they take the test multiple times within a reasonable period, assuming no change in their cognitive state.	Test-retest reliability was calculated in a sample of 116 individuals ages 12-59 who were part of the standardization sample. They completed an initial baseline assessment on an iPad and a second baseline within 7-21 days (mean=12.7 days, SD=4.3 days).
Pearson's Product-Moment Correlation coefficients and Intra-class correlation coefficients (ICCs) were calculated for ImPACT Composite Scores and Two Factor Scores.
The reported Pearson's correlations and ICCs were "consistent with those from the test-retest coefficients obtained using Mouse and Trackpad inputs of the predicate device." This indicates that the iPad version maintains the reliability characteristics of the established predicate device.
3. Safety and Effectiveness (Overall Equivalence): The device modifications (specifically the iPad platform and related software for touchscreen input and normative data) must not adversely affect the safety or effectiveness for its intended use, nor raise new questions of safety and effectiveness.	The document states: "The differences between the two devices described above do not affect the safety or effectiveness of ImPACT Version 4.1 for its intended use and do not raise new questions of safety and effectiveness, which was demonstrated through risk management and performance testing including software verification and validation, clinical investigations and non-clinical assessments."
Risk management activities were conducted per ISO 14971, assuring risks are controlled and the new device has the same safety and risk profile as the predicate.
Software verification and validation (IEC 62304, FDA Guidance "General Principles of Software Validation") included code reviews, design reviews, automation/manual testing, and regression testing, with all tests meeting acceptance criteria.

Detailed Study Information:

Sample Size and Data Provenance:
- Test Set (Normative Data):
  - Sample Size: 1495 subjects.
  - Data Provenance: Prospectively collected from 4 different sites across the US in 2022 and 2023.
  - Type: Prospective.
- Test Set (Test-retest Reliability):
  - Sample Size: 116 individuals, a subset of the normative sample.
  - Data Provenance: Prospectively collected from the same US sites as the normative data, in 2022 and 2023.
  - Type: Prospective.
Number of Experts and Qualifications for Ground Truth:
- The document does not explicitly state the number or qualifications of experts used to establish a "ground truth" in the traditional sense of a diagnostic consensus.
- For the normative and test-retest studies, the "ground truth" is primarily based on the self-reported health status of the participants (e.g., "not suffering from a concussion or being treated for a concussion in the past 6 months," "no known physical, neurological, behavioral or psychological impairment"). The inclusion criteria, which participants had to meet, effectively defined the "healthy" or "normal" population against which the device's performance is normed.
- Clinical experts were part of the "cross-functional team" involved in "Walkthroughs and design reviews of mock-ups and prototypes" during software verification, but not explicitly for ground truth establishment for the clinical study data itself.
Adjudication Method for the Test Set:
- The document does not describe an adjudication method for the clinical study data. The studies focused on collecting representative data from healthy individuals to establish norms and assess reliability, rather than evaluating the device's diagnostic accuracy against a separate, adjudicated clinical diagnosis.
Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study:
- No, an MRMC comparative effectiveness study was not performed. This device is a neurocognitive test battery, not an imaging AI diagnostic aid meant to be used by multiple readers on the same cases. The "human-in-the-loop" aspect is not about human readers interpreting AI output, but rather healthcare professionals using the device's objective measures to aid in assessment and management.
Standalone Performance:
- The device ImPACT is a "computer-based neurocognitive test battery." The performance described (normative data collection, test-retest reliability) is inherently the "algorithm only" performance, where the "algorithm" refers to the test battery itself and its scoring methodology. The device provides "objective measure of neurocognitive functioning," which is its standalone output. The interpretation of these results is then done by a qualified healthcare professional.
- So, yes, a standalone performance assessment was done in the form of normative data collection and test-retest reliability, demonstrating the device's intrinsic ability to measure cognitive function consistently in an uninjured population.
Type of Ground Truth Used:
- The primary ground truth for the normative database was the self-reported clinical status and inclusion/exclusion criteria of the participants, aiming for a "healthy" or "uninjured" cognitive state. This serves as the reference for "normal" cognitive functioning.
- For the test-retest reliability, the implicit ground truth is that the cognitive state of the healthy individuals should not have changed significantly between the two tests, allowing for an evaluation of the device's consistency.
Sample Size for the Training Set:
- The document does not explicitly describe a separate "training set" for an AI or machine learning model in the context of this 510(k) submission.
- The term "normative database" sometimes functions similarly to a reference or training set in that it establishes what "normal" looks like. In this case, the 1495 subjects for the normative database building represent the data used to define the expected range of performance.
- The "training" of the neurocognitive test itself, being a standardized battery, is more akin to its initial development and validation stages prior to the specific modifications addressed in this 510(k). The current submission focuses on adapting an existing, validated test to a new platform (iPad) and updating its normative reference.
How the Ground Truth for the Training Set Was Established:
- As mentioned above, if the normative database is considered the "training set" for defining normal performance, its "ground truth" was established by prospectively enrolling individuals who met strict inclusion criteria indicating they were healthy, English-speaking, not suffering from or recently treated for a concussion, and without other known neurological/physical impairments that would affect test performance. This was overseen by ethics boards and involved multiple US sites.

Ask a Question

Ask a specific question about this device

K Number

K202485

Device Name

ImPACT Version 4

Manufacturer

ImPACT Applications, Inc.

Date Cleared

2020-12-25

(116 days)

Product Code

Regulation Number

Type

Panel

Reference & Predicate Devices

K181223

Intended Use

ImPACT is intended for use as a computer-based neurocognitive test battery to aid in the assessment and management of concussion.

ImPACT is a neurocognitive test battery that provides healthcare professionals with objective measures of neurocognitive functioning as an assessment aid and in the management of concussion in individuals ages 12-80.

Device Description

AI/ML Overview

Here's a breakdown of the acceptance criteria and the study information for the ImPACT Version 4 device, based on the provided text:

1. Table of Acceptance Criteria and Reported Device Performance

The FDA submission primarily focuses on demonstrating substantial equivalence to a predicate device (ImPACT Version 3.3.0) rather than listing specific, quantitative acceptance criteria for each functional aspect. The document indicates that all software verification and validation tests met the required acceptance criteria, but these criteria are described generally rather than with specific metrics.

However, based on the summary of performance testing and clinical data, the implicit acceptance criteria relate to:

Software Functionality: The software performs as intended, modifications did not affect existing functionality.
Safety and Effectiveness: The device modifications do not raise new questions of safety and effectiveness.
Test-Retest Reliability (for extended age range): Cognitive performance should remain stable over time.
Construct Validity (for extended age range): ImPACT Version 4 scores should correlate with established neuropsychological tests.
Normative Database Quality: The normative data should be accurately established for the specified age ranges, differentiating between input device types.

Acceptance Criteria (Inferred from testing)	Reported Device Performance
Software functionality met design specifications.	All software verification activities (code reviews, design reviews, walkthroughs, software V&V testing, regression testing) met "required acceptance criteria." Modifications did not affect existing functionality.
Device modifications do not affect safety or effectiveness.	"The differences between the two devices... do not affect the safety or effectiveness of ImPACT Version 4 for its intended use and do not raise new questions of safety and effectiveness, which was demonstrated through risk management and performance testing." Risk management concluded all individual risk is acceptable and the new device has "virtually the same safety characteristics... and same risk profile" as the predicate.
Test-retest reliability is established (for ages 60-80).	For a subset of 93 individuals (ages 60-80), only a "small percentage" (0-1% for composite scores, 0-2% for factor scores) of scores showed "reliable or 'significant' change" over an average of 16.04 days. This "suggest[s] the cognitive performance of test takers at baseline remained stable over a one-month period."
Construct validity correlated with established tests (for ages 60-80).	ImPACT Verbal Memory Composite scores correlated significantly (P24).

*   For construct validity, the reference standards were scores from other validated neuropsychological tests administered by trained neuropsychologists.

5. If a Multi Reader Multi Case (MRMC) Comparative Effectiveness Study Was Done, If So, What Was the Effect Size of How Much Human Readers Improve with AI vs Without AI Assistance

No, the document does not describe a multi-reader multi-case (MRMC) comparative effectiveness study evaluating how human readers (healthcare professionals) improve with ImPACT Version 4 assistance versus without it.
ImPACT is presented as an "aid" in assessment and management, providing objective measures, but the study design did not assess the human reader's diagnostic performance with and without the device.

6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) Was Done

Yes, the performance testing described for ImPACT Version 4 is in a standalone capacity. The device itself (the neurocognitive test battery) collects and processes data, and its outputs (scores, validity indicators) are what were evaluated for reliability and validity against other tests or over time. The "performance" being assessed is that of the computerized test generating consistent and correlated results, not a human using the device to make a decision.

7. The Type of Ground Truth Used

For the normative database: The "ground truth" for subject inclusion was based on self-reported health status (no recent concussion, no neurological issues, no ADHD/LD) and, for the older age group, a basic cognitive screen (MMSE score > 24). The purpose was to establish norms for a healthy population.
For test-retest reliability: The "ground truth" was the expectation that cognitive performance in a stable individual should not significantly change over a short period.
For construct validity: The "ground truth" was established by scores from other widely utilized and previously validated traditional neuropsychological tests (HVLT, BVMT-R, SDMT).

8. The Sample Size for the Training Set

The document does not explicitly mention a "training set" in the context of machine learning model development. ImPACT is described as a "computer-based neurocognitive test battery" and its modifications primarily involve updating the normative database and output scores, not training a new predictive algorithm.
The "normative database" could be considered analogous to a reference dataset that the device's scoring relies upon.
- For ages 12-59: The normative database was constructed from 71,815 subjects.
- For ages 60-80: The normative database was constructed from 554 subjects.

9. How the Ground Truth for the Training Set Was Established

As mentioned above, there isn't a "training set" for a machine learning model described.
For the normative databases, which serve as the reference for interpreting individual test results, the "ground truth" for inclusion was:
- For 12-59 age range: De-identified data selected from a large company database. Subjects had to meet criteria such as no concussion in the past 6 months, no other neurological issues, no ADHD/LD diagnosis. These were self-reported or existing medical history.
- For 60-80 age range: Prospective data collection where subjects met strict inclusion criteria: ages 60-80, primary English speaking, not in a skilled nursing facility, not suffering from/being treated for a concussion, no known impairing physical/neurological/behavioral/psychological conditions, corrected hearing/vision within normal limits, and a Mini-Mental State Examination (MMSE) score of 24 or greater. This was established through screening and clinical assessment at 8 different sites.

Ask a Question

Ask a specific question about this device

Page 1 of 1