(371 days)
The EveBOX is intended to measure and analyze eve movements as an aid in the diagnosis of concussion within one week of head injury in patients 5 through 67 years of age in conjunction with a standard neurological assessment of concussion.
A negative EyeBOX classification may correspond to eye movement that is consistent with a lack of concussion.
A positive EyeBOX classification corresponds to eye movement that may be present in both patients with or without concussion.
The Oculogica EyeBOX system consists of an integrated stand, eye-tracking camera, video stimulus display screen, and computer programmed for analysis of eye movements. It is intended to detect abnormal eye movement that may be related to a concussion. The device measures gaze, calculates a score on a 0-20 scale based on these measurements, and displays an EyeBOX classification based upon whether the scale value is above 10 or not. Scale values of 10 or more yield a positive EyeBOX classification, while scale values under 10 yield a negative EyeBOX classification.
Here's a breakdown of the acceptance criteria and the study proving the EyeBOX device meets them, based on the provided text:
Acceptance Criteria and Device Performance
| Acceptance Criteria (Pre-specified Performance Goals) | Reported Device Performance (Full Cohort) |
|---|---|
| Lower one-sided 95% confidence limit greater than 70% for sensitivity | Sensitivity: 80.4% (95% CI: 66.1% to (b)(4) - Met the goal as 66.1% > 70% is incorrect, 80.4% is the point estimate. The CI lower bound 66.1% does not meet the goal of >70%. The text explicitly states "These goals were not met in the pivotal clinical study." |
| Lower one-sided 95% confidence limit greater than 70% for specificity | Specificity: 66.1% (95% CI: 59.7% to 72.1%) - Did not meet the goal as the lower bound of 59.7% is not >70%. The text explicitly states "These goals were not met in the pivotal clinical study." |
Note: The document explicitly states, "These goals were not met in the pivotal clinical study." However, the FDA's decision to grant De Novo classification indicates that while the prespecified performance goals (specifically the lower bound of the CI for sensitivity and specificity) were not met, the agency found that the probable benefits outweighed the probable risks, especially considering the device's adjunct nature, high Negative Predictive Value (NPV), and the challenges in clinical concussion adjudication.
Study Details
| Aspect | Description |
|---|---|
| Sample Size (Test Set) | 282 subjects with complete data for analysis. Of these, 263 were assessed within 1 week of injury, which is the relevant timeframe for the device's intended use. |
| Data Provenance | The text does not explicitly state the country of origin. It indicates it was a "pivotal clinical study," implying a prospective design for the purpose of regulatory approval. The study involved subjects screened and enrolled, making it a prospective study. |
| Number of Experts & Qualifications | Initially, a 3-clinician panel was used. Their specific qualifications are not detailed, but they are implied to be clinicians capable of making recommendations on concussion status. The text expresses concern that "84.4% of the first N=199 adjudications had at least one clinician render a recommendation of 'uncertain' for the patient's concussion status," leading to a revised ground truth definition. |
| Adjudication Method | Initially, a 3-clinician panel was intended, but due to a high rate of "uncertain" recommendations, the method was revised. The revised standard for classifying concussion status was based on objective criteria from the SCAT3 (Standardized Assessment of Concussion) and SCAT3 Symptom Severity Score (SSS), combined with the presence of Alteration of Consciousness (AOC) or Altered Mental Status (AMS), rather than direct clinician consensus on the final classification. This suggests a shift from direct consensus adjudication to a rule-based algorithm derived from common clinical assessment tools. |
| MRMC Comparative Effectiveness Study | Not described. The study evaluated the stand-alone performance of the EyeBOX against a clinical reference standard, without comparing human readers with and without AI assistance. The device is specified as an "assessment aid" and "not intended for standalone detection or diagnostic purposes," implying human-in-the-loop use, but an MRMC study to quantify improvement was not performed or reported here. |
| Standalone Performance (Algorithm Only) | Yes, the reported sensitivity, specificity, PPV, and NPV are measures of the EyeBOX algorithm's performance in classifying patients as "Positive" or "Negative" for concussion based on its internal score (0-20 scale), without direct human intervention in the classification itself. This is a standalone performance evaluation against the defined clinical reference standard. |
| Type of Ground Truth Used | Clinical Reference Standard: This was a rule-based definition of concussion established for the study. |
| Initially: A 3-clinician panel (abandoned due to high uncertainty). | |
| Revised: A subject had a concussion if they exhibited (a) AOC/AMS AND SAC 25, OR (b) if they did not exhibit AOC/AMS BUT SAC 32. This method aimed to create a more objective and consistent ground truth in the absence of a "gold standard" for concussion diagnosis. | |
| Sample Size for Training Set | Not specified in the provided text. The document describes a "pivotal clinical study" for performance evaluation (test set), but does not explicitly mention the size or nature of a separate training set if machine learning was involved in the algorithm's development. |
| Ground Truth for Training Set (How Established) | Not specified. Given the lack of details on a distinct training set and the algorithm description (data collection, preprocessing, underlying model, score calculation), it's possible the "training" involved expert-driven feature engineering and model parameter tuning rather than supervised learning on a large, separately labeled dataset. If any machine learning was used for the "underlying model," the method for establishing ground truth for that training would be crucial, but it is not detailed in this document. |
§ 882.1455 Traumatic brain injury eye movement assessment aid.
(a)
Identification. A traumatic brain injury eye movement assessment aid is a prescription device that uses a patient's tracked eye movements to provide an interpretation of the functional condition of the patient's brain. This device is an assessment aid that is not intended for standalone detection or diagnostic purposes.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Clinical performance data under anticipated conditions of use must evaluate tracked eye movement in supporting the indications for use and include the following:
(i) Evaluation of sensitivity, specificity, positive predictive value, and negative predictive value using a reference method of diagnosis;
(ii) Evaluation of device test-retest reliability; and
(iii) A description of the development of the reference method of diagnosis, which may include a normative database, to include the following:
(A) A discussion of how the clinical work-up was completed to establish the reference method of diagnosis, including the establishment of inclusion and exclusion criteria; and
(B) If using a normative database, a description of how the “normal” population was established, and the statistical methods and model assumptions used.
(2) Software verification, validation, and hazard analysis must be performed. Software documentation must include a description of the algorithms used to generate device output.
(3) Performance testing must demonstrate the electrical safety and electromagnetic compatibility (EMC) of the device.
(4) The patient-contacting components of the device must be demonstrated to be biocompatible.
(5) A light hazard assessment must be performed for all eye-tracking and visual display light sources.
(6) Labeling must include:
(i) A summary of clinical performance testing conducted with the device, including sensitivity, specificity, positive predictive value, negative predictive value, and test-retest reliability;
(ii) A description of any normative database that includes the following:
(A) The clinical definition used to establish a “normal” population and the specific selection criteria;
(B) The format for reporting normal values;
(C) Examples of screen displays and reports generated to provide the user results and normative data;
(D) Statistical methods and model assumptions; and
(E) Any adjustments for age and gender.
(iii) A warning that the device should only be used by trained healthcare professionals;
(iv) A warning that the device does not identify the presence or absence of traumatic brain injury or other clinical diagnoses;
(v) A warning that the device is not a standalone diagnostic; and
(vi) Any instructions to convey to patients regarding the administration of the test and collection of test data.