(203 days)
The Neurometric Analysis System (NAS) is to be used by qualified medical professionals for the post-hoc statistical evaluation of the human electroencephalogram (EEG).
The Neurometric Analysis System (NAS) is a software program for the post-hoc statistical analysis of the human electroencephalogram (EEG). Digital EEG data from a host system is transferred to the NAS for display and user-review. The system requires that the user select approximately 2.00 minutes of artifact-free, eyes-closed, resting EEG from the recording for analysis. Analysis consists of the Fast-Fourier Transformation (FFT) of the data to extract the spectral power for each of the four primary frequency bands (delta, theta, alpha, and beta), and frequency information from the EEG. The results of this analysis are then subjected to univariate, bivariate, and multivariate statistical analyses and displayed in statistical tables and topographical brain maps of absolute and relative power, power asymmetry, and coherence for 19 monopolar and 8 selected bipolar derivations of the EEG. In all over 1200 measures are derived for comparison against a carefully constructed and statistically controlled age-regressed, normative database in which the variables have been transformed and confirmed for their Gaussian distribution. Each variable extracted by the analysis is compared to the database using parametric statistical procedures that express the differences between the patient and their appropriate age-matched reference group in the form of Z-scores. Multivariate features are compared to the normative database using Mahalanobis Distance Statistics. The Mahalanobis Distance statistic controls for the interrelationship of the measures of brain cortical function in the feature set, and provides an accurate estimate of their difference from normal. The multivariate measures permit an evaluation of regional indices of brain function that reflect the perfusion fields of the brain. Extracted feature sets are further analyzed to determine if the pattern of 'hits' (statistically, significant feature score values identified for the patient) are consistent with patterns of 'hits' identified in prior neurometric evaluations of clinical patients with known disorders. A step-wise discriminant analysis program classifies the patient in terms of their similarity to known neurometric-defined patterns of abnormality, providing a probability estimate of the patient's profile with the average profile of groups of individuals constituting the normative and clinical database. The discriminant classification program is restricted by confiniters potential outcomes to specific patient symptoms derived from the patient history. profile. Established discriminant functions were evaluated through the use of Receiver Operating Characteristic (ROC) curves for their sensitivity and specificity. The outcome of the statistical analysis is presented in report form that includes (a) patient demographic and history information, (b) selected EEG epochs, (c) statistical tables of monopolar, the lastery and multivariated feature values, and topographical brain maps. This information is to be read and interpreted within the context of the current clinical assessment of the patient by the attending physician. The decision to accept or reject the results of the neurometric analysis, and incorporate these results into their clinical appraisal of the patient, is dependent upon the judgment of the attending physician.
The Neurometric Analysis System is complete in a set of five 3.5 diskettes, which contains. a demonstration program with sample neurometric studies, the NAS program, and the a demonstration program. The NAS was designed for implementation under DOS and Windows, and programmed using C++. The user interface was carefully designed and implemented to programmed comb procedures are used to record steps used in program usage, and the conduct of the analysis to insure appropriate function and operation of the software. The NAS can be installed in any appropriately configured IBM-compatible computer system, including systems designed specifically for the recording of digital EEG. The system functions with systems access of standard computer platforms and input-output devices, and printers.
The provided text describes the Neurometric Analysis System (NAS) and its evaluation. However, it does not explicitly state quantitative acceptance criteria or a formal study designed to demonstrate compliance with such criteria in the way a modern medical device submission might. Instead, it focuses on demonstrating substantial equivalence to predicate devices and verifying the system's consistency and accuracy with established methods.
Here's an attempt to extract the requested information based on the provided text, acknowledging where specific details are not available:
1. Table of Acceptance Criteria and Reported Device Performance
As explicit, quantitative acceptance criteria are not presented in the document, this table will reflect the qualitative criteria and claims made about the device's performance based on consistency with established methods and existing data.
| Acceptance Criteria (Inferred from text) | Reported Device Performance |
|---|---|
| Non-clinical: | |
| - Accuracy of signal generation and frequency/power analysis. | Control signals (generated waveforms) were analyzed for frequency and power, and EEG signals were analyzed for conformity between the host digital EEG system and the NAS. The NAS accurately reproduced the sampling frequency in the host digital EEG system and allowed correct translation of EEG waveforms. |
| - Consistency and accuracy of NAS analysis compared to prior neurometric methods/software using stored subject data. | Analysis of stored subject data had to conform to that of prior analyses conducted using the same method, procedures, algorithms, and analysis as implemented on the NAS. This consistency and accuracy was confirmed. |
| Clinical: | |
| - Agreement of NAS analysis results (statistical tables, topographical brain maps) with results from the host system at Brain Research Laboratory. | The results of the NAS analysis (statistical tables and topographical brain maps) were in agreement with the results of the analysis conducted on the host system used in processing patient information at BRL. |
| - Consistency of discriminant analysis outcome without misclassification errors compared to the host system at BRL. | The outcome of the discriminant analysis was consistent, not resulting in misclassification errors (i.e., classification on NAS was consistent with that of the host system at BRL). |
| - Reproducibility of results within an acceptable degree of variation when using artifact-free, eyes-closed, resting EEG. | When eyes-closed, resting, and artifact-free EEG was selected for analysis, the results were reproducible within an acceptable degree of variation consistent with reliability estimates identified in normative studies. |
| - Safety and effectiveness as an adjunctive aid for diagnosis, treatment planning, and follow-up. | Demonstrated through 20 years of non-clinical and clinical testing, concluding the NAS is safe and effective for quantitative analysis of eyes-closed resting EEG in alert human subjects, providing complementary and supplementary information to traditional EEG. When properly used as an adjunctive aid, it significantly reduces the likelihood of introducing error into diagnosis and treatment. |
2. Sample Size Used for the Test Set and Data Provenance
- Test Set Sample Size: The document does not specify a distinct "test set" sample size for the clinical testing in terms of a specific number of individuals for a standalone validation study. Instead, it refers to "individuals who ranged in age from 6 to 90 years, and who were either volunteers or clinical patients referred for neurometric evaluation to the Brain Research Laboratory."
- Data Provenance: The data used for clinical testing originated from the Brain Research Laboratory (BRL) at New York University's Medical Center. This involved a 20-year effort to construct a normative and clinical database. The data appears to be retrospective in the sense that the initial development and evaluation were based on this existing, large database and methods developed at BRL over time. The "clinical testing" section describes confirming the NAS's output against the "host system" used at BRL, suggesting re-analyzing existing patient data.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications of those Experts
- The document does not explicitly state the "number of experts" used to establish ground truth for a discrete test set. The ground truth appears to be implicitly established by the long-standing clinical practice, established neurometric methods, and accumulated knowledge within the Brain Research Laboratory (BRL) at New York University's Medical Center, including comparisons to "known disorders" and established "patterns of abnormality."
- Qualifications of Experts: The initial development and database construction involved "numerous government and privately funded normative and clinical database projects" carried out at the BRL. This implies highly qualified researchers and medical professionals (including those from the Department of Psychiatry and Department of Neurology at NYU Medical Center) were involved in generating and interpreting this data over 20 years. The interpretation of the NAS's output is also left to the "attending physician" who is a "qualified medical professional."
4. Adjudication Method for the Test Set
- The document does not describe a formal adjudication method (e.g., 2+1, 3+1) for establishing ground truth for a specific test set. The clinical evaluation primarily focused on ensuring the NAS's results were consistent with analyses performed on the "host system" at the Brain Research Laboratory, which itself utilized discriminant analysis evaluated by Receiver Operating Characteristic (ROC) curves for sensitivity and specificity. The "ground truth" seems to be derived from the extensive pre-existing normative and clinical database and established methods at BRL, rather than a de novo expert adjudication process for this specific submission.
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was done, and the effect size of how much human readers improve with AI vs. without AI assistance
- No MRMC comparative effectiveness study was described. The evaluation focused on the device's consistency with established methods and its utility as an adjunct to clinical assessment, rather than a direct comparison of human performance with and without the device. The text emphasizes that the NAS provides information that "complements and supplements the outcome of the analysis of a traditional EEG" and that "the decision to accept or reject the results... is dependent upon the judgment of the attending physician." It also warns against using the device as a "standalone diagnostic tool."
6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) was done
- Yes, a form of standalone performance evaluation was done, specifically in the non-clinical and clinical testing where the NAS's output was compared directly against the results from the "host system" at the Brain Research Laboratory. The non-clinical testing involved analyzing generated waveforms and ensuring conformity between the host digital EEG system and the NAS, as well as confirming consistency with prior analyses of stored subject data. The clinical testing confirmed the agreement of statistical tables and brain maps, and the consistency of discriminant analysis with the BRL host system. This suggests an evaluation of the algorithm's output in isolation compared to a known reference standard (the BRL-established methods and system).
7. The Type of Ground Truth Used
- The ground truth used is primarily expert consensus and established clinical practice/data from a long-standing research institution (Brain Research Laboratory at NYU Medical Center). This includes:
- Comparison against a "carefully constructed and statistically controlled age-regressed, normative database."
- Comparison of 'hits' against "patterns of 'hits' identified in prior neurometric evaluations of clinical patients with known disorders."
- Discriminant analysis based on known neurometric-defined patterns of abnormality.
- ROC curves for sensitivity and specificity of established discriminant functions.
8. The Sample Size for the Training Set
- The document does not explicitly delineate a "training set" in modern machine learning terms with a specific sample size. Instead, it refers to a "carefully constructed and statistically controlled age-regressed, normative database" and a "clinical database" developed over a 20-year effort at the Brain Research Laboratory. This extensive database served as the foundation for the statistical models and comparisons used by the NAS. While no single number is given, the implication is a very large and comprehensive dataset accumulated over two decades.
9. How the Ground Truth for the Training Set Was Established
- The ground truth for this extensive database (which effectively serves as the "training" and "reference" for the NAS) was established through:
- Rigorous data collection: "numerous government and privately funded normative and clinical database projects."
- Careful construction of normative data: "carefully constructed and statistically controlled age-regressed, normative database in which the variables have been transformed and confirmed for their Gaussian distribution."
- Clinical validation: Inclusion of "clinical patients with known disorders" whose "patterns of 'hits'" (statistically significant feature score values) were identified in prior neurometric evaluations.
- Statistical methods: Use of parametric statistical procedures, Z-scores, Mahalanobis Distance Statistics, and step-wise discriminant analysis.
- Expert experience: The entire process was guided by the "extensive, 20-year effort" and expertise at the Brain Research Laboratory, involving medical professionals from Neurology and Psychiatry departments who defined clinical reference groups and patterns of abnormality.
§ 882.1400 Electroencephalograph.
(a)
Identification. An electroencephalograph is a device used to measure and record the electrical activity of the patient's brain obtained by placing two or more electrodes on the head.(b)
Classification. Class II (performance standards).