(53 days)
BrainInsight is intended for automatic labeling, spatial measurement, and volumetric quantification of brain structures from a set of low-field MR images and returns annotated and segmented images, color overlays and reports.
BrainInsight is a fully automated MR imaging post-processing medical software that provides image alignment, whole brain segmentation, ventricle segmentation, and midline shift measurements of brain structures from a set MR images. The BrainInsight processing architecture includes a proprietary automated internal pipeline based on machine learning tools. The output annotated and segmented images are provided in standard image format using segmented color overlays and reports that can be displayed on third-party workstations and FDA-cleared Picture Archive and Communications Systems (PACS). The high throughput capability makes the software suitable for use in routine patient care as a support tool for clinicians in assessment of low-field (0.064 T) structural MRIs. BrainInsight provides overlays and reports based on 0.064 T 3D MRI series of T1 Gray/White, T2-Fast, and FLAIR images.
Here's a breakdown of the acceptance criteria and the study details for the BrainInsight™ device, based on the provided text:
1. Table of Acceptance Criteria and Reported Device Performance
The acceptance criteria were defined based on non-inferiority testing, aiming for the model performance to be no worse than the average annotator's discrepancy.
Midline Shift Discrepancy (Lower is Better)
| Application | Modality | Acceptance Criteria (Model 2 to 12 years (20.6%), >12 to 18 to 90 years (70.6%)
* Gender: 33% Female / 41% Male / 25% Anonymized
* Pathology: Stroke (Infarct), Hydrocephalus, Hemorrhage (SAH, SDH, IVH, IPH), Mass/Edema, Tumor.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Their Qualifications
- Number of Experts: The document states that the datasets for training and validation were annotated by "multiple experts." It then mentions that "The entire group of training image sets was divided into segments and each segment was given to a single expert." This phrasing is somewhat ambiguous for the test set specifically. It is implied that multiple experts were involved in the ground truth establishment for the overall process, but it doesn't clearly state how many experts independently evaluated each case in the test set, nor if the "single expert per segment" approach also applied to the test set ground truth.
- Qualifications of Experts: Not specified beyond being referred to as "experts" and "annotators."
4. Adjudication Method for the Test Set
The adjudication method varies by application:
- Midline Shift: Ground truth was determined based on the average shift distance of all annotators. This implies a form of consensus or averaging method rather than a strict adjudication by a senior expert.
- Segmentation (Lateral Ventricles, Whole Brain): Ground truth for segmentation was calculated using Simultaneous Truth and Performance Level Estimation (STAPLE). STAPLE is an algorithm that estimates a "true" segmentation from multiple segmentations, weighting them based on their estimated performance. This is an algorithmic adjudication method.
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
- Was a MRMC study done? No, a traditional MRMC comparative effectiveness study that measures how human readers improve with AI vs. without AI assistance was not explicitly described for this submission. The study focuses on standalone performance of the AI model against expert annotations and the "mean annotator" performance.
- Effect Size of Human Improvement (if applicable): Not applicable, as an MRMC comparative effectiveness study was not detailed.
6. Standalone (Algorithm Only Without Human-in-the-Loop Performance) Study
- Was a standalone study done? Yes, the described performance evaluation appears to be a standalone (algorithm only) study. The device's performance is compared directly against the ground truth established by annotators, and against the mean discrepancy of the annotators themselves. There is no mention of human readers using the AI output to improve their performance compared to a baseline.
7. Type of Ground Truth Used
The type of ground truth used varies by the measurement:
- Midline Shift: Expert consensus, calculated as the average shift distance of all annotators.
- Segmentation (Lateral Ventricles, Whole Brain): Algorithmic consensus, calculated using Simultaneous Truth and Performance Level Estimation (STAPLE) based on expert annotations.
- General: It is based on expert annotations of images acquired from the Hyperfine Swoop portable MRI system.
8. Sample Size for the Training Set
- Sample Size for Training Set: The exact numerical sample size for the training set is not explicitly stated. The document only mentions that the data collection for the training and validation datasets was done at "multiple sites."
9. How the Ground Truth for the Training Set Was Established
- The data collection for the training and validation datasets was done at multiple sites.
- The datasets were annotated by multiple experts.
- The "entire group of training image sets was divided into segments and each segment was given to a single expert."
- "The expert's determination became the ground truth for each image set in their segment." This implies a form of single-reader ground truth for each segmented batch, rather than multi-reader consensus for every single case within the training set.
§ 892.2050 Medical image management and processing system.
(a)
Identification. A medical image management and processing system is a device that provides one or more capabilities relating to the review and digital processing of medical images for the purposes of interpretation by a trained practitioner of disease detection, diagnosis, or patient management. The software components may provide advanced or complex image processing functions for image manipulation, enhancement, or quantification that are intended for use in the interpretation and analysis of medical images. Advanced image manipulation functions may include image segmentation, multimodality image registration, or 3D visualization. Complex quantitative functions may include semi-automated measurements or time-series measurements.(b)
Classification. Class II (special controls; voluntary standards—Digital Imaging and Communications in Medicine (DICOM) Std., Joint Photographic Experts Group (JPEG) Std., Society of Motion Picture and Television Engineers (SMPTE) Test Pattern).