(183 days)
Rapid ASPECTS is a computer-aided diagnosis (CADx) software device used to assist the clinician in the assessment and characterization of brain tissue abnormalities using CT image data. The Software automatically registers images and segments and analyzes ASPECTS Regions of Interest (ROIs). Rapid ASPECTS extracts image data for the ROI(s) to provide analysis and computer analytics based on morphological characteristics. The imaging features are then synthesized by an artificial intelligence algorithm into a single ASPECT (Alberta Stroke Program Early CT) Score. Rapid ASPECTS is indicated for evaluation of adult patients presenting for diagnostic imaging workup, for evaluation of extent of disease. Extent of disease refers to the number of ASPECTS regions affected which is reflected in the total score. This device provides information that may be useful in the characterization of early ischemic brain tissue injury for ischemic stroke patient (typically
The Rapid platform is Software as a Medical Device (SaMD), which provides for the visualization and study of changes in tissue and vasculature using digital images captured by diagnostic imaging systems including CT (Computed Tomography), CTA (CT Angiography), MRI (Magnetic Resonance Imaging) and MRA (MR Angiography) as an aid to physician diagnosis. Rapid can be installed on a customer's Server or it can be accessed online as a virtual system. It provides viewing, quantification, analysis, and reporting capabilities. The Rapid platform has multiple modules a clinician may elect to run and provide analysis for decision making.
Rapid ASPECTS provides an automatic ASPECT Score based on the case input file for the physician. The score includes which ASPECT regions are identified based on regional imaging features derived from Non-Contrast Computed Tomography (NCCT) brain image data. The results are generated based on the Alberta Stroke Program Early CT Score (ASPECTS) guidelines and provided to the clinician for review and verification. At the discretion of the clinician, the scores may be adjusted based on other clinical factors the clinician may integrate though the Rapid Platform Interface.
The ASPECTS software module processing pipeline performs four major tasks:
- Orientation and spatial normalization of the input imaging data (rigid registration/alignment with anatomical template).
- Delineation of pre-defined regions of interest on the normalized input data and computing numerical values characterizing underlying voxel values within those regions.
- Identification and highlighting previous/old stroke areas along with areas of early ischemic change; and
- Labeling of these delineated regions and providing a summary score reflecting the number of regions with early ischemic change as per ASPECTS guidelines.
Subsequently. the system notifies the physician of the availability of the ASPECT Score with an overlayed atlas. The ASPECTS information is then available for the physician to review and edit prior to sending the data to a PACS or Workstation. The final summary score together with the regions selected and underlying voxel values are then sent to the Picture Archiving and Communication System (PACS) to become a part of the permanent patient medical record.
The provided text describes the acceptance criteria and the study that proves the device meets those criteria for iSchemaView, Inc.'s Rapid ASPECTS (v3) CADx software.
1. Table of Acceptance Criteria and Reported Device Performance
Acceptance Criteria for Rapid ASPECTS (v3)
| Criterion | Reported Device Performance (Rapid ASPECTS v3) |
|---|---|
| Standalone Performance: Percent agreement of Rapid ASPECTS to the reference at the ASPECTS region level. | 82.8% |
| Standalone Performance: Percent agreement of Rapid ASPECTS to the reference at the scan level. | 82.8% (comparable, with overlapping CI, to pairwise agreement between any two of the three experts) |
| Clinical Validation Reader Improvement: Demonstrate that reader scoring of the 10 ASPECT regions is more closely aligned with the reference standard when read in conjunction with Rapid ASPECTS than without Rapid ASPECTS. | The fixed effect of the Rapid assist increases the percent agreement on average by about 0.02. Agreement increases from 82% without assistance to 84% with assistance (excluding the expert). The average agreement increases from 80.4% without assistance to 83.3% with assistance. A statistically significant improvement in the accuracy of the 6 readers' scores was demonstrated when scoring was performed with Rapid ASPECTS output. Most substantial benefit for non-neuroradiologist expert readers. No significant impact (positive or negative) on the expert neuroradiologist's score was observed. |
| Supplemental Confounder/Mimic Sensitivity Assessment: Assess impact of confounders/mimics. | Only 3 out of 115 reads (2.6%) changed based on Rapid results, showing minimal effect of confounders/mimics on ASPECTS performance. |
2. Sample Size Used for the Test Set and Data Provenance
- Standalone Performance Test Set Sample Size: 88 scans (from the "Suspected Stroke" category)
- Reader Improvement Test Set Sample Size: 102 scans (including 88 "Suspected Stroke" and 14 "Stroke Mimic" cases)
- Supplemental Confounder/Mimic Sensitivity Assessment Sample Size: This involved a separate set of supplemental data. While the number of scans directly used for this specific assessment is not explicitly stated as a single total, the types and counts of cases are listed: Abscess (3), Dural AVF (4), Hydrocephalus (4), Hypertensive Encephalopathy (2), Isodense SDH (4), Multiple Sclerosis (3), and Traumatic Brain Injury (3). These cases were reviewed for 115 reads.
- Data Provenance: The data included both US (79.41% for the reader improvement study test set) and OUS (20.59%) cases. It's a combination of different scanner manufacturers: GE (23), Siemens (28), Cannon/Toshiba (22), and Philips (29). The description suggests it is retrospective data, as it describes a collection of existing scans.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Their Qualifications
The ground truth for both the standalone performance and the reader improvement study was established using:
- Three experts to establish the reference standard for the standalone performance study.
- The clinical reader study involved one expert neuroradiologist and five non-expert typical readers. While the specific qualifications for "typical readers" aren't detailed, the text implies they represent general clinicians who evaluate CT scans in community hospitals and primary stroke centers. The neuroradiologist is explicitly identified as an expert.
4. Adjudication Method for the Test Set
The document explicitly states: "The primary reader improvement endpoint is to demonstrate that reader scoring of the 10 ASPECT regions is more closely aligned with the reference standard when read in conjunction with Rapid ASPECTS than without Rapid ASPECTS." And for standalone performance: "The percent agreement of Rapid ASPECTS to the reference at the ASPECTS region level and at the scan level is 82.8%. Both are comparable (overlapping CI) to the pairwise agreement between any two of the three experts."
This indicates that a reference standard was established by experts. While the specific method of reaching this reference standard (e.g., 2+1, consensus) is not explicitly detailed, the mention of "pairwise agreement between any two of the three experts" for the standalone performance suggests that the ground truth was derived from a consensus or adjudicated process involving these three experts.
5. Multi Reader Multi Case (MRMC) Comparative Effectiveness Study
Yes, a multi-reader multi-case (MRMC) comparative effectiveness study was done. This is referred to as the "Clinical Validation Reader Improvement" study.
- Effect Size of Human Readers Improve with AI vs. without AI assistance: The fixed effect of the Rapid assist increases the percent agreement on average by about 0.02. Specifically, agreement increases from 82% without assistance to 84% with assistance (excluding the expert). When including the non-expert readers, the average agreement increases from 80.4% without assistance to 83.3% with assistance.
- The benefit was most substantial among the non-neuroradiologist expert readers.
- The system allowed non-expert physicians to perform at an "expert-like level."
- There was no significant impact (positive or negative) on the score of the expert neuroradiologist.
6. Standalone Performance (i.e., algorithm only without human-in-the-loop performance)
Yes, a standalone performance study was done.
- Results: The percent agreement of Rapid ASPECTS to the reference at both the ASPECTS region level and at the scan level was reported as 82.8%. This was found to be comparable (with overlapping confidence intervals) to the pairwise agreement between any two of the three experts who established the ground truth.
7. The Type of Ground Truth Used
The ground truth used was expert consensus / expert reading. It was established by a panel of experts. The text refers to "the reference" established by "three experts" for the standalone performance and a "reference standard" for the reader improvement study.
8. The Sample Size for the Training Set
The document does not explicitly state the sample size for the training set. It only describes the test sets used for validation.
9. How the Ground Truth for the Training Set Was Established
As the training set sample size is not provided, the method for establishing its ground truth is also not specified in the provided text.
§ 892.2060 Radiological computer-assisted diagnostic software for lesions suspicious of cancer.
(a)
Identification. A radiological computer-assisted diagnostic software for lesions suspicious of cancer is an image processing prescription device intended to aid in the characterization of lesions as suspicious for cancer identified on acquired medical images such as magnetic resonance, mammography, radiography, or computed tomography. The device characterizes lesions based on features or information extracted from the images and provides information about the lesion(s) to the user. Diagnostic and patient management decisions are made by the clinical user.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Design verification and validation must include:
(i) A detailed description of the image analysis algorithms including, but not limited to, a detailed description of the algorithm inputs and outputs, each major component or block, and algorithm limitations.
(ii) A detailed description of pre-specified performance testing protocols and dataset(s) used to assess whether the device will improve reader performance as intended.
(iii) Results from performance testing protocols that demonstrate that the device improves reader performance in the intended use population when used in accordance with the instructions for use. The performance assessment must be based on appropriate diagnostic accuracy measures (
e.g., receiver operator characteristic plot, sensitivity, specificity, predictive value, and diagnostic likelihood ratio). The test dataset must contain sufficient numbers of cases from important cohorts (e.g., subsets defined by clinically relevant confounders, effect modifiers, concomitant diseases, and subsets defined by image acquisition characteristics) such that the performance estimates and confidence intervals of the device for these individual subsets can be characterized for the intended use population and imaging equipment.(iv) Standalone performance testing protocols and results of the device.
(v) Appropriate software documentation (
e.g., device hazard analysis; software requirements specification document; software design specification document; traceability analysis; and description of verification and validation activities including system level test protocol, pass/fail criteria, results, and cybersecurity).(2) Labeling must include:
(i) A detailed description of the patient population for which the device is indicated for use.
(ii) A detailed description of the intended reading protocol.
(iii) A detailed description of the intended user and recommended user training.
(iv) A detailed description of the device inputs and outputs.
(v) A detailed description of compatible imaging hardware and imaging protocols.
(vi) Warnings, precautions, and limitations, including situations in which the device may fail or may not operate at its expected performance level (
e.g., poor image quality or for certain subpopulations), as applicable.(vii) Detailed instructions for use.
(viii) A detailed summary of the performance testing, including: Test methods, dataset characteristics, results, and a summary of sub-analyses on case distributions stratified by relevant confounders (
e.g., lesion and organ characteristics, disease stages, and imaging equipment).