(133 days)
The AIMI-Triage CXR PTX Application is a notification-only triage workflow tool for use by hospital networks and clinics to identify and help prioritize chest X-rays acquired in the acute setting for review by hospital radiologists. The device operates in parallel to and independent of standard of care image interpretation workflow. Specifically, the device uses an artificial intelligence algorithm to analyze images for features suggestive of moderate to large sized pneumothorax; it makes caselevel output available to a PACS/workstation for worklist prioritization or triage. Identification of suspected cases of moderate to large sized pneumothorax is not for diagnostic use beyond notification.
The AIMI-Triage CXR PTX Application is limited to analysis of imaging data as a guide to possible urgency of adult chest X-ray image review, and should not be used in lieu of full patient evaluation or relied upon to make or confirm diagnoses. Notified radiologists are responsible for engaging in appropriate patient evaluation as per local hospital procedure before making care-related decisions or requests. The device does not replace review and diagnosis of the X-rays by radiologists. The device is not intended to be used with plain film X-rays.
The AIMI-Triage CXR PTX provides a chest X-ray prioritization service for use by radiologists to identify features suggestive of moderate to large sized pneumothorax. The artificial intelligence algorithm, trained via pattern recognition, processes each chest X-ray and flags those that appear to contain a moderate to large sized pneumothorax for urgent radiologist review. X-rays without an identified anomaly are placed in the worklist for routine review, which is the current standard of care. The user interface is minimal, consisting of the radiologist's existing picture archiving and communication system (PACS) viewer and worklist in which positively identified images are flagged by the software to notify of the suspected anomaly. Images are not marked or otherwise altered, and no diagnoses are provided.
The device does not have any direct accessories. However, it interacts with hospital communication and database systems in order to read and analyze cases in the worklist of the hospital's PACS system in order to identify suspected abnormal findings and transmit corresponding notifications to reflect its recommended prioritization of patient examinations for radiologist review. The software output is compatible with any PACS viewer and worklist.
Acceptance Criteria and Study Details for AIMI-Triage CXR PTX
1. Acceptance Criteria and Reported Device Performance
| Criteria | Accepted Performance Goal | Reported Device Performance |
|---|---|---|
| Overall AUC | Substantially equivalent to the predicate device, meeting a required performance goal (specific numerical value not explicitly stated, but implied to be in line with predicate's performance). | 0.967 (95% CI: [0.950, 0.984]) |
| Sensitivity | Not explicitly stated as an acceptance criterion, but device performance is reported. | 92% (95% CI: [86%, 96%]) and 90% (95% CI: [84%, 95%]) for unspecified categories/cohorts within the overall dataset. |
| Time to Analyze and Notify | Substantially equivalent to the predicate device (22.1 seconds). | 20.3 seconds |
| Performance by Dataset and Region | Not explicitly stated as acceptance criteria, but further detailed performance is provided for evaluation. | NIH (US): Sensitivity 97.6% (93.2,99.2), Specificity 90.8% (84.5,94.7), AUROC 0.987 (0.973,0.999) |
| PADCHEST (OUS): Sensitivity 85.3% (79.0,90.8), Specificity 89.7% (83.6,93.9), AUROC 0.949 (0.918,0.979) | ||
| Performance by Scanner Spatial Resolution | Not explicitly stated as acceptance criteria, but further detailed performance is provided for evaluation. | High range (0.170 lp/mm): Sensitivity 93.1% (84.8,98.3), Specificity 91.4% (80.7,96.5), AUROC 0.946 (0.911,0.980) |
2. Sample Size Used for the Test Set and Data Provenance
- Sample Size: 300 frontal chest X-rays (PA/AP).
- Data Provenance: Collected from US and OUS (Outside US) patients, representative of the intended population. The datasets are specifically identified as NIH (US) and PADCHEST (OUS). The study was retrospective.
- Patient Demographics: 168 (56%) male with mean age 51.6 years (SD=18.6, range 18-91), and 132 (44%) female with mean age 51.8 years (SD=16.2, range 23-86).
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications
- Number of Experts: 3 independent radiologists.
- Qualifications: US-board certified radiologists.
4. Adjudication Method for the Test Set
- The ground truth was established by 3 independent US-board certified radiologists.
- Each "Truther" involved in the ground truthing process was blinded to any other Truther's results, to any existing report, and to the results obtained by the AIMI-Triage CXR PTX software. This implies that the ground truth was established by individual assessment and likely involved a consensus or majority vote among the 3 radiologists, although the specific "2+1" or "3+1" method is not explicitly stated. The emphasis on independent and blinded assessment supports a robust adjudication process.
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
- It is not explicitly stated that an MRMC comparative effectiveness study was done to measure human reader improvement with AI assistance. The study focuses on standalone AI performance compared to a ground truth established by experts.
6. Standalone (Algorithm Only) Performance Study
- Yes, a standalone performance study was conducted. The "AIMI-Triage CXR PTX output was compared to the ground truth established by 3 independent US-board certified radiologists." This indicates the algorithm's performance without direct human-in-the-loop assistance during the evaluation.
7. Type of Ground Truth Used
- Expert Consensus: The ground truth was established by "3 independent US-board certified radiologists." While "consensus" isn't explicitly used, the involvement of multiple blinded experts suggests a robust process to define the true positive/negative cases.
8. Sample Size for the Training Set
- The document does not specify the sample size for the training set. It only mentions that the artificial intelligence algorithm was "trained via pattern recognition."
9. How the Ground Truth for the Training Set Was Established
- The document does not specify how the ground truth for the training set was established. It only indicates that the algorithm was "trained via pattern recognition."
§ 892.2080 Radiological computer aided triage and notification software.
(a)
Identification. Radiological computer aided triage and notification software is an image processing prescription device intended to aid in prioritization and triage of radiological medical images. The device notifies a designated list of clinicians of the availability of time sensitive radiological medical images for review based on computer aided image analysis of those images performed by the device. The device does not mark, highlight, or direct users' attention to a specific location in the original image. The device does not remove cases from a reading queue. The device operates in parallel with the standard of care, which remains the default option for all cases.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Design verification and validation must include:
(i) A detailed description of the notification and triage algorithms and all underlying image analysis algorithms including, but not limited to, a detailed description of the algorithm inputs and outputs, each major component or block, how the algorithm affects or relates to clinical practice or patient care, and any algorithm limitations.
(ii) A detailed description of pre-specified performance testing protocols and dataset(s) used to assess whether the device will provide effective triage (
e.g., improved time to review of prioritized images for pre-specified clinicians).(iii) Results from performance testing that demonstrate that the device will provide effective triage. The performance assessment must be based on an appropriate measure to estimate the clinical effectiveness. The test dataset must contain sufficient numbers of cases from important cohorts (
e.g., subsets defined by clinically relevant confounders, effect modifiers, associated diseases, and subsets defined by image acquisition characteristics) such that the performance estimates and confidence intervals for these individual subsets can be characterized with the device for the intended use population and imaging equipment.(iv) Stand-alone performance testing protocols and results of the device.
(v) Appropriate software documentation (
e.g., device hazard analysis; software requirements specification document; software design specification document; traceability analysis; description of verification and validation activities including system level test protocol, pass/fail criteria, and results).(2) Labeling must include the following:
(i) A detailed description of the patient population for which the device is indicated for use;
(ii) A detailed description of the intended user and user training that addresses appropriate use protocols for the device;
(iii) Discussion of warnings, precautions, and limitations must include situations in which the device may fail or may not operate at its expected performance level (
e.g., poor image quality for certain subpopulations), as applicable;(iv) A detailed description of compatible imaging hardware, imaging protocols, and requirements for input images;
(v) Device operating instructions; and
(vi) A detailed summary of the performance testing, including: test methods, dataset characteristics, triage effectiveness (
e.g., improved time to review of prioritized images for pre-specified clinicians), diagnostic accuracy of algorithms informing triage decision, and results with associated statistical uncertainty (e.g., confidence intervals), including a summary of subanalyses on case distributions stratified by relevant confounders, such as lesion and organ characteristics, disease stages, and imaging equipment.