(199 days)
The AEYE-DS device is indicated for use by health care providers to automatically detect more than mild diabetic retinopathy (mtmDR) in adults diagnosed with diabetes who have not been previously diagnosed with diabetic retinopathy. The AEYE-DS is indicated for use with the Topcon NW400.
AEYE-DS is a retinal diagnostic software device that incorporates an algorithm to evaluate ophthalmic images for diagnostic screening to identify retinal diseases or conditions. Specifically, the AEYE-DS is designed to perform diagnostic screening for the condition of more-than-mild diabetic retinopathy (mtmDR).
The AEYE-DS is comprised of 5 software components: (1) Client; (2) Service; (3) Analytics; (4) Reporting and Archiving; and (5) System Security.
The AEYE-DS device is based on the main technological principle of Artificial Intelligence (AI) software as a medical device. The software as a medical device uses artificial intelligence technology to analyze specific disease features from fundus retinal images for diagnostic screening of diabetic retinopathy.
The AEYE-DS device is based on the principle of operation, whereby a fundus camera is used to obtain retinal images. The fundus camera is attached to a computer, where the Client module/software is installed. The Client module/software guides the user to acquire the images and enables the user to interact with the server-based analysis software over a secure internet connection. Using the Client module/software, users identify the fundus images per eye to be dispatched to the Service module/software. The Service module/software is installed on a server hosted at a secure datacenter, receives the fundus images and transfers them to the Analytics module/software. The Analytics module/software, which runs alongside the Service module/software, processes the fundus images and returns information on the image quality and the presence or absence of mtmDR to the Service module/software. The Service module/software then returns the results to the Client module/software.
Here's a breakdown of the acceptance criteria and the study that proves the AEYE-DS device meets them, based on the provided text:
1. Table of Acceptance Criteria and Reported Device Performance
The pivotal clinical study evaluated two configurations: 1 image per eye (macula-centered) and 2 images per eye (macula-centered and optic disc-centered). The acceptance criteria for both sensitivity and specificity were pre-defined performance goals.
Acceptance Criteria and Performance (1 Image Per Eye)
| Metric | Acceptance Criteria (Lower One-Sided 97.5% CI Bound) | Reported Device Performance (Lower One-Sided 97.5% CI Bound) | Met? |
|---|---|---|---|
| Sensitivity | ≥ 82% | 83.3% | Yes |
| Specificity | ≥ 87% | 88.22% | Yes |
Acceptance Criteria and Performance (2 Images Per Eye)
| Metric | Acceptance Criteria (Lower One-Sided 97.5% CI Bound) | Reported Device Performance (Lower One-Sided 97.5% CI Bound) | Met? |
|---|---|---|---|
| Sensitivity | ≥ 82% | 85.63% | Yes |
| Specificity | ≥ 87% | 85.18% | No |
Additional Performance Metrics (for both 1 and 2 images per eye)
| Metric | 1 Image Per Eye Performance | 2 Images Per Eye Performance |
|---|---|---|
| Imageability | 99.1% [CI: 97.8%; 99.7%] | 99.1% [CI: 97.8%; 99.7%] |
| PPV | 60.23% [CI: 49.78%; 69.82%] | 54% [CI: 44.26%; 63.44%] |
| NPV | 98.93% [CI: 97.28%; 99.58%] | 99.17% [CI: 97.59%; 99.72%] |
Note: While the specificity for 2 images per eye was slightly below the pre-defined performance goal, the document states that this "does not involve any risks" as sensitivity was high and mtmDR+ subjects would not be missed.
2. Sample Size and Data Provenance
- Test Set Sample Size:
- Pivotal Clinical Study: 531 subjects screened and enrolled.
- For the 1 image per eye analysis, there were 57 mtmDR+ and 405 mtmDR- fully analyzable subjects. The total number of fully analyzable subjects is 462.
- For the 2 images per eye analysis, the exact number of fully analyzable subjects is not explicitly stated in the summary, but the sensitivity and specificity values are provided for a certain number of images, suggesting the same or a very similar subject pool.
- Precision Study: 22 participants.
- Pivotal Clinical Study: 531 subjects screened and enrolled.
- Data Provenance: Prospective, multi-center, single-arm, blinded study conducted at 8 study sites in the United States (7 sites) and Israel (1 site). Enrollment from October 2020 through November 2021.
3. Number of Experts and Qualifications for Ground Truth
- Number of Experts: Not explicitly stated as a number of individual experts. The professional images (dilated four widefield stereo color fundus images, lens photography, and macular OCT) were sent to an "independent reading center."
- Qualifications of Experts: The reading center determined the severity of retinopathy and diabetic macular edema (DME) according to the Early Treatment for Diabetic Retinopathy Study (ETDRS) severity scale. This implies that the experts were highly qualified in retinal imaging and diabetic retinopathy grading, typically ophthalmologists or trained graders with specific expertise in ETDRS.
4. Adjudication Method for the Test Set
The document does not explicitly describe an adjudication method like 2+1 or 3+1. It states that "The Reading Center diagnostic results formed the reference standard (ground truth) for the study." This suggests that the Reading Center's determination was considered the definitive ground truth, implying a consensus or expert-driven process within the center to establish this standard.
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
No, a multi-reader multi-case (MRMC) comparative effectiveness study was not done. The study focused on the standalone performance of the AEYE-DS device against an expert-determined ground truth, not on how human readers' performance might improve with AI assistance.
6. Standalone Performance Study (Algorithm Only)
Yes, a standalone (algorithm only) performance study was conducted. The "Clinical Performance Data" section describes how the AEYE-DS device automatically processed fundoscopy images and produced a diagnostic result ("more than mild DR (mtmDR) detected" or "more than mild DR not detected"). These results were then compared to the "reference standard (ground truth)" established by the independent reading center, directly assessing the algorithm's performance without human intervention in the diagnosis.
7. Type of Ground Truth Used
The ground truth used was expert consensus / expert reading of multi-modal imaging data. Specifically, it was established by an independent reading center based on:
- Dilated four widefield stereo color fundus images.
- Lens photography for media opacity assessment.
- Macular optical coherence tomography (OCT) imaging.
- Severity of retinopathy and DME determined according to the Early Treatment for Diabetic Retinopathy Study (ETDRS) severity scale.
8. Sample Size for the Training Set
The document does not explicitly state the sample size for the training set. The clinical study described is the pivotal clinical study for validation, not the training of the AI model.
9. How the Ground Truth for the Training Set Was Established
The document does not explicitly state how the ground truth for the training set was established, as it focuses on the performance claims from the pivotal clinical study. However, given that it's an AI/ML device, it can be inferred that a similar process of expert grading of images would have been used for the training data, likely by ophthalmologists or trained graders applying recognized clinical standards (e.g., ETDRS).
§ 886.1100 Retinal diagnostic software device.
(a)
Identification. A retinal diagnostic software device is a prescription software device that incorporates an adaptive algorithm to evaluate ophthalmic images for diagnostic screening to identify retinal diseases or conditions.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Software verification and validation documentation, based on a comprehensive hazard analysis, must fulfill the following:
(i) Software documentation must provide a full characterization of technical parameters of the software, including algorithm(s).
(ii) Software documentation must describe the expected impact of applicable image acquisition hardware characteristics on performance and associated minimum specifications.
(iii) Software documentation must include a cybersecurity vulnerability and management process to assure software functionality.
(iv) Software documentation must include mitigation measures to manage failure of any subsystem components with respect to incorrect patient reports and operator failures.
(2) Clinical performance data supporting the indications for use must be provided, including the following:
(i) Clinical performance testing must evaluate sensitivity, specificity, positive predictive value, and negative predictive value for each endpoint reported for the indicated disease or condition across the range of available device outcomes.
(ii) Clinical performance testing must evaluate performance under anticipated conditions of use.
(iii) Statistical methods must include the following:
(A) Where multiple samples from the same patient are used, statistical analysis must not assume statistical independence without adequate justification.
(B) Statistical analysis must provide confidence intervals for each performance metric.
(iv) Clinical data must evaluate the variability in output performance due to both the user and the image acquisition device used.
(3) A training program with instructions on how to acquire and process quality images must be provided.
(4) Human factors validation testing that evaluates the effect of the training program on user performance must be provided.
(5) A protocol must be developed that describes the level of change in device technical specifications that could significantly affect the safety or effectiveness of the device.
(6) Labeling must include:
(i) Instructions for use, including a description of how to obtain quality images and how device performance is affected by user interaction and user training;
(ii) The type of imaging data used, what the device outputs to the user, and whether the output is qualitative or quantitative;
(iii) Warnings regarding image acquisition factors that affect image quality;
(iv) Warnings regarding interpretation of the provided outcomes, including:
(A) A warning that the device is not to be used to screen for the presence of diseases or conditions beyond its indicated uses;
(B) A warning that the device provides a screening diagnosis only and that it is critical that the patient be advised to receive followup care; and
(C) A warning that the device does not treat the screened disease;
(v) A summary of the clinical performance of the device for each output, with confidence intervals; and
(vi) A summary of the clinical performance testing conducted with the device, including a description of the patient population and clinical environment under which it was evaluated.