(213 days)
Not Found
Not Found
Yes
The device description explicitly states that the GI GENIUS™ is an artificial intelligence/machine learning (AI/ML) device system.
No
The device is described as a "computer-assisted reading tool" designed to aid in the detection of lesions during endoscopy. It provides "superimposed markers" and is "not intended to replace clinical decision making." This indicates it is an assistive diagnostic tool, not a device that directly treats or provides therapy.
Yes
The device is designed to "aid endoscopists in detecting colonic mucosal lesions" and provides "superimposed markers (in the form of green boxes) that appear when a lesion is detected," which are functions consistent with a diagnostic aid.
No
The device description explicitly states that the GI GENIUS™ is an artificial intelligence/machine learning (AI/ML) device system comprised of software, hardware, and accessories. It also describes how the system is inserted into the video stream using SDI input and output, indicating a hardware component is involved in processing the video signal.
Based on the provided information, the GI Genius System is not an In Vitro Diagnostic (IVD).
Here's why:
- Intended Use: The intended use clearly states that the device is a "computer-assisted reading tool designed to aid endoscopists in detecting colonic mucosal lesions... in real time during standard white-light endoscopy examinations". It is used during a medical procedure (endoscopy) to assist the physician in interpreting visual information.
- Device Description: The device description explains that it takes a video stream from an endoscope and superimposes markers on the live video displayed to the user. This is a real-time image processing and display tool, not a device that analyzes samples in vitro (outside the body).
- Lack of In Vitro Analysis: There is no mention of the device analyzing biological samples (like tissue, blood, etc.) in a laboratory setting. The input is a video stream from an endoscope, which is an in vivo (within the body) imaging modality.
IVD devices are typically used to examine specimens derived from the human body to provide information for diagnosis, monitoring, or screening. The GI Genius System operates on live video data acquired during a procedure, directly assisting the physician in their visual assessment.
No
The letter does not explicitly state that the FDA has reviewed and approved or cleared a Predetermined Change Control Plan (PCCP) for this specific device. The provided text indicates "Not Found" under "Control Plan Authorized (PCCP) and relevant text".
Intended Use / Indications for Use
The GI Genius System is a computer-assisted reading tool designed to aid endoscopists in detecting colonic mucosal lesions (such as polyps and adenomas) in real time during standard white-light endoscopy examinations of patients undergoing screening and surveillance endoscopic mucosal evaluations. The GI Genius computer-assisted detection device is limited for use with standard white-light endoscopy imaging only. This device is not intended to replace clinical decision making.
Product codes
QNP
Device Description
The GI GENIUS™ is an artificial intelligence/machine learning (AI/ML) device system comprised of software, hardware, and accessories that is intended for polyp detection during standard white-light colonoscopy. The device system generates a video on the main endoscopy display that contains the original live video together with superimposed markers (in the form of green boxes) that appear when a lesion is detected.
The GI Genius takes the Serial Digital Interface (SDI) output stream from the video endoscope processor as an input and then generates an SDI output stream to the existing monitor/display system containing the original video stream with additional markers superimposed on it. In essence, the system is inserted into the video stream just prior to it being displayed to the user/operator. The technological characteristics of the GI Genius system are described below.
The GI Genius software, operating in the described hardware, is comprised of the major functional software components (modules) described below:
The software's architecture is module-based, with the following modules:
- Main Module: This module starts the main application components and initializes the video acquisition components.
- Video Capture Module: This module handles frame collection, video frame management (e.g., colorspace transformations, cropping, and resizing) and providing the frames to the detection module.
- Al/Detection Module: This module is responsible for identifying potential mucosal lesions. The main component consists of a convoluted neural network.
- Overlay Module: This module generates markers and superimposes them on the endoscopic video stream.
- Application Log Module: This module traces events, such as overlay activation/deactivation and software errors.
- GUI Handler Module: This module generates the menu user interface, in order to allow user actions such as volume regulation, field of view setting and to check the system status.
- Launcher Module: This module provides integrity checks on the files necessary for the correct execution of the software.
The device description included a list of compatible hardware video processors, compatible endoscope characteristics, and the software architecture description. This includes the convolutional neural network (CNN) of the AI/ML algorithm.
Mentions image processing
Yes
Mentions AI, DNN, or ML
Yes
Input Imaging Modality
White-light endoscopy
Anatomical Site
Colonic mucosal
Indicated Patient Age Range
The study enrolled subjects between 40 and 80 years of age.
Intended User / Care Setting
Endoscopists in a clinical setting (white-light endoscopy examinations).
Description of the training set, sample size, data source, and annotation protocol
The dataset used for the standalone performance testing was also used for algorithm training, and was originally from a study titled, "The Safety and Efficacy of Methylene Blue MMX® Modified Release Tablets Administered to Subjects Undergoing Screening or Surveillance Colonoscopy" [NCT01694966]. The multi-arm study included colonoscopy videos in which methylene blue was used (725 videos) and a control arm in which no methylene blue was used (480 videos). Training of the GI Genius AI software was done on a subset of videos in which polyps were identified, either in the presence or absence of methylene blue. To correct for bias in the data set due to training on methylene blue, a second fine-tuning training was performed on a subset of that same dataset, using only those videos without methylene blue. The fine-tuning training is not illustrated in the figure below. The training set included 568 subjects. The polyps visible in each frame were annotated by endoscopists.
Description of the test set, sample size, data source, and annotation protocol
The sponsor performed an independent validation or Holdout Testing using a total of 150 colonoscopy videos, without methylene blue. Of those videos, 105 included a total of 338 excised polyps with histology confirmation. The remaining 45 videos did not include polyps or lesions. The 150 colonoscopy videos had a total of 5,805,587 frames. To assess true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN), a reference standard was established. The standalone reference standard was created by having endoscopists review the video clips around all histologically confirmed polyps and placing an annotation box around the polyps visible in each frame.
Summary of Performance Studies (study type, sample size, AUC, MRMC, standalone performance, key results)
Standalone Performance Testing:
- Purpose: To demonstrate that the object-level, frame-level and overall algorithmic performance is sufficient to fulfill the indications for use of the GI Genius.
- Sample Size: 150 colonoscopy videos (105 with 338 excised polyps and 45 without polyps). Total of 5,805,587 frames.
- Key Results:
- Activation Time: GI Genius detected polyps on average 1270 ms (95% CI: 857 ms, 1684 ms) before the average endoscopist.
- Object-Level Performance: GI Genius detects about 82% of polyps before the average endoscopist detection time with about 156 false positive objects per colonoscopy exam.
- Frame-Level Performance:
- Logistic Regression Mixed Model: Overall TPR/Frame: 47.46% [42.51%; 52.45%], Overall FPR/Frame: 1.44% [1.27%; 1.63%]. AUC = 0.787 (95% CI: 0.755-0.817).
- Non-parametric Cluster Bootstrap Analysis: Overall TPR/Frame: 49.57% [45.24%; 54.06%], Overall FPR/Frame: 2.02% [1.72%; 2.35%]. AUC = 0.723 (95% CI: 0.684-0.762).
- The frame-based results show a ~45% frame-detection rate for diminutive ( 0 ms)
- Object-Level False Positives: 156.31 [135.61; 177.00] FP Objects/Patient (for markers persisting > 0 ms)
- Frame-Level True Positive Rate per Frame (TPR / FRAME):
- Logistic Regression Mixed Model: 47.46% [42.51%; 52.45%]
- Non-parametric Cluster Bootstrap Analysis: 49.57% [45.24%; 54.06%]
- Frame-Level False Positive Rate per Frame (FPR / FRAME):
- Logistic Regression Mixed Model: 1.44% [1.27%; 1.63%]
- Non-parametric Cluster Bootstrap Analysis: 2.02% [1.72%; 2.35%]
- Percentage of Polyps Detected (at least one frame): 99.70% (337/338) (both models)
Clinical Study (mITT population, adjusted estimates):
- Adenoma Detection Rate (ADR):**
- GI Genius: 55.1% [44.0%; 65.8%]
- Standard: 42.0% [31.3%; 53.4%]
- Adenomas per Colonoscopy (APC):**
- GI Genius: 0.81 [0.57; 1.15]
- Standard: 0.57 [0.39; 0.82]
- Positive Percent Agreement (PPA):
- GI Genius: 62.1% [43.4%; 77.8%]
- Standard: 65.2% [46.0%; 80.4%]
- Exploratory Endpoint, False Positive Rate (FPR) for resected/biopsied tissues:
- GI Genius: 0.9%
- Standard: 1.2%
Predicate Device(s)
Not Found
Reference Device(s)
Not Found
Predetermined Change Control Plan (PCCP) - All Relevant Information
Not Found
§ 876.1520 Gastrointestinal lesion software detection system.
(a)
Identification. A gastrointestinal lesion software detection system is a computer-assisted detection device used in conjunction with endoscopy for the detection of abnormal lesions in the gastrointestinal tract. This device with advanced software algorithms brings attention to images to aid in the detection of lesions. The device may contain hardware to support interfacing with an endoscope.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Clinical performance testing must demonstrate that the device performs as intended under anticipated conditions of use, including detection of gastrointestinal lesions and evaluation of all adverse events.
(2) Non-clinical performance testing must demonstrate that the device performs as intended under anticipated conditions of use. Testing must include:
(i) Standalone algorithm performance testing;
(ii) Pixel-level comparison of degradation of image quality due to the device;
(iii) Assessment of video delay due to marker annotation; and
(iv) Assessment of real-time endoscopic video delay due to the device.
(3) Usability assessment must demonstrate that the intended user(s) can safely and correctly use the device.
(4) Performance data must demonstrate electromagnetic compatibility and electrical safety, mechanical safety, and thermal safety testing for any hardware components of the device.
(5) Software verification, validation, and hazard analysis must be provided. Software description must include a detailed, technical description including the impact of any software and hardware on the device's functions, the associated capabilities and limitations of each part, the associated inputs and outputs, mapping of the software architecture, and a description of the video signal pipeline.
(6) Labeling must include:
(i) Instructions for use, including a detailed description of the device and compatibility information;
(ii) Warnings to avoid overreliance on the device, that the device is not intended to be used for diagnosis or characterization of lesions, and that the device does not replace clinical decision making;
(iii) A summary of the clinical performance testing conducted with the device, including detailed definitions of the study endpoints and statistical confidence intervals; and
(iv) A summary of the standalone performance testing and associated statistical analysis.
0
DE NOVO CLASSIFICATION REQUEST FOR GI GENIUS
REGULATORY INFORMATION
FDA identifies this generic type of device as:
Gastrointestinal lesion software detection system. A gastrointestinal lesion software detection system is a computer-assisted detection device used in conjunction with endoscopy for the detection of abnormal lesions in the gastrointestinal tract. This device with advanced software algorithms brings attention to images to aid in the detection of lesions. The device may contain hardware to support interfacing with an endoscope.
NEW REGULATION NUMBER: 21 CFR 876.1520
CLASSIFICATION: Class II
PRODUCT CODE: QNP
BACKGROUND
DEVICE NAME: GI Genius
SUBMISSION NUMBER: DEN200055
DATE DE NOVO RECEIVED: September 8, 2020
SPONSOR INFORMATION:
Cosmo Artificial Intelligence - AI, LTD Riverside II, Sir John Rogerson's Quay Dublin, Dublin 2 D02 KV60 Ireland
INDICATIONS FOR USE
The GI Genius is indicated as follows:
The GI Genius System is a computer-assisted reading tool designed to aid endoscopists in detecting colonic mucosal lesions (such as polyps and adenomas) in real time during standard white-light endoscopy examinations of patients undergoing screening and surveillance endoscopic mucosal evaluations. The GI Genius computer-assisted detection device is limited for use with standard white-light endoscopy imaging only. This device is not intended to replace clinical decision making.
LIMITATIONS
1
The sale, distribution, and use of the GI Genius are restricted to prescription use in accordance with 21 CFR 801.109.
The device is not intended to be used as a stand-alone diagnostic device.
The device is not intended to characterize lesions in a manner that would potentially replace biopsy sampling
The device is not intended to replace clinical decision making.
The device is not intended to be used with equipment that it was not tested against during validation activities.
The device has not been studied in patients with Inflammatory Bowel Disease (IBD), history of CRC, or previous colonic resection. The device performance may be negatively impacted by mucosal irregularities such as background inflammation from certain underlying disease.
PLEASE REFER TO THE LABELING FOR A COMPLETE LIST OF WARNINGS, PRECAUTIONS, AND CONTRAINDICATIONS.
DEVICE DESCRIPTION
The GI GENIUS™ is an artificial intelligence/machine learning (AI/ML) device system comprised of software, hardware, and accessories that is intended for polyp detection during standard white-light colonoscopy. The device system generates a video on the main endoscopy display that contains the original live video together with superimposed markers (in the form of green boxes) that appear when a lesion is detected.
Image /page/1/Picture/9 description: The image is a close-up shot of a colonoscopy, showing the inner lining of the colon. The colon appears to be healthy, with a smooth, pinkish surface. There is a green box around a polyp. The image also contains some text, including "EC-XAP-V/L" and "BL-X000".
Figure 1. Example of a colonoscopy image in which the device has detected a lesion (green box)
2
The GI Genius takes the Serial Digital Interface (SDI) output stream from the video endoscope processor as an input and then generates an SDI output stream to the existing monitor/display system containing the original video stream with additional markers superimposed on it. In essence, the system is inserted into the video stream just prior to it being displayed to the user/operator. The technological characteristics of the GI Genius system are described below.
Image /page/2/Figure/1 description: The image shows a medical cart with a monitor and other equipment on it. The cart has multiple shelves and is on wheels. There is text on the right side of the image that says "CB-17-08 compatible hardware" with an arrow pointing to one of the shelves on the cart.
Figure 10-1 - GI GENTUS Compatible Hardware on Video Endoscopy Trolly
Image /page/2/Picture/3 description: The image shows a white electronic device, possibly a set-top box or similar media player. The device has a rectangular shape with rounded edges and a matte finish. On the front panel, there are several circular buttons arranged in a pattern, along with a power button on the left side.
Figure 10-2 - GI GENIUS Front View
Image /page/2/Picture/5 description: The image shows the back of a white electronic device. The back panel has a power switch and an AC power input. There are also two BNC connectors labeled "DISPLAY". The back panel also has ventilation holes for cooling.
Figure 10-3 - GI GENIUS Rear View
Figure 2. Images of the GI Genius in relation to compatible hardware
The GI Genius software, operating in the described hardware, is comprised of the major functional software components (modules) described below:
The software's architecture is module-based, with the following modules:
3
· Main Module: This module starts the main application components and initializes the video acquisition components.
· Video Capture Module: This module handles frame collection, video frame management (e.g., colorspace transformations, cropping, and resizing) and providing the frames to the detection module.
· Al/Detection Module: This module is responsible for identifying potential mucosal lesions. The main component consists of a convoluted neural network.
· Overlay Module: This module generates markers and superimposes them on the endoscopic video stream.
· Application Log Module: This module traces events, such as overlay activation/deactivation and software errors.
· GUI Handler Module: This module generates the menu user interface, in order to allow user actions such as volume regulation, field of view setting and to check the system status.
· Launcher Module: This module provides integrity checks on the files necessary for the correct execution of the software.
The device description included a list of compatible hardware video processors, compatible endoscope characteristics, and the software architecture description. This includes the convolutional neural network (CNN) of the AI/ML algorithm.
SUMMARY OF NONCLINICAL/BENCH STUDIES
| Test | Purpose | Method | Acceptance Criteria | Results |
|---|---|---|---|---|
| Video delay | To assess the time | |||
| needed by the GI | ||||
| Genius to transmit the | ||||
| original colonoscopy | ||||
| video to the display | Measure the timing | |||
| difference between the | ||||
| original endoscopic video | ||||
| and the GI Genius output | ||||
| of the same frame | Time is less than or | |||
| equal to 5.75 | ||||
| milliseconds | (b) (4) | |||
| microseconds | ||||
| (b) (4) % | ||||
| Annotation | ||||
| delay | To determine the delay | |||
| in annotating the video | ||||
| with the annotation box | Timing diagram of the | |||
| frame capture, AI | ||||
| processing, AI overlay, and | ||||
| frame transmit pipeline | Time is less than or | |||
| equal to 120 | ||||
| milliseconds | (b) (4) or (b) (4) | |||
| milliseconds | ||||
| (b) (4) frames) | ||||
| Video quality | ||||
| integrity test | To assess that the image | |||
| quality did not degrade | ||||
| from the endoscopic | ||||
| video processor through | ||||
| the GI Genius and then | ||||
| the display | Pixel-wise comparison | |||
| between the original | ||||
| endoscopic video and the | ||||
| GI Genius video with the | ||||
| marker overlays | There should be no | |||
| degradation in image | ||||
| quality, meaning that | ||||
| all corresponding pixels | ||||
| (from the endoscopic | ||||
| video processor and | ||||
| from the GI Genius) are | ||||
| identical in the three | ||||
| color channels | The test data | |||
| show no pixel- | ||||
| level | ||||
| discrepancies | ||||
| except for the | ||||
| pixels overlaid | ||||
| with markers |
4
ELECTROMAGNETIC CAPABILITY (EMC) & ELECTROMAGNETIC SAFETY
The hardware components of GI Genius were tested per the FDA-recognized standards ANSI/AAMI/IEC 60601-1-2:2014 and IEC 60601-1:2005 + A1:2012 (Ed. 3.1). The results from the testing pass the acceptance criteria outlined in the EMC and Electromagnetic Safety standards. The device is electrically safe for use in its intended environment.
SOFTWARE/CYBERSECURITY
GI Genius was identified as having a moderate level of concern as defined in the FDA guidance document "Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices." The software documentation included:
- Software/Firmware Description 1.
-
- Device Hazard Analysis
-
- Software Requirement Specifications
- Architecture Design Chart 4.
- న్. Software Design Specifications
- Traceability 6.
- Software Development Environment Description 7.
-
- Verification and Validation Documentation
- Revision Level History 9.
-
- Unresolved Anomalies
Risk analysis was provided for the software with a description of the hazards, their causes and severity as well as acceptable methods for control of the identified risks. GI Genius provided a description, with test protocols including pass/fail criteria and report of results, of acceptable verification and validation activities at the unit, integration and system level.
Regarding the cybersecurity, the documentation included all the recommended information from the FDA guidance document "Content of Premarket Submissions for Management of Cybersecurity in Medical Devices." This includes a threat model, cybersecurity mitigation information, a malware-free shipping plan, an upgrade plan, and other information for safeguarding the algorithms.
PERFORMANCE TESTING - BENCH - STANDALONE PERFORMANCE
The purpose of the standalone performance testing is to demonstrate that the object-level. frame-level and overall algorithmic performance is sufficient to fulfill the indications for use of the GI Genius. This involves verification and validation of not only the software, but also additional performance testing of the algorithm alone to verify that it achieves acceptable detection performance, both overall and in important sub-populations. Standalone testing is also used to benchmark that performance as part of device labeling. The results provide an adequate benchmark for improved lesion detection and valid
5
scientific evidence that the chosen variable labels, features, and classifiers are sufficient to provide clinicians an aid for improved lesion detection.
The dataset used for the standalone performance testing was also used for algorithm training, and was originally from a study titled, "The Safety and Efficacy of Methylene Blue MMX® Modified Release Tablets Administered to Subjects Undergoing Screening or Surveillance Colonoscopy" [NCT01694966]. A diagram describing the use of the videos from that study is included in Figure 3 below. The multi-arm study included colonoscopy videos in which methylene blue was used (725 videos) and a control arm in which no methylene blue was used (480 videos). Training of the GI Genius AI software was done on a subset of videos in which polyps were identified, either in the presence or absence of methylene blue. To correct for bias in the data set due to training on methylene blue, a second fine-tuning training was performed on a subset of that same dataset, using only those videos without methylene blue. The fine-tuning training is not illustrated in the figure below.
In addition to algorithm training, the sponsor performed an independent validation or Holdout Testing using a total of 150 colonoscopy videos, without methylene blue. Of those videos, 105 included a total of 338 excised polyps with histology confirmation. The remaining 45 videos did not include polyps or lesions. The testing on these 150 colonoscopy videos is collectively referred to as standalone performance testing, and is separate from the clinical performance testing of the GI Genius that is described further below. The 150 colonoscopy videos had a total of 5,805,587 frames.
Image /page/5/Figure/3 description: This image shows a flowchart of the procedures used in a study. The study started with 1205 procedures, which were divided into two groups: Methylene Blue (MMX) Arms (725) and Control (no Methylene Blue) (480). The MMX group had 516 recorded videos and polyps, while the control group had 324. After data randomization, the MMX group was divided into Training (Polyps+ MMX) (349) and Training (Polyps+ No MMX) (219), while the control group was divided into Testing (Polyps + No MMX) (105) and Testing (No Polyps + No MMX) (45).
Figure 3. Diagram outlining the curation of the training and validation data
6
The patient demographics of the dataset used for training and the Holdout Test Set are provided in Table 1.
Table 1. Demographics information of MMX trial.
| | Training
(568 subjects) | Holdout Test Set
(150 subjects) | Overall
(718 subjects) |
|-------------------------------------------|----------------------------|------------------------------------|---------------------------|
| Mean Age, years (SD) | 61.6 (6.58) | 61.5 (6.32) | 61.6 (6.59) |
| Sex, N (%) | | | |
| Male | 370 (65.1%) | 93 (62.0%) | 463 (64.5%) |
| Female | 198 (34.9%) | 57 (38.0%) | 255 (35.5%) |
| Indication for Colonoscopy, N (%) | | | |
| Screening | 270 (47.5%) | 73 (46.7%) | 343 (47.8%) |
| Surveillance ≤ 2 years | 43 (7.6%) | 7 (4.7%) | 50 (7.0%) |
| Surveillance > 2 years | 255 (44.9%) | 70 (48.7%) | 325 (45.3%) |
| Race/Ethnicity | | | |
| White or Caucasian | 522 (91.9%) | 141 (94.0%) | 663 (92.3%) |
| Black or African American | 34 (6.0%) | 5 (3.3%) | 39 (5.4%) |
| Hispanic or Latino | 7 (1.2%) | 0 (0%) | 7 (1.0%) |
| Asian | 3 (0.5%) | 3 (2.0%) | 6 (0.8%) |
| Native Hawaiian or other Pacific Islander | 1 (0.2%) | 1 (0.7%) | 2 (0.3%) |
The most relevant characteristics of the lesions used in the standalone testing are reported in Figure 4. The charts below also show a wide distribution of the polyps that are meant to be detected. Approximately half of the 338 lesions in the Holdout Test Set were confirmed adenomas, and half were non-adenomas. The lesions were found throughout the colon, from the cecum to the rectum. Less than two-thirds of lesions had polypoid morphology, and the remaining had non-polypoid morphology. Approximately 70% of lesions were diminutive (less than 5 mm), approximately 20% were small (6-9 mm) and about 10% were considered to be large polyps (≥10 mm).
7
Image /page/7/Figure/0 description: The image contains four bar charts displaying data from a holdout test set. The first chart, titled "Holdout Test Set - Morphology," shows that 63.6% of the polyps were polypoid and 36.4% were non-polypoid. The second chart, "Holdout Test Set - Polyp Anatomical Location," shows the distribution of polyps across different anatomical locations: Cecum (7.4%), Ascending (18.9%), Transverse (20.1%), Descending (13.0%), Sigmoid (23.7%), and Rectum (16.9%). The third chart, "Holdout Test Set - Histology Characteristics," shows that 48.2% of the polyps were adenomas, 48.5% were non-adenomas, and 3.3% were not available. The fourth chart, "Holdout Test Set - Polyp Size," shows that 70.4% of the polyps were diminutive, 19.8% were small, and 9.8% were large.
Figure 4. Charts summarizing the characteristics of the polyps used as part of the standalone performance dataset.
Standalone performance on the Holdout Test Set contained multiple elements including an assessment of the algorithm's activation time followed by an assessment of both object- and frame-level detection performance in terms of sensitivity, false positive rate and receiver operating characteristic (ROC) performance.
To assess true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN), a reference standard was established. The standalone reference standard was created by having endoscopists review the video clips around all histologically confirmed polyps and placing an annotation box around the polyps visible in each frame. Those same video clips (without annotation) were analyzed by the GI Genius device; the device placed a marker on each frame in which the device identified a lesion. An assessment was then conducted to analyze the overlap between the endoscopists' annotation of lesions and the GI Genius marker for lesions, using an Intersection over Union (IoU) criterion, which is a metric of object detector accuracy.
Activation Time
8
The activation time refers to the time required for the device to detect a lesion and display a computer-aided detection (CAD) marker. For the device to function as intended, the device must detect and mark a lesion as it entered the field of view before the endoscopist identifies the lesion and before it exits the field of view.
To establish when the lesion is in this critical time frame, a panel of five expert endoscopists reviewed all the video clips containing polyps of the Holdout Test Set, along with an additional set of sham videoclips showing no polyps, in random order and recorded the moment of their first detection after accounting for endoscopists' base reaction time. GI Genius was found on average to have an activation time of 120 ms, which means that it detected a polyp 1270 ms (95% CI: 857 ms, 1684 ms) before the average endoscopist in this study. This result met the acceptance criterion of the device initializing and detecting a lesion faster than the reaction time of the endoscopists.
Object-Level Performance
The purpose of the object-level performance test is to measure the accuracy between the endoscopists' object detection and the GI Genius marker. To define when polyp detection is providing a true benefit, an experiment was first carried out to establish the offset in each excised polyp video-clip where the endoscopist did first spot the lesion. The endoscopist reviewed 338 video clips of the polyps after estimating each readers' Baseline Reaction Time using 15 calibration videoclips with only the last 10 used to estimate reaction time as a correction to the activation time. Each expert endoscopist reviewed all the videoclips of the activation time dataset in addition to 49 60-seconds sham videoclips showing no polyps. The endoscopist played each videoclip and stopped the playback when a polyp was detected. Then, the endoscopist had to localize the polyp. If the position was incorrect (based on a location 0 | 156.31
[135.61; 177.00] | 81.96%
[77.35%; 85.97%] | |
| > 100 ms | 65.00
[54.92;75.08] | 70.03%
[64.75%; 74.95%] | |
| > 200 ms | 33.09
[27.12; 39.06] | 59.33%
[53.79%; 64.70%] | |
| > 300 ms | 19.47
[15.46; 23.49] | 48.62%
[43.09%; 54.19%] | |
| > 400 ms | 14.09
[10.96; 17.22] | 42.81%
[37.38%; 48.37%] | |
| > 500 ms | 9.89
[7.50; 12.28] | 35.47%
[30.29; 40.93%] | |
| > 1000 ms | 3.92
[2.71; 5.13] | 20.80%
[16.53%; 25.6] | |
| > 1500 ms | 2.03
[1.34; 2.73] | 12.84%
[9.42%; 16.96%] | |
| > 2000 ms | 1.25
[0.80; 1.70] | 10.40%
[7.31%; 14.23%] | |
10
Image /page/10/Figure/0 description: The image is a plot of sensitivity versus false positive objects per patient. The x-axis represents the number of false positive objects per patient, ranging from 0 to 180. The y-axis represents sensitivity, ranging from 0.0 to 1.0. The plot shows a curve that starts at the origin and increases as the number of false positive objects per patient increases, with labels indicating persistence values at different points along the curve.
Figure 5. Graphical representation of Table 2, demonstrating the persistence of polyp-based Sensitivity and the number False Positive objects
The object-level performance shows that the GI Genius detects about 82% of the polyps before the average endoscopist detection time with about 156 false positive objects per colonoscopy exam. The number of false positive objects and true positive objects decreases as the length of time a target is marked increases. Many of the marks appear for a relatively small number of frames.
Frame-Level Performance
The frame-level performance is an assessment of the accuracy of the algorithm at sorting endoscopic images for quantification of false positives, false negatives, true positives, and true negatives. The sponsor expected a false positive rate per frame of 4.85% or lower based on an estimation from the video database in the Methylene Blue clinical trial.
Tables 3 and 4 summarize the performances of GI Genius when analyzing all 5,805,587 frames as individual statistical events in the Holdout Test Set calculated using two different statistical methods: Logistic regression mixed model analysis and Nonparametric cluster bootstrap analysis. Two different statistical methods were used for analysis, because the two models showed different performance levels between the two different analyses, and there is insufficient information to determine which analysis method is most appropriate.
11
The following definitions apply:
- True Positive Rate per Frame (TPR / FRAME) is the proportion of frames ● containing a polyp that were correctly detected by GI Genius;
- . False Positive Rate per Frame (FPR / FRAME) is the proportion of frames not containing a polyp in which GI Genius did show a detection.
Table 3. Logistic Regression Mixed Model, with lesion random model for TPR and patient random model for FPR
| Category | | Mean Rate
[95% Confidence Interval] | Percentage
of Polyps
Detected | Number
of Videos |
|-----------------------------------|-----------------------------|-----------------------------------------|-------------------------------------|---------------------|
| Overall | | TPR / Frame: 47.46%
[42.51%; 52.45%] | 99.70%
(337/338) | 105 |
| Histology | Adenoma | 57.59%
[50.62%; 64.26%] | 99.39%
(162/163) | 69 |
| | Non-
Adenoma | 38.68%
[32.25%; 45.52%] | 100.00%
(163/163) | 79 |
| | Unknown | 32.04%
[14.30%; 57.11%] | 100.00%
(12/12) | 9 |
| Lesion
Size | Diminutive
(0-5 mm) | 44.54%
[38.88%; 50.35%] | 99.60%
(237/238) | 92 |
| | Small
(6-9 mm) | 64.95%
[54.44%; 74.18%] | 100.00%
(67/67) | 42 |
| | Large
(≥10 mm) | 32.92%
[20.76%; 47.89%] | 100.00%
(33/33) | 25 |
| Compatible
Video
Processors | Olympus
CV-180 | 54.13%
[46.45%; 61.62%] | 99.28%
(137/138) | 44 |
| | Olympus
CV-190 | 42.6%
[33.79%; 51.91%] | 100.00%
(93/93) | 28 |
| | Pentax
EPK-i7000 | 25.31%
[10.58%; 49.26%] | 100.00%
(12/12) | 5 |
| | Fujifilm
VP-4450HD | 45.9%
[33.95%; 58.34%] | 100.00%
(52/52) | 16 |
| Overall | | FPR / Frame 1.44%
[1.27%; 1.63%] | N/A | 150 |
| Compatible
Video
Processors | Olympus
CV-180 | 1.8%
[1.50%; 2.16%] | N/A | 63 |
| | Olympus
CV-190 | 1.26%
1.01%; 1.57%] | N/A | 44 |
| | Pentax
EPK-i7000 | 1.45%
[0.84%; 2.49%] | N/A | 7 |
| Fujifilm
VP-4450HD | $0.85%$
$[0.62%; 1.15%]$ | N/A | 23 | |
12
- In this table, a polyp is considered detected if the GI Genius bounding box (overlay marker) adequately overlaps with the reference standard bounding box in at least one frame.
| Table 4. Non-parametric Cluster Bootstrap analysis, considering within-patient correlation | |||
|---|---|---|---|
| Category | Mean Rate
[95% Confidence Interval] | Percentage
of Polyps Detected | Number
of Videos | |
|-----------------------------------|----------------------------------------|----------------------------------|----------------------|----|
| Overall | TPR / Frame 49.57%
[45.24%; 54.06%] | 99.70%
(337/338) | 105 | |
| Histology | Adenoma | 55.24%
[48.38%; 62.50%] | 99.39%
(162/163) | 69 |
| | Non-Adenoma | 43.98%
[38.25%; 49.91%] | 100.00%
(163/163) | 79 |
| | Unknown | 53.63%
[31.68%; 73.87%] | 100.00%
(12/12) | 9 |
| Lesion Size | Diminutive
(0-5 mm) | 45.91%
[41.22%; 50.92%] | 99.60%
(237/238) | 92 |
| | Small
(6-9 mm) | 61.65%
[53.87%; 69.75%] | 100.00%
(67/67) | 42 |
| | Large
(≥10 mm) | 50.18%
[35.77%; 61.40%] | 100.00%
(33/33) | 25 |
| Compatible
Video
Processors | Olympus
CV-180 | 55.44%
[48.63%; 62.38%] | 99.28%
(137/138) | 44 |
| | Olympus
CV-190 | 42.44%
[36.14%; 50.43%] | 100.00%
(93/93) | 28 |
| | Pentax
EPK-i7000 | 39.93%
[21.07%; 63.06%] | 100.00%
(12/12) | 5 |
| | Fujifilm
VP-
4450HD | 47.71%
[40.88%; 59.24%] | 100.00%
(52/52) | 16 |
| Overall | FPR / Frame 2.02%
[1.72%; 2.35%] | N/A | 150 | |
| Compatible
Video
Processors | Olympus
CV-180 | 2.22%
[1.83%; 2.66%] | N/A | 63 |
| | Olympus
CV-190 | 1.80%
[1.28%; 2.49%] | N/A | 44 |
| | Pentax
EPK-i7000 | 1.89%
[0.97%; 3.17%] | N/A | 7 |
| Fujifilm
VP-
4450HD | 1.25%
[0.76%; 1.98%] | N/A | 23 | |
13
- In this table, a polyp is considered detected if the GI Genius bounding box (overlay marker) adequately overlaps with the reference standard overlay marker in at least one frame.
The algorithm Receiver Operating Characteristic (ROC) curve, Area Under the Curve (AUC) and 95% confidence interval are shown in Figure 6, calculated with two different statistical methods:
- On the left: using a Logistic Regression Mixed Model, with lesion random model . for TPR and patient random model for FPR
- . On the right: using a Non-Parametric Cluster Bootstrap analysis, considering within-patient correlation
Image /page/13/Figure/5 description: The image contains two ROC curves titled "Frame-Based TPr/FPr ROC Curve". The left curve has an AUC of 0.787 with a 95% confidence interval of 0.755-0.817. The right curve has an AUC of 0.723 with a 95% confidence interval of 0.684-0.762. Both curves plot TPr on the y-axis and FPr on the x-axis.
Figure 6. ROC and AUC curves for the TPR and FPR for the two models.
The frame-level performance shows that the device performs adequately for polyp detection. The overall false positive rate for the GI Genius was 1.44% and 2.02% per each of the frames in the Logistic Regression Mixed Model and Non-Parametric Cluster Bootstrap model, respectively. The results also demonstrate the TP and FP rates with a variety of endoscope video processors, although caution should be applied when interpreting the results when there is a small sample size. The frame-based results show a ~45% frame-detection rate for diminutive ( 2 in any colonic segment), previous colonic resection, or antithrombotic therapy precluding polyp resection. The primary analyses population (mITT, or modified Intentto-Treat), which constituted the basis for the assessment of efficacy and safety of GI Genius was a subset of that population. The mITT population was limited to patients at low risk for CRC, i.e.
17
patients undergoing colonoscopy for primary screening of CRC or for surveillance within 3 to 10 years from previous colonoscopy. Limiting analysis to subjects at low risk for cancer is more likely to obtain consistency in the data and the two arms of the study, because even a few patients with high risk of cancer may have large numbers of polyps that can skew the results in that arm of the study. Furthermore, given that the prevalence of polyps in the low risk population is expected to be lower than in the high risk population (and, therefore, more difficult to detect), the assessment of performance in the low risk population is considered a "worst-case" testing scenario.
The mITT group comprised 263 patients in total, of whom 136 were randomized to GI Genius+colonoscopy and 127 were randomized to standard colonoscopy. Results of the study are presented only for the mITT group of subjects at low risk for CRC. The demographics of this low-risk group of patients is reported in the table below.
| | GI Genius
(136 subjects) | Standard
Colonoscopy
(127 subjects) | Overall
(263 subjects) |
|---------------------------------------------------------------------------------------------------|-----------------------------|-------------------------------------------|---------------------------|
| Mean Age, years (SD) | 60.6 (9.74) | 59.9 (11.18) | 60.3 (10.13) |
| Sex, N (%) | | | |
| Male | 73 (53.7%) | 62 (48.8%) | 135 (51.3%) |
| Female | 63 (46.3%) | 65 (51.2%) | 128 (48.7%) |
| Adequate bowel cleansing (total score ≥6 and no
score 98%) Caucasian. It is assumed that the majority of study participants are Caucasian. We expect the performance as an aid to adenoma detection to be comparable in a US population, but the racial/ethnicity difference is an area of uncertainty and discussed further below in the Benefit-Risk Determination Section.
Study Results
In the original AID study, with the entire Intent to Treat population (n=700), the GI Genius+colonoscopy arm met the pre-specified 10% non-inferiority margin and subsequent superiority analysis for ADR, and demonstrated superiority for APC and 15% non-inferiority margin for PPA, compared to standard colonoscopy.
Re-analysis of the clinical study to limit the patient population to the mITT population (patients at low risk for CRC) and ADR*, ADR, APC*, APC, and PPA results are shown below.
a) Primary Endpoint - Adenoma Detection Rate (ADR*)
The ADR* was analyzed in the mITT set through a logistic regression mixed model, with treatment group, age (