(120 days)
Koios DS is an artificial intelligence (Al)/machine learning (ML)-based computer-aided diagnosis (CADx) software device intended for use as an adjunct to diagnostic ultrasound examinations of lesions or nodules suspicious for breast or thyroid cancer.
Koios DS allows the user to select or confirm regions of interest (ROIs) within an image representing a single lesion or nodule to be analyzed. The software then automatically characterizes the selected image data to generate an AI/ML-derived cancer risk assessment and selects applicable lexicon-based descriptors designed to improve overall diagnostic accuracy as well as reduce interpreting physician variability.
Koios DS may also be used as an image viewer of multi-modality digital images, including ultrasound and mammography. The software includes tools that allow users to adjust, measure and document images, and output into a structured report.
Koios DS software is designed to assist trained interpreting physicians in analyzing the breast ultrasound images of adult (>= 22 years) female patients with soft tissue breast lesions and/or thyroid ultrasounds of all adult (>= 22 years) patients with thyroid nodules suspicious for cancer. When utilized by an interpreting physician who has completed the prescribed training, this device provides information that may be useful in recommending appropriate clinical management.
Limitations:
· Patient management decisions should not be made solely on the results of the Koios DS analysis.
· Koios DS software is not to be used for the evaluation of normal tissue, on sites of post-surgical excision, or images with doppler, elastography, or other overlays present in them.
· Koios DS software is not intended for use on portable handheld devices (e.g. smartphones or tablets) or as a primary diagnostic viewer of mammography images.
• The software does not predict the thyroid nodule margin descriptor, extra-thyroidal extension. In the event that this condition is present, the user may select this category manually from the margin descriptor list.
Koios DS is a software application designed to assist trained interpreting physicians in analyzing breast and thyroid ultrasound images. The software device is a web application that is deployed to a Microsoft IIS web server and accessed by a user through a compatible client. Once logged in and granted access to the Koios DS application, the user examines selected breast or thyroid ultrasound DICOM images. The user selects Regions of Interest (ROls) of orthogonal views of a breast lesion or thyroid nodule for processing by Koios DS. The ROI(s) are transmitted electronically to the Koios DS server for image processing and the results are returned to the user for review.
The Koios Medical, Inc. Koios DS device is an AI/ML-based computer-aided diagnosis (CADx) software that assists in the analysis of breast and thyroid ultrasound images.
Here's an overview of its acceptance criteria and the studies proving it meets them:
Acceptance Criteria and Reported Device Performance
| Criteria (Metric) | Acceptance Criteria (Target) | Reported Device Performance (Koios DS) |
|---|---|---|
| Breast Functionality | (Based on predicate device K190442 performance) | |
| System AUC (Standalone) | Not explicitly stated as a minimum threshold, but improvement expected over predicate. | 0.929 [0.913, 0.945 95% CI] (on 900 cases) |
| Compared to predicate (Koios DS Breast v2.0): Significant increase in AUC (5%), no change in sensitivity, significant increase in specificity (24%). | ||
| 0.930 [0.914, 0.946 95% Cl] (on 50 additional cases, demonstrating robustness to dataset drift). | ||
| System Sensitivity (Standalone) | Not explicitly stated as a minimum threshold. | 0.97 [0.96, 0.99] |
| System Specificity (Standalone) | Not explicitly stated as a minimum threshold. | 0.61 [0.57, 0.66] |
| Reader AUC Improvement (MRMC) | Significant improvement in AUC with Koios DS assistance. | 0.0370 [0.030, 0.044] (mean AUC improvement at α = .05) from an earlier study (K190442). The subject device's updated breast engine showed superior standalone performance, implying equivalent or greater benefit in reader performance. |
| Inter-operator Variability | Reduction in variability. | Average Kendall Tau-B of USE + DS was 0.6797 [0.6653, 0.6941] compared to USE Alone at 0.5404 [0.5301, 0.5507], demonstrating a significant increase (reduction in variability). |
| Intra-operator Variability | Reduction in variability. | USE + DS class switching rate was 10.8% compared to USE Alone at 13.6% (p = 0.042), demonstrating a statistically significant reduction. |
| Thyroid Functionality | (New functionality, establishing performance thresholds) | |
| System AUC (Standalone) | Not explicitly stated as a minimum threshold, but acceptable performance. | 0.798 when applied to ACR TI-RADS guidelines. |
| System Sensitivity (Standalone) (Biopsy recommendation) | Not explicitly stated as a minimum threshold. | 0.644 [0.545, 0.744] |
| System Specificity (Standalone) (Biopsy recommendation) | Not explicitly stated as a minimum threshold. | 0.612 [0.566, 0.658] |
| Reader AUC Improvement (MRMC) (All readers, all data) | Significant improvement in AUC with Koios DS assistance. | +0.083 [0.066, 0.099] (parametric); +0.079 [0.062, 0.096] (non-parametric). |
| Reader AUC Improvement (MRMC) (US readers, US data) | Significant improvement in AUC with Koios DS assistance. | +0.074 [0.051, 0.098] (parametric); +0.073 [0.049, 0.096] (non-parametric). This met the explicit criterion for the Thyroid module. |
| Reader AUC Improvement (MRMC) (EU readers, EU data) | Significant improvement in AUC with Koios DS assistance. | +0.022 [0.005, 0.039] (parametric); +0.019 [0.001, 0.037] (non-parametric). |
| Inter-Reader Variability | Reduction in variability. | 40.7% relative change (all readers, all data); 37.4% (US readers, US data); 49.7% (EU Readers, EU Data) in association of TI-RADS points assigned. |
| Interpretation Time (MRMC) | Reduction in interpretation time. | -23.6% (all readers, all data); -22.7% (US readers, US data); -32.4% (EU Readers, EU Data). |
Study Details:
2. Sample Sizes and Data Provenance:
-
Test Set (Clinical Study):
- Breast Functionality: 900 lesions from 900 different patients. (From predicate K190442, used for comparison). An additional 50 new cases were added to the breast set to test for robustness to dataset drift.
- Thyroid Functionality: 650 retrospectively collected cases (lesions) from 650 different patients.
- 500 cases from United States locations.
- 150 cases from European locations.
- Data Provenance: Retrospective for both breast and thyroid. Sourced from a wide variety of ultrasound hardware.
-
Training Set:
- Breast Engine: "A large database of known cases." (Specific number not provided in the summary, but the test set of 900 lesions was "set aside from the system's training data").
- Thyroid Engine: "A large database of known cases." (Specific number not provided, but the test set of 500 lesions was "set aside from the system's training data"). The training data was separate from the independent site data used in bench testing.
3. Number of Experts and Qualifications for Ground Truth:
- The document implies that ground truth for the clinical studies relied on pathology/follow-up outcomes, meaning clinical experts (pathologists, clinicians) established the definitive diagnosis.
- For the reader studies (MRMC), the "readers" themselves were the experts whose performance was being evaluated.
- Breast Study (K190442): 15 readers. Their qualifications varied:
- Board Certification/Specialty: Diagnostic Radiology, Breast Surgeon, OB/GYN, Interventional Radiology.
- Breast Fellowship Trained and/or Dedicated Breast Imager: 6 out of 15 had this.
- Years of Experience (Mammography and/or Breast Ultrasound): Ranged from 0 years to 30 years.
- Academic Institution Affiliation: Mixed (Yes/No).
- MQSA Qualified Interpreting Physician: Mixed (Yes/No).
- Thyroid Study (CRRS-3): 15 readers. Their qualifications varied:
- Reader Category: Domestic Endocrinologist (End), Domestic Radiologist (Rad), European Rad, European End.
- Experience (post-residency): Ranged from
- Breast Study (K190442): 15 readers. Their qualifications varied:
§ 892.2060 Radiological computer-assisted diagnostic software for lesions suspicious of cancer.
(a)
Identification. A radiological computer-assisted diagnostic software for lesions suspicious of cancer is an image processing prescription device intended to aid in the characterization of lesions as suspicious for cancer identified on acquired medical images such as magnetic resonance, mammography, radiography, or computed tomography. The device characterizes lesions based on features or information extracted from the images and provides information about the lesion(s) to the user. Diagnostic and patient management decisions are made by the clinical user.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Design verification and validation must include:
(i) A detailed description of the image analysis algorithms including, but not limited to, a detailed description of the algorithm inputs and outputs, each major component or block, and algorithm limitations.
(ii) A detailed description of pre-specified performance testing protocols and dataset(s) used to assess whether the device will improve reader performance as intended.
(iii) Results from performance testing protocols that demonstrate that the device improves reader performance in the intended use population when used in accordance with the instructions for use. The performance assessment must be based on appropriate diagnostic accuracy measures (
e.g., receiver operator characteristic plot, sensitivity, specificity, predictive value, and diagnostic likelihood ratio). The test dataset must contain sufficient numbers of cases from important cohorts (e.g., subsets defined by clinically relevant confounders, effect modifiers, concomitant diseases, and subsets defined by image acquisition characteristics) such that the performance estimates and confidence intervals of the device for these individual subsets can be characterized for the intended use population and imaging equipment.(iv) Standalone performance testing protocols and results of the device.
(v) Appropriate software documentation (
e.g., device hazard analysis; software requirements specification document; software design specification document; traceability analysis; and description of verification and validation activities including system level test protocol, pass/fail criteria, results, and cybersecurity).(2) Labeling must include:
(i) A detailed description of the patient population for which the device is indicated for use.
(ii) A detailed description of the intended reading protocol.
(iii) A detailed description of the intended user and recommended user training.
(iv) A detailed description of the device inputs and outputs.
(v) A detailed description of compatible imaging hardware and imaging protocols.
(vi) Warnings, precautions, and limitations, including situations in which the device may fail or may not operate at its expected performance level (
e.g., poor image quality or for certain subpopulations), as applicable.(vii) Detailed instructions for use.
(viii) A detailed summary of the performance testing, including: Test methods, dataset characteristics, results, and a summary of sub-analyses on case distributions stratified by relevant confounders (
e.g., lesion and organ characteristics, disease stages, and imaging equipment).