(224 days)
Second Opinion PC is a computer aided detection ("CADe") software to aid dentists in the detection of periapical radiolucencies by drawing bounding polygons to highlight the suspected region of interest.
It is designed to aid dental health professionals to review periapical radiographs of permanent teeth in patients 12 years of age or older as a second reader.
Second Opinion PC (Periapical Radiolucency Contouring) is a radiological, automated, computer-assisted detection (CADe) software intended to aid in the detection of periapical radiolucencies on periapical radiographs using polygonal contours. The device is not intended as a replacement for a complete dentist's review or their clinical judgment which considers other relevant information from the image, patient history, or actual in vivo clinical assessment.
Second Opinion PC consists of three parts:
- Application Programing Interface ("API")
- Machine Learning Modules ("ML Modules")
- Client User Interface ("Client")
The processing sequence for an image is as follows:
- Images are sent for processing via the API
- The API routes images to the ML modules
- The ML modules produce detection output
- The UI renders the detection output
The API serves as a conduit for passing imagery and metadata between the user interface and the machine learning modules. The API sends imagery to the machine learning modules for processing and subsequently receives metadata generated by the machine learning modules which is passed to the interface for rendering.
Second Opinion PC uses machine learning to detect periapical radiolucencies. Images received by the ML modules are processed yielding detections which are represented as metadata. The final output is made accessible to the API for the purpose of sending to the UI for visualization. Detected periapical radiolucencies are displayed as polygonal overlays atop the original radiograph which indicate to the practitioner which teeth contain which detected periapical radiolucencies that may require clinical review. The clinician can toggle over the image to highlight a potential condition for viewing.
Here's a detailed breakdown of the acceptance criteria and the study proving the device meets them, based on the provided FDA 510(k) clearance letter:
Acceptance Criteria and Device Performance Study
The Pear Inc. "Second Opinion Periapical Radiolucency Contours" (Second Opinion PC) device aims to aid dentists in detecting periapical radiolucencies using polygonal contours, functioning as a second reader. The device's performance was evaluated through a standalone clinical study demonstrating non-inferiority to its predicate device, which used bounding boxes.
1. Table of Acceptance Criteria and Reported Device Performance
The submission document primarily focuses on demonstrating non-inferiority to the predicate device rather than explicitly stating pre-defined acceptance criteria with specific thresholds for "passing." However, the implicit acceptance criteria are that the device is non-inferior to its predicate (Second Opinion K210365) in detecting periapical radiolucencies when using polygonal contours.
| Acceptance Criterion (Implicit) | Reported Device Performance (Second Opinion PC) |
|---|---|
| Non-inferiority in periapical radiolucency detection accuracy compared to predicate device (Second Opinion K210365) using bounding boxes. | wAFROC-FOM (Estimated Difference): 0.15 (95% CI: 0.10, 0.21) compared to Second Opinion (predicate) |
| (Lower bound of 95% CI (0.10) exceeded -0.05, demonstrating non-inferiority at 5% significance level) | |
| Overall detection accuracy (wAFROC-FOM) | wAFROC-FOM: 0.85 (95% CI: 0.81, 0.89) |
| Overall detection accuracy (HR-ROC-AUC) | HR-ROC-AUC: 0.93 (95% CI: 0.90, 0.96) |
| Lesion level sensitivity | Lesion Level Sensitivity: 77% (95% CI: 69%, 84%) |
| Average false positives per image | Average False Positives per Image: 0.28 (95% CI: 0.23, 0.33) |
2. Sample Size Used for the Test Set and Data Provenance
- Sample Size: 500 unique unannotated periapical radiographs.
- Data Provenance: The dataset is characterized by a representative distribution across:
- Geographical Regions (within the United States):
- Northwest: 116 radiographs (23.2%)
- Southwest: 46 radiographs (9.2%)
- South: 141 radiographs (28.2%)
- East: 84 radiographs (16.8%)
- Midwest: 113 radiographs (22.6%)
- Patient Cohorts (Age Distribution):
- 12-18 years: 4 radiographs (0.8%)
- 18-75 years: 209 radiographs (41.8%)
- 75+ years: 8 radiographs (1.6%)
- Unknown age: 279 radiographs (55.8%)
- Imaging Devices: A variety of devices were used, including Carestream-Trophy (RVG6100, RVG5200, RVG6200), DEXIS (DEXIS, DEXIS Platinum, KaVo Dental Technologies DEXIS Titanium), Kodak-Trophy KodakRVG6100, XDR EV71JU213, and unknown devices.
- Geographical Regions (within the United States):
- Retrospective or Prospective: Not explicitly stated, but the description of "representative distribution" and diverse origins suggests a retrospective collection of existing images.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Their Qualifications
- Number of Experts: Four expert readers.
- Qualifications of Experts: Not explicitly stated beyond "expert readers."
4. Adjudication Method for the Test Set
- Adjudication Method: Consensus approach based on agreement among at least three out of four expert readers (3+1 adjudication).
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was done
- Was an MRMC study done? No, a traditional MRMC comparative effectiveness study was not performed for the subject device (Second Opinion PC).
- Effect Size of Human Readers with AI vs. without AI: Not applicable for this specific study of Second Opinion PC. The predicate device (Second Opinion K210365) did undergo MRMC studies, demonstrating statistically significant improvement in aided reader performance for that device. The current study focuses on the standalone non-inferiority of Second Opinion PC compared to its predicate.
6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) was done
- Was a standalone study done? Yes, a standalone clinical study was performed. The study compared the performance of Second Opinion PC (polygonal localization) directly with Second Opinion (bounding box localization) in detecting periapical radiolucencies.
- Metrics: wAFROC-FOM and HR-ROC-AUC were used.
- Key Finding: Second Opinion PC was found to be non-inferior to Second Opinion.
7. The Type of Ground Truth Used
- Type of Ground Truth: Expert consensus. The ground truth (GT) was established by the consensus of at least three out of four expert readers who independently marked periapical radiolucencies using the smallest possible polygonal contour.
8. The Sample Size for the Training Set
- Sample Size for Training Set: Not explicitly mentioned in the provided text. The document focuses on the clinical validation (test set).
9. How the Ground Truth for the Training Set Was Established
- Ground Truth for Training Set: Not explicitly mentioned in the provided text. It is implied that the device was "developed using machine learning techniques" from "open-source models using supervised machine learning," which typically requires a labeled training set, but specifics on its establishment are absent.
§ 892.2070 Medical image analyzer.
(a)
Identification. Medical image analyzers, including computer-assisted/aided detection (CADe) devices for mammography breast cancer, ultrasound breast lesions, radiograph lung nodules, and radiograph dental caries detection, is a prescription device that is intended to identify, mark, highlight, or in any other manner direct the clinicians' attention to portions of a radiology image that may reveal abnormalities during interpretation of patient radiology images by the clinicians. This device incorporates pattern recognition and data analysis capabilities and operates on previously acquired medical images. This device is not intended to replace the review by a qualified radiologist, and is not intended to be used for triage, or to recommend diagnosis.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Design verification and validation must include:
(i) A detailed description of the image analysis algorithms including a description of the algorithm inputs and outputs, each major component or block, and algorithm limitations.
(ii) A detailed description of pre-specified performance testing methods and dataset(s) used to assess whether the device will improve reader performance as intended and to characterize the standalone device performance. Performance testing includes one or more standalone tests, side-by-side comparisons, or a reader study, as applicable.
(iii) Results from performance testing that demonstrate that the device improves reader performance in the intended use population when used in accordance with the instructions for use. The performance assessment must be based on appropriate diagnostic accuracy measures (
e.g., receiver operator characteristic plot, sensitivity, specificity, predictive value, and diagnostic likelihood ratio). The test dataset must contain a sufficient number of cases from important cohorts (e.g., subsets defined by clinically relevant confounders, effect modifiers, concomitant diseases, and subsets defined by image acquisition characteristics) such that the performance estimates and confidence intervals of the device for these individual subsets can be characterized for the intended use population and imaging equipment.(iv) Appropriate software documentation (
e.g., device hazard analysis; software requirements specification document; software design specification document; traceability analysis; description of verification and validation activities including system level test protocol, pass/fail criteria, and results; and cybersecurity).(2) Labeling must include the following:
(i) A detailed description of the patient population for which the device is indicated for use.
(ii) A detailed description of the intended reading protocol.
(iii) A detailed description of the intended user and user training that addresses appropriate reading protocols for the device.
(iv) A detailed description of the device inputs and outputs.
(v) A detailed description of compatible imaging hardware and imaging protocols.
(vi) Discussion of warnings, precautions, and limitations must include situations in which the device may fail or may not operate at its expected performance level (
e.g., poor image quality or for certain subpopulations), as applicable.(vii) Device operating instructions.
(viii) A detailed summary of the performance testing, including: test methods, dataset characteristics, results, and a summary of sub-analyses on case distributions stratified by relevant confounders, such as lesion and organ characteristics, disease stages, and imaging equipment.