(389 days)
P980025
Not Found
Yes
The device description explicitly states that it employs "computer vision technology, developed using machine learning techniques" and utilizes "computer vision neural network algorithms, developed from open-source models using supervised machine learning techniques."
No.
The Explanation for use states that it is "designed to aid dental health professionals to review bitewing and periapical radiographs of permanent teeth in patients 12 years of age or older as a second reader." This indicates an interpretative function, not a therapeutic one.
Yes
Second Opinion® is explicitly stated as "designed to aid dental health professionals to review bitewing and periapical radiographs of permanent teeth... to identify and mark regions in relation to suspected dental findings which include Caries, Discrepancy at the margin of an existing restoration, Calculus, Periapical radiolucency, Crown (metal, including zirconia & non-metal), Filling (metal & non-metal), Root canal, Bridge and Implants." These actions directly contribute to diagnosing or aiding in the diagnosis of various dental conditions.
Yes
The device description explicitly states that Second Opinion® is a "computer aided detection ("CADe") software device" and details its components as software elements (Client, API, CV Models) residing either in the office or a cloud platform. There is no mention of accompanying or required hardware components beyond the standard computing infrastructure needed to run the software and display images.
Based on the provided information, this device is not an In Vitro Diagnostic (IVD).
Here's why:
- IVDs analyze biological samples: In Vitro Diagnostics are designed to examine specimens taken from the human body, such as blood, urine, tissue, etc., to provide information about a person's health.
- This device analyzes medical images: Second Opinion® analyzes radiographs (medical images) of teeth. It does not interact with or analyze any biological samples.
- The intended use is image analysis: The intended use clearly states that the software is designed to aid dental health professionals in reviewing bitewing and periapical radiographs.
Therefore, while this device is a medical device that uses AI/ML for diagnostic purposes, it falls under the category of medical image analysis software rather than an In Vitro Diagnostic.
No
The letter does not explicitly state that the FDA has reviewed and approved or cleared a PCCP for this specific device.
Intended Use / Indications for Use
Second Opinion® is a computer aided detection ("CADe") software to identify and mark regions in relation to suspected dental findings which include Caries, Discrepancy at the margin of an existing restoration, Calculus, Periapical radiolucency, Crown (metal, including zirconia & non-metal), Filling (metal & non-metal), Root canal, Bridge and Implants.
It is designed to aid dental health professionals to review bitewing and periapical radiographs of permanent teeth in patients 12 years of age or older as a second reader.
Product codes
MYN
Device Description
Second Opinion®is a computer aided detection ("CADe") software device indicated for use by dental health professionals as an aid in their assessment of bitewing and periapical radiographs of permanent teeth in patients 12 years of age or older. Second Opinion® employs computer vision technology, developed using machine learning techniques, to detect and draw attention as second reader to regions on bitewing and periapical radiographs where distinct pathologic and/or nonpathologic dental features may appear.
Second Opinion® consists of three parts:
- · In-office application or Client User Interface ("Client")
- · Application Programing API ("API")
- · Computer Vision Models ("CV Model", "CV Models")
The Client resides in the clinician's office. The API and CV Models reside in a cloud computing platform, where image processing takes place.
The CV Models create and append to a metadata file information denoting pixel regions and other associated properties of each radiograph. Those associated properties include:
- Normal anatomy (e.g., Teeth) ●
- Nine radiological dental findings, which include five restorations (crowns, bridges, . implants, root canals, fillings) and four pathologies (caries, marqin discrepancy -MD, calculus, periapical radiolucency - PR)
The API delivers the metadata back to Second Opinion® via the cloud. The metadata information is displayed in graphical form to clinical users by way of the Second Opinion® Client's user interface.
Mentions image processing
Yes
Mentions AI, DNN, or ML
Yes
Input Imaging Modality
Radiograph (bitewing and periapical radiographs)
Anatomical Site
Dental
Indicated Patient Age Range
12 years of age or older
Intended User / Care Setting
Dental health professionals / dental office setting
Description of the training set, sample size, data source, and annotation protocol
Not Found
Description of the test set, sample size, data source, and annotation protocol
The ground truth (GT) was assessed using the consensus approach (based on agreement among at least three out of four expert readers). Each GT expert independently marked areas on any radiograph wherein they marked (using the smallest possible rectangular bounding box to encompass the entire region identified) and identified the pathologic and/or non-pathologic features. All experts went through the training and reading of the images over the same period of time. 2,010 images reviewed by all four GT readers were used for the standalone and MRMC studies.
Summary of Performance Studies (study type, sample size, AUC, MRMC, standalone performance, key results)
Standalone Testing
- Study type: Standalone study
- Sample size: 2,010 images (1,640 normal, 370 lesion-containing for Caries; 1,741 normal, 269 lesion-containing for MD; 1,766 normal, 244 lesion-containing for Calculus; 1,887 normal, 123 lesion-containing for PR.)
- Standalone performance:
- wAFROC-FOM 95% CI (0.73, 0.79) for Caries with Jaccard Index of 0.4.
- wAFROC-FOM 95% CI (0.71, 0.78) for MD with Jaccard Index of 0.4.
- wAFROC-FOM 95% CI (0.78, 0.85) for Calculus with Jaccard Index of 0.4.
- wAFROC-FOM 95% CI (0.75, 0.84) for PR with Jaccard Index of 0.4.
- wAFROC-FOM 95% CI (0.61, 0.68) for Caries with Jaccard Index of 0.5.
- wAFROC-FOM 95% CI (0.62, 0.68) for MD with Jaccard Index of 0.5.
- wAFROC-FOM 95% CI (0.75, 0.81) for Calculus with Jaccard Index of 0.5.
- wAFROC-FOM 95% CI (0.69, 0.78) for PR with Jaccard Index of 0.5.
- Key results: Second Opinion® CADe exhibited comparable performance to unaided readers in detecting four pathologic features and five restorations based on Jaccard Index (JI) of ≥ 0.4 for LL (Lesion Localization).
MRMC Testing
- Study type: Fully-crossed multi-reader, multi-case (MRMC) retrospective reader study.
- Sample size: 25 readers, each reading a total of 2,010 images (1,005 images unaided and 1,005 images aided).
- Key results: The performance of readers aided by the use of Second Opinion® demonstrated statistically significant improvement over the performance of unaided readers for caries, margin discrepancy, calculus, and periapical radiolucency. No statistically significant reductions in performance were observed when readers used Second Opinion® as an assistive aid. All pathologies met the pre-specified endpoints for the MRMC study.
Key Metrics (Sensitivity, Specificity, PPV, NPV, etc.)
Standalone Testing
- Sensitivity: The standalone sensitivity of the product was in the range of 76.39% – 89.77%.
- False Positive Rate (FPPI): The false positive rate was in the range of 0.46 - 4.85.
MRMC Testing
- Improvement in sensitivity of a single dental finding was in the range of 0.9% - 11.7%.
- 17/25 (68%) improved the sensitivity of detecting Caries.
- 19/25 (76%) improved the sensitivity of detecting MD.
- 22/25 (88%) improved the sensitivity of detecting Calculus.
- 25/25 (100%) improved the sensitivity of detecting PR.
- Improvement in false positive rate of a single dental pathology was in the range of 0.08 - 0.136.
- 23/25 (92%) improved the rate of FPPI for Caries.
- 24/25 (96%) improved the rate of FPPI for MD.
- 25/25 (100%) improved the rate of FPPI for Calculus.
- 9/25 (36%) improved the rate of FPPI for PR.
- Unaided Reader wAFROC-FOM: 0.740
- Aided Reader wAFROC-FOM: 0.758
Predicate Device(s)
P980025
Reference Device(s)
Not Found
Predetermined Change Control Plan (PCCP) - All Relevant Information
Not Found
§ 892.2070 Medical image analyzer.
(a)
Identification. Medical image analyzers, including computer-assisted/aided detection (CADe) devices for mammography breast cancer, ultrasound breast lesions, radiograph lung nodules, and radiograph dental caries detection, is a prescription device that is intended to identify, mark, highlight, or in any other manner direct the clinicians' attention to portions of a radiology image that may reveal abnormalities during interpretation of patient radiology images by the clinicians. This device incorporates pattern recognition and data analysis capabilities and operates on previously acquired medical images. This device is not intended to replace the review by a qualified radiologist, and is not intended to be used for triage, or to recommend diagnosis.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Design verification and validation must include:
(i) A detailed description of the image analysis algorithms including a description of the algorithm inputs and outputs, each major component or block, and algorithm limitations.
(ii) A detailed description of pre-specified performance testing methods and dataset(s) used to assess whether the device will improve reader performance as intended and to characterize the standalone device performance. Performance testing includes one or more standalone tests, side-by-side comparisons, or a reader study, as applicable.
(iii) Results from performance testing that demonstrate that the device improves reader performance in the intended use population when used in accordance with the instructions for use. The performance assessment must be based on appropriate diagnostic accuracy measures (
e.g., receiver operator characteristic plot, sensitivity, specificity, predictive value, and diagnostic likelihood ratio). The test dataset must contain a sufficient number of cases from important cohorts (e.g., subsets defined by clinically relevant confounders, effect modifiers, concomitant diseases, and subsets defined by image acquisition characteristics) such that the performance estimates and confidence intervals of the device for these individual subsets can be characterized for the intended use population and imaging equipment.(iv) Appropriate software documentation (
e.g., device hazard analysis; software requirements specification document; software design specification document; traceability analysis; description of verification and validation activities including system level test protocol, pass/fail criteria, and results; and cybersecurity).(2) Labeling must include the following:
(i) A detailed description of the patient population for which the device is indicated for use.
(ii) A detailed description of the intended reading protocol.
(iii) A detailed description of the intended user and user training that addresses appropriate reading protocols for the device.
(iv) A detailed description of the device inputs and outputs.
(v) A detailed description of compatible imaging hardware and imaging protocols.
(vi) Discussion of warnings, precautions, and limitations must include situations in which the device may fail or may not operate at its expected performance level (
e.g., poor image quality or for certain subpopulations), as applicable.(vii) Device operating instructions.
(viii) A detailed summary of the performance testing, including: test methods, dataset characteristics, results, and a summary of sub-analyses on case distributions stratified by relevant confounders, such as lesion and organ characteristics, disease stages, and imaging equipment.
0
Image /page/0/Picture/0 description: The image shows the logo of the U.S. Food and Drug Administration (FDA). On the left is the Department of Health & Human Services logo. To the right of that is the FDA logo, which is a blue square with the letters "FDA" in white. To the right of the blue square is the text "U.S. FOOD & DRUG ADMINISTRATION" in blue.
March 4, 2022
Pearl Inc. % Zvi Ladin, Ph.D. Principal Boston MedTech Advisors 990 Washington Street, Suite #204 Dedham MA 02026
Re: K210365
Trade/Device Name: Second Opinion® Regulation Number: 21 CFR 892.2070 Regulation Name: Medical Image Analyzer Regulatory Class: Class II Product Code: MYN Dated: January 20, 2022 Received: January 24, 2022
Dear Zvi Ladin:
We have reviewed your Section 510(k) premarket notification of intent to market the device referenced above and have determined the device is substantially equivalent (for the indications for use stated in the enclosure) to legally marketed predicate devices marketed in interstate commerce prior to May 28, 1976, the enactment date of the Medical Device Amendments, or to devices that have been reclassified in accordance with the provisions of the Federal Food, Drug, and Cosmetic Act (Act) that do not require approval of a premarket approval application (PMA). You may, therefore, market the device, subject to the general controls provisions of the Act. Although this letter refers to your product as a device, please be aware that some cleared products may instead be combination products. The 510(k) Premarket Notification Database located at https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm identifies combination product submissions. The general controls provisions of the Act include requirements for annual registration, listing of devices, good manufacturing practice, labeling, and prohibitions against misbranding and adulteration. Please note: CDRH does not evaluate information related to contract liability warranties. We remind you, however, that device labeling must be truthful and not misleading.
If your device is classified (see above) into either class II (Special Controls) or class III (PMA), it may be subject to additional controls. Existing major regulations affecting your device can be found in the Code of Federal Regulations, Title 21, Parts 800 to 898. In addition, FDA may publish further announcements concerning your device in the Federal Register.
1
Please be advised that FDA's issuance of a substantial equivalence determination does not mean that FDA has made a determination that your device complies with other requirements of the Act or any Federal statutes and regulations administered by other Federal agencies. You must comply with all the Act's requirements, including, but not limited to: registration and listing (21 CFR Part 807); labeling (21 CFR Part 801); medical device reporting of medical device-related adverse events) (21 CFR 803) for devices or postmarketing safety reporting (21 CFR 4, Subpart B) for combination products (see https://www.fda.gov/combination-products/guidance-regulatory-information/postmarketing-safety-reportingcombination-products); good manufacturing practice requirements as set forth in the quality systems (QS) regulation (21 CFR Part 820) for devices or current good manufacturing practices (21 CFR 4, Subpart A) for combination products; and, if applicable, the electronic product radiation control provisions (Sections 531-542 of the Act); 21 CFR 1000-1050.
Also, please note the regulation entitled, "Misbranding by reference to premarket notification" (21 CFR Part 807.97). For questions regarding the reporting of adverse events under the MDR regulation (21 CFR Part 803), please go to https://www.fda.gov/medical-device-safety/medical-device-reportingmdr-how-report-medical-device-problems.
For comprehensive regulatory information about medical devices and radiation-emitting products, including information about labeling regulations, please see Device Advice (https://www.fda.gov/medicaldevices/device-advice-comprehensive-regulatory-assistance) and CDRH Learn (https://www.fda.gov/training-and-continuing-education/cdrh-learn). Additionally, you may contact the Division of Industry and Consumer Education (DICE) to ask a question about a specific regulatory topic. See the DICE website (https://www.fda.gov/medical-device-advice-comprehensive-regulatoryassistance/contact-us-division-industry-and-consumer-education-dice) for more information or contact DICE by email (DICE@fda.hhs.gov) or phone (1-800-638-2041 or 301-796-7100).
Sincerely.
Laurel Burk Assistant Director Diagnostic X-ray Systems Team Division of Radiological Health OHT7: Office of In Vitro Diagnostics and Radiological Health Office of Product Evaluation and Quality Center for Devices and Radiological Health
Enclosure
2
Indications for Use
510(k) Number (if known) K210365
Device Name Second Opinion®
Indications for Use (Describe)
Second Opinion® is a computer aided detection ("CADe") software to identify and mark regions in relation to suspected dental findings which include Caries, Discrepancy at the margin of an existing restoration, Calculus, Periapical radiolucency, Crown (metal, including zirconia & non-metal), Filling (metal & non-metal), Root canal, Bridge and Implants.
It is designed to aid dental health professionals to review bitewing and periapical radiographs of permanent teeth in patients 12 years of age or older as a second reader.
| Type of Use (Select one or both, as applicable) | |
|---|---|
| Prescription Use (Part 21 CFR 801 Subpart D) | Over-The-Counter Use (21 CFR 801 Subpart C) |
CONTINUE ON A SEPARATE PAGE IF NEEDED.
This section applies only to requirements of the Paperwork Reduction Act of 1995.
DO NOT SEND YOUR COMPLETED FORM TO THE PRA STAFF EMAIL ADDRESS BELOW.
The burden time for this collection of information is estimated to average 79 hours per response, including the time to review instructions, search existing data sources, gather and maintain the data needed and complete and review the collection of information. Send comments regarding this burden estimate or any other aspect of this information collection, including suggestions for reducing this burden, to:
Department of Health and Human Services Food and Drug Administration Office of Chief Information Officer Paperwork Reduction Act (PRA) Staff PRAStaff(@fda.hhs.gov
"An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB number."
3
K210365 510(K) SUMMARY
1. Submitter's Identification
Pearl Inc. 2515 Benedict Canyon Dr. Beverly Hills, CA, 90210 USA (239) 450-8829
Contact Person: Bill Birdsall
Position: Chief Compliance Officer
Date Summary Prepared: March 2, 2022
2. Trade Name of the Device
Second Opinion®
3. Common or Usual Name
Analyzer, Medical Image
4. Classification Name, Regulatory Classification & Product Code
Classification Name: Analyzer, Medical Image Requlatory Classification: 21CFR 892.2070, Class II Product Code: MYN
5. Predicate Device Information
Predicate device: Logicon Caries Detection by Carestream Dental LLC. (P980025 & Supplements 1, 2, 3 & 4)
6. Device Description
Second Opinion®is a computer aided detection ("CADe") software device indicated for use by dental health professionals as an aid in their assessment of bitewing and periapical radiographs of permanent teeth in patients 12 years of age or older. Second Opinion® employs computer vision technology, developed using machine learning techniques, to detect and draw attention as second reader to regions on bitewing and periapical radiographs where distinct pathologic and/or nonpathologic dental features may appear.
4
Second Opinion® consists of three parts:
- · In-office application or Client User Interface ("Client")
- · Application Programing Interface ("API")
- · Computer Vision Models ("CV Model", "CV Models")
The Client resides in the clinician's office. The API and CV Models reside in a cloud computing platform, where image processing takes place.
The CV Models create and append to a metadata file information denoting pixel regions and other associated properties of each radiograph. Those associated properties include:
- Normal anatomy (e.g., Teeth) ●
- Nine radiological dental findings, which include five restorations (crowns, bridges, . implants, root canals, fillings) and four pathologies (caries, marqin discrepancy -MD, calculus, periapical radiolucency - PR)
The API delivers the metadata back to Second Opinion® via the cloud. The metadata information is displayed in graphical form to clinical users by way of the Second Opinion® Client's user interface.
7. Indications for Use
Second Opinion® is a computer aided detection ("CADe") software to identify and mark regions in relation to suspected dental findings which include Caries, Discrepancy at the margin of an existing restoration, Calculus, Periapical radiolucency, Crown (metal, including zirconia & non-metal), Filling (metal & non-metal), Root canal, Bridge and Implants.
It is designed to aid dental health professionals to review bitewing and periapical radiographs of permanent teeth in patients 12 years of age or older as a second reader.
8. Summary of Substantial Equivalence:
The predicate device and candidate device are similar CADe devices in the following ways:
-
- Intended use: Both devices are intended to be used to aid dental clinicians in their detection of pathologic dental features in radiographs of permanent teeth.
-
- Technology characteristics: Both devices employ computer vision and machine learning to output detections.
-
- Safety: As both the candidate and predicate device are CADe systems, neither pose any direct safety hazard to the patient.
-
- Clinical Performance: Both devices have undergone clinical studies which demonstrate statistically significant improvement in aided reader performance. However, due to technological differences (e.g., the need to mark the region of interest by the user of the predicate device), there are challenges for a direct technological and performance comparison with the predicate device.
5
| For substantial equivalence comparison, the follow table is provided: | ||
|---|---|---|
| Item | Candidate device:
Second Opinion® | Predicate Device:
Logicon Caries
Detection (P980025) | Comments |
|-------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------|
| Manufacturer | Pearl Inc. | Carestream DentalLLC | N/A |
| Classification | 892.2070 | 892.2070 | Same |
| Product Code | MYN | MYN | Same (Productcode MYN
wasrecently
reclassified toclass II) |
| Image Modality | Radiograph | Radiograph | Same |
| Intended Use | To aid in clinical detection of
pathologic and/or non-
pathologic dental features in
radiographs
of permanent teeth, as a
second reader, only after the
initial read is completed | To aid in clinical
detection of pathologic
features in
radiographs of adult teeth | Same indication as
predicate device as both
devices are intended to
identify pathologic features
in dental radiographs. |
| Indications for
Use | Second Opinion® is a
computer aided detection
("CADe") software to
identify landmark regions in
relation to suspected dental
findings which include
Caries, Discrepancy at the
margin of an existing
restoration, Calculus,
Periapical radiolucency,
Crown (metal, including
zirconia & non-metal),
Filling (metal & non- metal),
Root canal, Bridge and
Implants.It is designed to
aid dental health
professionals to review
bitewing and periapical
radiographs of permanent
teeth in patients 12 years of
age or older as a second
reader. | The Logicon Caries
Detector is a software
device that is an aid in the
diagnosis of caries that
have penetrated into the
dentin, on un- restored
proximal surfaces of
secondary dentition
through the statistical
analysis of digital intraoral
radiographic imagery. The
device provides additional
information for the
clinician to use in his/her
diagnosis of a tooth
surface suspected of
being carious. It is
designed to work in
conjunction with an
existing Carestream
DentalRVG digital x-ray
radiographic system with
Carestream Dental
Imaging Software
(DIS/CSI) for WINDOWS
7 or higher. | |
| Intended body part | Dental | Dental | Same as predicate device |
| Intended User | Dental Clinicians | Dental Clinicians | Same aspredicate
device |
| Marker
Type/Size | Bounding boxes /
Fixed | User may mark ROI with V
tool (to select a pie-wedge
shape: narrow endat the
center of thetooth, wide
end extending through
surface over the
suspicious region)
or the pencil tool todraw
the ROIs. | |
| Prescription
or OTC | For Prescription Use | For Prescription Use | Same |
| Algorithm | Utilizes computer vision
neural network algorithms,
developed from open-source
models using supervised
machine learning
techniques. | Utilizes computer vision
neural networkalgorithms
developed using proprietary
techniques. | Same |
| Reader
workflow | Second reader workflow | The user can select specific
regions of interest (ROI) to
be analyzed by the
program | |
| Clinical Study | Standalone study for
pathological (Caries,
discrepancy at the
margin, calculus, and
periapical radiolucency)
and non-pathologic
feature detection
performance (Crowns,
fillings, rootcanal,
bridges, and Implants).
• Multiple-Reader,
Multiple-Case
(MRMC) study for
pathologic dental
features
• Analysis included:
• wAFROC-FOM
analysis for
primary endpoints
• Determination of
the changes in
sensitivity and
change in number
of false positive | • MRMC study for two
types of dental caries:
approximal/enamel into
dentin
• Analysis included: ROC
analysis for primary
endpoints | All devices included
MRMC studies to assess
effectiveness. |
| | dental pathologies
of a given type per
image (FPPI). | | |
| Image Source | Accepts image formats
from RVG, DICOM,
JPEG, TIFF,
and PNG and
converts to JPEG. | Limited to
Carestream
proprietary
radiography
equipment. | Different |
| Type of
Device | CADe | CADe | Same |
| Hardware
Requirements | WINDOWS 7 or higher | WINDOWS 7 or
higher | Same |
| Analysis of digital
intraoral
radiographic
imagery | Bitewing & Periapical | Bitewing only | Similar |
6
7
Table 1. Comparison of Second Opinion® with the predicate device
9. Technological Comparison to Predicate Device
The fundamental technological principle for both the candidate and predicate devices is theautomatic computerized lesion detection of the dental finding of interest, as an aid to dental health professionals to review patients' dental radiographs.
The candidate and predicate devices are technologically equivalent as follows:
- Both are software devices designed to run on Windows operating systems. ●
- Both devices are designed to process digital intraoral bitewing radiographs.
- Both devices use neural network-based computer vision algorithms for lesion ● detection.
- Both devices demarcate detections within the user interface with a graphical overlay on the radiograph.
- Both devices produce near-instantaneous detection results.
- Both devices are considered to be of "moderate" level of software concern. ●
- Both devices passed all verification and validation testing requirements.
The candidate and predicate devices are technologically different as follows:
- The candidate device includes computer vision algorithms capable of detecting other . dental features in addition to carious lesions.
- . The candidate device includes computer vision algorithms capable of making detections in periapical radiographs in addition to bitewing radiographs
- The candidate device uses a cloud-based rather than CD-ROM method of software ● installation
- The candidate device is capable of making detections in images captured on
8
radiography devices from a range of manufacturers, rather than in conjunction only with a Carestream digital imaqing device.
- . The candidate device outputs detections as a second reader only and physician needs to make final determinations. The user does not select specific regions of interest (ROI) to be analyzed by the program
- The candidate device does not predict lesion depth or severity. .
10. Assessment of Benefit-Risk, Safety and Effectiveness, and Substantial Equivalence to Predicate Device:
Pearl has demonstrated the benefits of the device through a standalone and MRMC clinical studies of aided and unaided reader accuracy across the four pathologic features includedin Second Opinion®'s Indications for Use: caries, margin discrepancy, calculus, and periapical radiolucency. While detection accuracy improvement between aided and undead readers were seen to vary by pathologic feature, the results of the MRMC studiesshowed statistically significant aided-reader improvement in detection accuracy across all four pathologic features. When the probable benefits and probable risks of Second Opinion® are weighed against one another, the weight of benefits significantly exceeds that of risks. This judgement can be made based on review of the submitted materials showing that Second Opinion® meets the design verification and labeling Special Controls required for clearance of Class II medical image analyzers. It is, thus, concluded that Second Opinion® can be considered safe and effective such that the devicewill aid users in the indicated user population in their radiographic detection of certain abnormal dental features.
11. Discussion of Non-Clinical Tests Performed
The device is a software-only device, so most testable characteristics common to other device types, including Biocompatibility/Materials, Shelf Life/Sterility, Electromagnetic Compatibility and Electrical Safety, Magnetic Resonance (MR) Compatibility, are not applicable to this device.
Software Verification and Validation Testing
Second Opinion® verification testing of software, unit testing, software integration testing, and software system testing were conducted. Verification and validation activities for Second Opinion® were conducted to provide evidence that the design meets user needs. intended use and application specification. The testing results support that all the software specifications have met the acceptance criteria and the claims of substantial equivalence.
12. Discussion of Clinical Tests Performed
Clinical evaluation of Second Opinion® was performed to validate the clinical efficacy of the system in helping dentists review four dental pathologies (caries, margin discrepancy, calculus, and periapical radiolucency) on intra oral radiographs. Second Opinion® was clinically tested as a standalone device and in a fully-crossed multi-case (MRMC) reader study.
The Weighted Alternative Free-Response Receiver Operating Characteristic (wAFROC) paradigm was used as the metric of efficacy for all studies. The ground truth (GT) was assessed using the consensus approach (based on agreement among at least three out of
9
four expert readers). Each GT expert independently marked areas on any radiograph wherein they marked (using the smallest possible rectangular bounding box to encompass the entire region identified) and identified the pathologic and/or non-pathologic features. All experts went through the training and reading of the images over the same period of time. The studies were conducted as retrospective, unblinded open-label, multi-site trials that produced clinically useful information on the potential application of this device in a dental office setting. 2,010 images reviewed by all four GT readers were used for the standalone and MRMC studies:
| Feature | Caries | MD | Calculus | PR |
|---|---|---|---|---|
| # of normal radiographs | 1,640 | 1,741 | 1,766 | 1,887 |
| # of lesion-containing radiographs | 370 | 269 | 244 | 123 |
| Number of lesions | 655 | 355 | 467 | 144 |
| Average # of lesions/image | 1.77 | 1.32 | 1.91 | 1.17 |
Table 2. Images reviewed by all four GT readers
Standalone Testing
In the standalone study, the Second Opinion® CADe exhibited comparable performance to unaided readers in detecting four pathologic features and five restorations based on Jaccard Index (JI) of ≥ 0.4 for LL (Lesion Localization). This value of JI optimized lesion localization of unaided readers and was also used to process the Standalone performance of the candidate device. Use of other values of JI will clearly affect the overall performance of the product.
- . Jaccard Index of 0.4 corresponds to an overlap area between the device's outputs and truth of 57% in theory. Using Jaccard Index of 0.4 leads to the device's standalone performance wAFROC-FOM 95% Cl (0.73, 0.79), (0.71, 0.78), (0.78, 0.85) and (0.75,0.84) for Caries, MD, Calculus, and PR with consensus truthing method. respectively.
- Jaccard Index of 0.5 corresponds to an overlap area between the device's outputs . and truth of 67% in theory. Using Jaccard Index of 0.5 leads to the device's standalone performance wAFROC-FOM 95% Cl (0.61, 0.68), (0.62, 0.68), (0.75, 0.81) and (0.69,0.78) for Caries, MD, Calculus, and PR with consensus truthing method, respectively.
Product's standalone sensitivity and false positive rate were also assessed. Sensitivity is defined as the number of dental pathologies (of a given type) detected as a percentage of the same type GT pathologies on a given slide. The false positive rate is defined as the number of false positive findings of a given type identified on a given slide and expressed in terms of FPPI (false positive per image).
10
The standalone sensitivity of the product was in the range of 76.39% – 89.77% and the false positive rate was in the range of 0.46 - 4.85.
MRMC Testing and Results
Pearl conducted a fully-crossed multi-reader, multi-case (MRMC) retrospective reader study to determine the impact of Second Opinion® on reader performance in detecting fourdental pathologies. The primary objective of the study was to determine whether the detection accuracy of readers aided by Second Opinion® is superior to the detection accuracy of readers unaided by Second Opinion®.
Twenty-five readers were asked to determine the locations of all identified (i.e., classified) lesions, in a setting designed to increase user specificity in detection. Each reader reada total of 2,010 images – 1,005 images unaided and 1,005 images aided (using Second Opinion®).
The performance of readers aided by the use of Second Opinion® demonstrated statistically significant improvement over the performance of unaided readers for caries, margin discrepancy, calculus, and periapical radiolucency.
Moreover, the improvement in sensitivity of a single dental finding was in the range of 0.9% 11.7% and examination of the individual improvement in sensitivity documented that:
- 17/25 (68%) improved the sensitivity of detecting Caries .
- . 19/25 (76%) improved the sensitivity of detecting MD
- 22/25 (88%) improved the sensitivity of detecting Calculus .
- . 25/25 (100%) improved the sensitivity of detecting PR
The improvement in false positive rate of a single dental pathology was in the range of 0.08 - 0.136. Examination of the individual improvements in the rate of FPPI documented that:
- 23/25 (92%) improved the rate of FPPI for Caries ●
- . 24/25 (96%) improved the rate of FPPI for MD
- . 25/25 (100%) improved the rate of FPPI for Calculus
- . 9/25 (36%) improved the rate of FPPI for PR
No statistically significant reductions in performance were observed when readers used Second Opinion® as an assistive aid. All pathologies met the pre-specified endpoints for the MRMC study.
The results have demonstrated that, in a significant portion of the target population, detection performances using the proposed device improved with statistical significance. Therefore, the proposed device could provide potential assistance for dentists in the reviewof dental pathologies found in intra oral radiographs when used as a second read.
13. Comparison to Predicate Clinical Outcomes
Predicate Device: Logicon Caries Detection by Carestream Dental LLC.
The predicate device's clinical study was based on a dataset of 175 tooth surfaces and
11
18 readers. The primary endpoint was to determine the device's effect on reader sensitivity (true positive identification), specificity (true negative), and accuracy (fraction of all correct diagnosis made by a dentist) as they pertain to detection of caries lesions into the dentin on proximal tooth surfaces. Truth was defined as the dentist's clinical assessment of the exposed lesion prior to restoration. Results from this study showed a significant increase in reader sensitivity and no significant change in reader specificity (slightly reduced when aided by Logicon) when readers used the device. Average reader accuracy, reported as the comparison of detection accuracy unaided readers to aided readers, were determined using the ROC paradigm and showed an improvement of 12.8 percent:
| Unaided Reader Accuracy | Aided Readers Accuracy |
|---|---|
| 0.756 | 0.883 |
Table 5. Logicon Caries Detector (predicate device) accuracy – Aided vs. Unaided
However, this observed median difference is not significant based on the Wilcoxon test (P=0.0537). Therefore, the improvement in diagnostic accuracy associated with the Logicon Caries device is mostly associated with the improvement in sensitivity.
The candidate device's clinical study was based on a dataset of 2010 radiographs which included 370 carious lesions. Three primary studies were performed: Standalone, MRMC, and CADe vs. Reader. The endpoints were designed to determine the candidate device's ability to identify dental features as a standalone system and to determine the device's effect on reader performance.
In comparison to the predicate device's clinical study, caries detection performance of the candidate device, measured based on wAFROC-FOM, for unaided and Second Opinion® aided readers with Second Opinion® is:
| Unaided Reader wAFROC-FOM | Aided Reader wAFROC-FOM |
|---|---|
| 0.740 | 0.758 |
| Table 6. Second Opinion® wAFROC-FOM - Aided vs. Unaided | |
|---|---|
| --------------------------------------------------------- | -- |
This difference is significant with P=0.0062.
Therefore, similar to the predicate device, the candidate device also demonstrated improved detection accuracy in detecting carious lesions. Although candidate device performance was determined using a different statistical paradigm (wAFROC) than predicate device performance (ROC), the trend is the same: When aided by the device. readers' ability to detect carious lesions improves.
14. Conclusions
Based on the information presented above, Second Opinion® and its predicate device, Logicon Caries Detector, are deemed to have similar intended uses as devices which aid n
12
the detection of pathologic features that can appear in dental radiographic imagery. Second Opinion®'s clinical trial results demonstrate that like its predicate, the device effectively improves the performance of its intended users,as a second read CADe system.
As Second Opinion® raises no new or different questions of safety or effectiveness, performs in accordance with its specifications, meets user needs, meets the intended use and therefore was found substantially equivalent to the predicate device.