(199 days)
E-Ortho shoulder is intended to be used as an information tool to assist in the preoperative surgical planning and visualization of a primary total shoulder replacement.
e-Ortho Shoulder software is a web-based surgical planning software application. e-Ortho Shoulder provides a pre-surgical planning tool for surgeons that helps them understand their patient's anatomy prior to surgery. Compared to using two-dimensional (2D) images to plan a shoulder arthroplasty (current method used by FH-Orthopedic surgeons), e-Ortho supplies information to surgeons to help prepare an intraoperative plan. E-Ortho allows surgeons to work in three-dimensional (3D) visualization, implant visualization and positioning within the specific patient's bone model (scapula and humerus), using reliable landmarks. This allows surgeons to preoperatively select the needed implant and determine its desired position.
The subject submission seeks to add humeral planning capabilities to the previously cleared FH E-Ortho Shoulder Software. Additional changes to the software have been made to improve functionality within the previously cleared intended use.
The e-Ortho Shoulder Software v1.1 is intended to be used as an information tool to assist in the preoperative surgical planning and visualization of a primary total shoulder replacement. The performance testing for this device is primarily focused on verification and validation activities, and a comparison against a "gold standard" software.
Here's a breakdown of the requested information based on the provided document:
-
Table of Acceptance Criteria and Reported Device Performance:
The document doesn't explicitly lay out acceptance criteria in a quantitative table with corresponding device performance metrics like sensitivity, specificity, or AUC as might be seen for diagnostic AI. Instead, the acceptance is demonstrated through the successful completion of verification and validation processes and equivalence to a "gold standard" software.
Acceptance Criterion Type Description Reported Device Performance Functional Verification Verification of functional components of the subject device through test campaigns. Five test campaigns carried out by five different evaluators in two different environments. Three minor bugs identified, but "not expected to impact the planning itself." Usability Validation Validation of critical features through usability testing. A usability test campaign conducted with five surgeons. "The result of the validation tests coincides with the expected results for each test case and no test failed." Accuracy Testing Comparison of implant values (version and inclination) obtained from the subject device against a "gold standard" software in simulated dangerous situations and potential harms. Simulations included right and left-sided scenarios, head-first/feet-first supine positioning, varying reaming depths, and implant visualization from varying angles. Compared to Materialise Innovation Suite (Mimics V22 and 3matic V14) and SolidWork 2016. "All tests passed." "Thus, the accuracy of e-Ortho is adequate to provide safe use of the product." -
Sample size used for the test set and the data provenance:
- Functional Verification Test Set: The sample size is not explicitly stated in terms of patient cases or images. Instead, it refers to "five test campaigns" carried out by "five different evaluators in two different environments" to verify "functional components." This suggests a focus on software functionality testing rather than patient data performance.
- Usability Validation Test Set: "Usability test campaign, with critical features requiring validation by five surgeons." The number of "test cases" or patient data involved in this usability validation is not specified.
- Accuracy Testing Test Set: Not specified in terms of patient cases. The testing involved "different virtual cases including right and left sided scenarios in head-first supine positioning of patient as well as feet-first supine patient positioning, varying reaming depths, and implant visualization from varying angles." This implies synthetically generated or modified cases rather than a specific set of retrospective or prospective patient data from a particular country. The data provenance is described as "simulated in different virtual cases."
-
Number of experts used to establish the ground truth for the test set and the qualifications of those experts:
- For Functional Verification and Usability Validation, the "experts" were the "five different evaluators" and "five surgeons" respectively. Their qualifications are not explicitly detailed beyond being "surgeons" for usability.
- For Accuracy Testing, the ground truth was established by "gold standard" software: Materialise innovation Suite (Mimics V22 and 3matic V14) and SolidWork 2016. No human experts are described as establishing the ground truth directly for this specific part of the testing.
-
Adjudication method (e.g. 2+1, 3+1, none) for the test set:
The document does not describe an adjudication method for establishing ground truth, as the accuracy testing relied on "gold standard" software rather than human consensus. For functional and usability testing, it appears to be direct observation of test results and comparison to expected outcomes.
-
If a multi-reader multi-case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance:
No MRMC comparative effectiveness study is mentioned in the document. The study described focuses on the standalone performance and accuracy of the software against "gold standard" software, and its usability. There is no comparison of human reader performance with and without AI assistance described.
-
If a standalone (i.e. algorithm only without human-in-the-loop performance) was done:
Yes, a form of standalone performance assessment was done, particularly in the "Accuracy Testing" section. The device's output for implant values (version and inclination) in various simulated scenarios was compared directly to the output of Materialise Innovation Suite (Mimics V22 and 3matic V14) and SolidWork 2016, which serve as the "gold standard" for these measurements. This is an evaluation of the algorithm's performance in generating specific measurements.
-
The type of ground truth used (expert consensus, pathology, outcomes data, etc):
The ground truth for the accuracy testing was established by results from "gold standard" commercial software: Materialise innovation Suite (Mimics V22 and 3matic V14) and SolidWork 2016. For the functional and usability testing, the ground truth was based on expected software behavior and user experience.
-
The sample size for the training set:
The document does not mention the use of a "training set" or machine learning algorithms in the conventional sense that would require a separate training dataset. The device is described as "web-based surgical planning software" that provides analysis tools and 3D visualization. The performance testing focuses on verification, validation, and accuracy against "gold standard" software, rather than the performance of a machine learning model trained on a specific dataset.
-
How the ground truth for the training set was established:
Since no training set is mentioned or described as part of the device's development or evaluation in the document, there is no information on how its ground truth was established.
§ 892.2050 Medical image management and processing system.
(a)
Identification. A medical image management and processing system is a device that provides one or more capabilities relating to the review and digital processing of medical images for the purposes of interpretation by a trained practitioner of disease detection, diagnosis, or patient management. The software components may provide advanced or complex image processing functions for image manipulation, enhancement, or quantification that are intended for use in the interpretation and analysis of medical images. Advanced image manipulation functions may include image segmentation, multimodality image registration, or 3D visualization. Complex quantitative functions may include semi-automated measurements or time-series measurements.(b)
Classification. Class II (special controls; voluntary standards—Digital Imaging and Communications in Medicine (DICOM) Std., Joint Photographic Experts Group (JPEG) Std., Society of Motion Picture and Television Engineers (SMPTE) Test Pattern).