(140 days)
The OrthoNext™ Platform System is indicated for assisting healthcare professionals in preoperative planning of orthopedic surgery and post-operative planning of orthopedic treatment. The device allows for overlaying of Orthofix Product templates on radiological images, and includes tools for performing measurements on the image and for positioning the template. Clinical judgments and experience are required to properly use the software.
The subject OrthoNext™ Platform System is a web-based modular software system, indicated for assisting healthcare professionals in planning of orthopedic surgery and treatment both preoperatively and postoperatively, including deformity analysis and correction with several Orthofix products.
The subject software system is intended for use by Healthcare Professionals (HCP), with full awareness of the appropriate orthopedic procedures, in the operating theatre only.
The subject software functions are intended to inform the HCP on orthopedic procedure treatment planning when the Orthofix external or internal fixation systems are used. These functions are evidence-based tools that support HCP when considering treatment digital planning options for a patient. The software functions do not treat a patient or determine a patient's treatment.
The software enables the HCP to import radiological images, display 2D views (frontal and lateral) of the radiological images, overlay the positioning of the template and simulate the treatment plan option, and to generate parameters and/or measurements to be verified or adjusted by the HCP based on their clinical judgment.
Here's a breakdown of the acceptance criteria and the study details for the OrthoNext Platform System, based on the provided document:
Acceptance Criteria and Device Performance
The OrthoNext Platform System was evaluated for measurement accuracy and an AI/ML algorithm's performance for automatic marker detection.
Table of Acceptance Criteria and Reported Device Performance
| Feature / Metric | Acceptance Criteria | Reported Device Performance |
|---|---|---|
| Measurement Accuracy | Overall accuracy verified under a representative worst-case scenario. | For measurements made using anatomical axes: |
- Mean error: 0.1 mm for linear measurements, 0.05 degrees for angular measurements.
- Mean percentage error: 0.24% (threshold 0.27% within 95th percentile) for linear measurements, 0.08% (threshold 1% within 95th percentile) for angular measurements.
For measurements made using mechanical axes:
- Mean error: 0.4 mm for linear measurements, 0.06 degrees for angular measurements.
- Mean percentage error: 1.73% (threshold 16.67% within 95th percentile) for linear measurements, 0.28% (threshold 1.27% within 95th percentile) for angular measurements. |
| Sensitivity of Measurements | Not explicitly stated as a separate acceptance criterion, but device performance reported. | 1 mm for linear measurements, 1 degree for angular measurements. |
| Specificity of Measurements | Not explicitly applicable as a direct acceptance criterion due to manual nature. | The device requires active user involvement for each measurement, relying on user expertise. |
| AI/ML Algorithm Accuracy | Not explicitly stated as an isolated acceptance criterion, but reported. | 0.8 |
| AI/ML Algorithm Specificity | Goal: zero false positives in the test set, resulting in a precision of 1. | 1 (Precision) |
| AI/ML Algorithm Sensitivity | Not explicitly stated as an isolated acceptance criterion, but reported. | 0.75 (TPR/Recall) |
| AI/ML Algorithm FPR | Not explicitly stated as an isolated acceptance criterion, but reported. | 0 |
| AI/ML Algorithm F1 Score | Not explicitly stated as an isolated acceptance criterion, but reported. | 0.86 |
| AI/ML Algorithm Intersection over Union (IoU) | Not explicitly stated as a direct acceptance criterion, but reported. | 79% |
| AI/ML Algorithm Center MAE | Not explicitly stated as a direct acceptance criterion, but reported. | 4.83 px |
| AI/ML Algorithm Center MAPE | Not explicitly stated as a direct acceptance criterion, but reported. | 0.38% |
| AI/ML Algorithm Radius MAE | Not explicitly stated as a direct acceptance criterion, but reported. | 1.29 px |
| AI/ML Algorithm Radius MAPE | Satisfying 3% Radius MAPE. | 3% Radius MAPE, contributing to an error of less than 1 mm. |
Study Details: Magnification Marker Detection Algorithm
The document focuses on the performance testing for the magnification marker detection algorithm, which is an AI/ML component of the OrthoNext Platform System.
-
Sample size used for the test set and the data provenance:
- Sample Size: 1000 X-ray images. Of these, 800 images depicted a magnification marker, and 200 images did not.
- Data Provenance: The test set consisted of real X-ray images. The document does not specify the country of origin of the data or whether it was retrospective or prospective. It only states that these images were "not used during training, ensuring independence."
-
Number of experts used to establish the ground truth for the test set and the qualifications of those experts:
- Number of Experts: Not explicitly stated. The document mentions "qualified personnel" were used for the truthing process.
- Qualifications of Experts: "Qualified personnel" are mentioned, but specific qualifications (e.g., radiologist with X years of experience) are not provided.
-
Adjudication method for the test set:
- The document describes a "review and discard process" implemented to ensure the quality of the annotations, but it does not specify an adjudication method like "2+1" or "3+1". This suggests a quality control step for annotations rather than a formal consensus process among multiple readers for ground truth establishment.
-
If a multi reader multi case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance:
- No MRMC comparative effectiveness study was done or reported in this document. The study evaluates the standalone performance of the AI/ML algorithm for magnification marker detection, not its impact on human reader performance.
-
If a standalone (i.e. algorithm only without human-in-the-loop performance) was done:
- Yes, a standalone performance evaluation of the AI/ML algorithm for magnification marker detection was explicitly done and reported. The performance metrics (Precision, Accuracy, TPR/Recall, FPR, F1 Score, IoU, MAE, MAPE) are all indicative of standalone algorithm performance.
-
The type of ground truth used:
- Expert Consensus (Annotation): The ground truth was established through a "truthing process... conducted by qualified personnel, who carefully overlaid a circular shape on each magnification marker using annotation software."
-
The sample size for the training set:
- Training Set Composition: 1500 X-ray images with random areas depicting magnification markers.
- These 1500 magnification markers were randomly overlaid on top of 4000 X-ray images.
- Image augmentation techniques (random rotations, brightness adjustments) generated 24,000 unique X-ray images for training.
-
How the ground truth for the training set was established:
- The training set involved synthetic images. Specifically, "1500 X-ray images with random areas depicting magnification markers" were created, and these markers were "randomly overlaid on top of 4000 X-ray images." Image augmentation was then applied. This suggests that the ground truth for the training set markers was generated as part of the synthetic image creation process (i.e., the location and characteristics of the overlaid markers were known by design). A "hash check" was used to ensure the uniqueness of these synthetic images.
§ 892.2050 Medical image management and processing system.
(a)
Identification. A medical image management and processing system is a device that provides one or more capabilities relating to the review and digital processing of medical images for the purposes of interpretation by a trained practitioner of disease detection, diagnosis, or patient management. The software components may provide advanced or complex image processing functions for image manipulation, enhancement, or quantification that are intended for use in the interpretation and analysis of medical images. Advanced image manipulation functions may include image segmentation, multimodality image registration, or 3D visualization. Complex quantitative functions may include semi-automated measurements or time-series measurements.(b)
Classification. Class II (special controls; voluntary standards—Digital Imaging and Communications in Medicine (DICOM) Std., Joint Photographic Experts Group (JPEG) Std., Society of Motion Picture and Television Engineers (SMPTE) Test Pattern).