Search Results

The KLS Martin Individual Patient Solutions (IPS) Planning System is intended for use as a software system and image segmentation system for the transfer of imaging information from a computerized tomography (CT) medical scan. The input data file is processed by the IPS Planning System and the result is an output data file that may then be provided as digital models or used as input to a rapid prototyping portion of the system that produces physical outputs including anatomical models, guides, and case reports for use in the marking and cutting of cranial bone in cranial surgery. The IPS Planning System is also intended as a pre-operative software tool for simulating / evaluating surgical treatment options. Information provided by the software and device output is not intended to eliminate, replace, or substitute, in whole or in part, the healthcare provider's judgment and analysis of the patient's condition.

The KLS Martin Individual Patient Solutions (IPS) Planning System is a collection of software and associated additive manufacturing (rapid prototyping) equipment intended to provide a variety of outputs to support reconstructive cranial surgeries. The system uses electronic medical images of the patients' anatomy (CT data) with input from the physician, to manipulate original patient images for planning and executing surgery. The system processes the medical images and produces a variety of patient specific physical and/or digital output devices which include anatomical models, guides, and case reports for use in the marking and cutting of cranial bone in cranial surgery.

The provided text is a 510(k) summary for the KLS Martin Individual Patient Solutions (IPS) Planning System. It details the device, its intended use, and comparisons to predicate and reference devices. However, it does not describe specific acceptance criteria and a study dedicated to proving the device meets those criteria in the typical format of a diagnostic AI/ML device submission.

Instead, the document primarily focuses on demonstrating substantial equivalence to a predicate device (K182889) and leveraging existing data from that predicate, as well as two reference devices (K182789 and K190229). The "performance data" sections describe traditional medical device testing (tensile, biocompatibility, sterilization, software V&V) and a simulated design validation testing and human factors and usability testing rather than a clinical study evaluating the accuracy of an AI/ML algorithm's output against a ground truth.

Specifically, there is no mention of:

• Acceptance criteria for an AI/ML model's performance (e.g., sensitivity, specificity, AUC).
• A test set with sample size, data provenance, or ground truth establishment details for AI/ML performance evaluation.
• Expert adjudication methods, MRMC studies, or standalone algorithm performance.

The "Simulated Design Validation Testing" and "Human Factors and Usability Testing" are the closest sections to a performance study for the IPS Planning System, but they are not framed as an AI/ML performance study as requested in the prompt.

Given this, I will extract and synthesize the information available regarding the described testing and attempt to structure it to address your questions, while explicitly noting where the requested information is not present in the provided document.

Acceptance Criteria and Device Performance (as inferred from the document)

The document primarily states that the device passes "all acceptance criteria" for various tests, but the specific numerical acceptance criteria (e.g., minimum tensile strength, maximum endotoxin levels) and reported performance values are generally not explicitly quantified in a table format. The closest to "performance" is the statement that "additively manufactured titanium devices are equivalent or better than titanium devices manufactured using traditional (subtractive) methods."

Since the document doesn't provide a table of acceptance criteria and reported numerical performance for an AI/ML model's accuracy, I will present the acceptance criteria and performance as described for the tests performed:

Detailed Study Information (Based on available text):

1. Sample size used for the test set and the data provenance:

   • Test Set for Simulated Design Validation Testing: A "representative cranial case" was "extrapolated to six (6) distinct age ranges for input data (CT scan) equals output data validation." This implies 6 simulated cases were tested, but no further details on the number of actual CT scans or patients are provided.
   • Test Set for Human Factors and Usability Testing: "Eighteen (18) cases were analyzed" (6 distinct age ranges, with outputs sent to 3 clinical experts, meaning 6 (age ranges) x 3 (experts) = 18 cases analyzed in total by the experts).
   • Data Provenance: Not specified for the "representative cranial case" in simulated design validation. For human factors, it implicitly used outputs derived from the "six (6) distinct age ranges" based on the system's processing. The document does not specify if the data was retrospective or prospective, or the country of origin.

2. Number of experts used to establish the ground truth for the test set and the qualifications of those experts:

   • Simulated Design Validation Testing: Not explicitly stated that experts established ground truth for this test. It seems to be a technical validation against the design specifications.
   • Human Factors and Usability Testing: "Three separate clinical experts" were used to review the outputs. Their qualifications are not specified beyond being "clinical experts." Their role was to analyze for potential use problems and make recommendations, and confirm applicability to real-life situations. This is not the establishment of ground truth in the sense of a diagnostic classification.

3. Adjudication method (e.g. 2+1, 3+1, none) for the test set:

   • No adjudication method is described for either the simulated design validation or human factors/usability testing. The human factors testing involved reviews by multiple experts, but no process for reconciling disagreements or establishing a consensus "ground truth" among them is mentioned; they each provided independent feedback.

4. If a multi-reader multi-case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance:

   • An MRMC comparative effectiveness study was not conducted according or described in this document. The device is not presented as an AI-assisted diagnostic tool that improves human reader performance in the traditional sense. It's a pre-operative planning system that processes CT data to create physical/digital outputs. The "Human Factors and Usability Testing" involved multiple readers (clinical experts) and multiple cases, but it was for usability assessment rather than a comparative effectiveness study of AI assistance.

5. If a standalone (i.e. algorithm only without human-in-the loop performance) was done:

   • The document describes "Software Verification and Validation" which is a form of standalone testing for the software applications used. It states that "all software requirements and specifications were implemented correctly and completely." However, this is a validation of the software's functionality and adherence to specifications, not a performance study of an AI/ML algorithm's accuracy in a diagnostic context. The system is explicitly described as requiring "trained employees/engineers who utilize the software applications to manipulate data and work with the physician to create the virtual planning session," indicating a human-in-the-loop process for generating the final outputs.

6. The type of ground truth used (expert consensus, pathology, outcomes data, etc):

   • Simulated Design Validation Testing: The "ground truth" appears to be the "initial input data (.STL)" and the design specifications; the test verifies that "output data (CT scan) equals output data validation" (likely intended to mean input equals output, or input from CT leads to correct output).
   • Human Factors and Usability Testing: The "ground truth" is effectively the "expert opinion" of the three clinical experts regarding the usability and applicability of the outputs, rather than a definitive medical diagnosis.

7. The sample size for the training set:

   • The document describes the KLS Martin IPS Planning System as using "commercially off-the-shelf (COTS) software applications" (Materialise Mimics and Geomagic® Freeform PlusTM) for image segmentation and manipulation. This implies that the core algorithms were pre-existing and not developed by KLS Martin as a novel AI/ML model that would require a distinct training set outlined in this submission. Therefore, no information on a training set size is provided for the device.

8. How the ground truth for the training set was established:

   • Not applicable, as no training set for a novel AI/ML model by KLS Martin is described. The COTS software validation would have been performed by their respective developers.

Ask a specific question about this device

The KLS Martin Individual Patient Solutions (IPS) Planning System is intended for use as a software system and image segmentation system for the transfer of imaging information from a computerized tomography (CT) medical scan. The input data file is processed by the IPS Planning System and the result is an output data file that may then be provided as digital models or used as input to a rapid prototyping portion of the system that produces physical outputs including anatomical models, guides and case reports for use in the marking of cranial surgery. The IPS Planning System is also intended as a pre-operative software tool for simulating surgical treatment options.

The KLS Martin Individual Patient Solutions (IPS) Planning System is a collection of software and associated additive manufacturing (rapid prototyping) equipment intended to provide a variety of outputs to support reconstructive cranial surgeries. The system uses electronic medical images of the patients' anatomy (CT data) with input from the physician, to manipulate original patient images for planning and executing surgery. The system processes the medical images and produces a variety of patient specific physical and/or digital output devices which include anatomical models, guides, and case reports for use in the marking of cranial bone in cranial surgery.

The KLS Martin Individual Patient Solutions (IPS) Planning System is a software system and image segmentation system used for transferring imaging information from a CT scan. The system processes input data to produce output data files, which can be digital models or physical outputs like anatomical models, guides, and case reports for cranial surgery. It is also a pre-operative software tool for simulating surgical treatment options.

Here's an analysis of the acceptance criteria and supporting studies based on the provided text:

1. Table of Acceptance Criteria and Reported Device Performance:

2. Sample Size Used for the Test Set and Data Provenance:

The document does not explicitly state the sample sizes used for the non-clinical performance tests (tensile, bending, biocompatibility, sterilization, pyrogenicity, software V&V). The data provenance for these tests is internal to the manufacturer, as they were "performed on the subject polyamide guides," "conducted on devices manufactured from polyamide," and for titanium, "leveraged from the reference device." This indicates the studies were likely conducted in a controlled lab or manufacturing environment. The studies are not described as retrospective or prospective clinical studies, but rather as non-clinical engineering and material performance tests.

3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications of Those Experts:

This information is not applicable. The device is a planning and segmentation system that outputs digital models or physical guides/anatomical models. The acceptance criteria and performance data described relate to the physical and software characteristics of these outputs (e.g., material strength, sterility, software functionality), not to a diagnostic or treatment outcome for which expert ground truth would be established from patient data. The process involves a physician providing input for model manipulation and interactive feedback, but this is part of the planning workflow, not establishing ground truth for a test set in the context of algorithm performance.

4. Adjudication Method for the Test Set:

This information is not applicable, as the described studies are non-clinical performance tests focused on material properties, sterility, and software functionality, not on qualitative assessments requiring adjudication from expert readers.

5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was Done:

No, an MRMC comparative effectiveness study was not done. The document explicitly states: "Clinical testing was not necessary for the determination of substantial equivalence."

6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) Study was Done:

Yes, the "Software Verification and Validation" section describes testing of individual software applications. This indicates a standalone assessment of the software's performance in implementing requirements and specifications, separate from a human-in-the-loop clinical scenario.

7. The Type of Ground Truth Used (expert consensus, pathology, outcomes data, etc.):

For the non-clinical tests:

• Tensile & Bending Testing: Ground truth is based on established engineering standards and material science principles (e.g., 85% retention of initial tensile strength after sterilization, comparison to traditional manufacturing methods).
• Biocompatibility Testing: Ground truth refers to pre-defined acceptance criteria in accordance with ISO 10993-1.
• Sterilization Testing: Ground truth is based on achieving a sterility assurance level (SAL) of $10^{-6}$ as per ISO 17665-1:2006.
• Pyrogenicity Testing: Ground truth is based on meeting USP allowed limits for endotoxin levels (< 2.15 EU/device) and pyrogen limit specifications.
• Software Verification & Validation: Ground truth is defined by "all software requirements and specifications" and "pre-defined specifications and acceptance criteria" derived from user requirements and risk analysis.

8. The Sample Size for the Training Set:

The document does not mention a training set. This device is described as utilizing a collection of software and additive manufacturing equipment for planning and producing physical outputs based on patient CT data, not a machine learning model that would require a dedicated training set. The software applications used (Materialise Mimics, Geomagic® Freeform Plus™) are "comedically off-the-shelf (COTS) software applications," implying they were developed and validated independently, prior to their integration into this system.

9. How the Ground Truth for the Training Set Was Established:

This information is not applicable, as there is no mention of a training set for a machine learning algorithm in the context of this device.

Ask a specific question about this device

The 3D Systems, Inc. VSP Cranial System is intended for use as a collection of software to provide image segmentation and transfer of imaging information from a CT based medical scanner. The is processed by the VSP Cranial System and the result is an output data file that may then be provided as digital models or used as input in the production of physical outputs including anatomical models, templates, and surgical guides for use in the marking of cranial bone in cranial surgery.

The 3D Systems VSP® Cranial System is a collection of Commercial Off-The-Shelf (COTS) software, third party medical device software, and custom software intended to provide a variety of outputs to support cranial reconstructive surgery. The system uses CT based imaging data of the patient's anatomy with input from the physician, to manipulate original patient images for planning and executing surgery. The system produces a variety of patient specific outputs including, anatomical models (physical and digital), surgical templates / guides, and patient specific case reports.

Here's a breakdown of the acceptance criteria and the study information for the VSP Cranial System, based on the provided text:

1. Table of Acceptance Criteria and Reported Device Performance:

2. Sample Size Used for the Test Set and Data Provenance:

The document details performance testing that was largely leveraged from the predicate device (VSP® System, K133907) because the planning/design process, materials, manufacturing process, cleaning methods, and sterilization methods are identical.

For the tests specific to the VSP Cranial System:

• Sample Size: Not explicitly stated but "sizes and shapes of VSP® Cranial System templates and guides were selected to challenge the system" for Dimensional Analysis. For bioburden and pyrogenicity, "VSP® Cranial System templates, guides, anatomical models, and metal accessories" were tested.
• Data Provenance: This appears to be prospective testing conducted by 3D Systems for the specific VSP Cranial System device. The country of origin of the data is not specified beyond being generated by the applicant, 3D Systems, Inc., which is based in Littleton, Colorado, USA.

3. Number of Experts Used to Establish the Ground Truth for the Test Set and Their Qualifications:

This information is not provided in the document. The testing described focuses on engineering, material, and sterilization validation rather than clinical performance or diagnostic accuracy that would require expert-established ground truth for a test set.

4. Adjudication Method for the Test Set:

This information is not provided in the document. As mentioned above, the tests are primarily engineering and material validations, which don't typically involve an adjudication method in the way a clinical study for diagnostic accuracy would.

5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was done:

No, an MRMC comparative effectiveness study was not done. The study described is not a clinical study involving human readers or comparative effectiveness of AI assistance. It's a technical performance and validation study to demonstrate substantial equivalence to a predicate device.

6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) was done:

The device is described as a "collection of software" that provides image segmentation and processing, resulting in an "output data file that may then be provided as digital models or used as input in the production of physical outputs." The system is operated by "trained 3D Systems employees," and the "physician does not directly input information" but provides "clinical input and review."

While there is "software verification and validation testing" (leveraged from the predicate), the document does not explicitly describe a standalone performance study for the algorithm's output in terms of accuracy against a ground truth independent of the human-in-the-loop (3D Systems employees and physician review) process. The nature of this device (planning tools for customized physical outputs) means the "human-in-the-loop" is integral to its intended use and output generation.

7. The Type of Ground Truth Used:

For the specific tests performed:

• Packaging Validation: Ground truth is defined by the technical specifications of the ASTM and National Motor Freight Classification standards.
• Sterilization Compatibility: Ground truth is visual and dimensional integrity after a validated sterilization cycle.
• Dimensional Analysis: Ground truth is the "product requirements" and specified dimensional tolerances.
• Bioburden: Ground truth is defined by the acceptable bioburden limits specified in ISO 11737-1, USP <61>, and USP <1227>.
• Pyrogenicity: Ground truth is the acceptance criteria of ≤ 2.15 EU/device as per AAMI ST72, USP <85>, and USP <161>.

For the overall system, the "ground truth" for its functionality is implicitly linked to the predicate device's established performance and safety/effectiveness, as the core technologies and processes are leveraged and deemed "substantially equivalent."

8. The Sample Size for the Training Set:

This information is not applicable/not provided. The VSP Cranial System is not described as a machine learning or AI algorithm that requires a "training set" in the conventional sense for developing predictive models. It's a software system for image segmentation and design of physical outputs based on CT data.

9. How the Ground Truth for the Training Set was Established:

This information is not applicable/not provided for the same reasons as point 8.

Ask a specific question about this device