(84 days)
RayStation is a software system for radiation therapy and medical oncology. Based on user input, RayStation proposes treatment plans. After a proposed treatment plan is reviewed and approved by authorized intended users, RayStation may also be used to administer treatments.
RayStation is a treatment planning system for planning, analysis and administration of radiation therapy and medical oncology treatment plans. It has a modern user interface and is equipped with fast and accurate dose and optimization engines.
RayStation consists of multiple applications:
- . The main RayStation application is used for treatment planning.
- . The RayPhysics application is used for commissioning of treatment machines to make them available for treatment planning and used for commissioning of imaging systems.
- . The RayTreat application is used for sending plans to treatment delivery devices for treatment and receiving records of performed treatments.
The device to be marketed, RayStation 11.0, marketing name "RayStation 11A", contains modified features compared to version 10.1. The main change is planning for the CyberKnife linear accelerator treatment unit.
The device to be marketed supports planning and dose calculation for M6/S7 CyberKnife treatment machines. Older CyberKnife versions are not supported. The CyberKnife has one delivery technique, and different node sets. The node set defines the possible robot positions for different collimations (cones or MLC) and patient geometry (head or body). All node sets available for patient treatments are included in the validation.
The document provided does not contain specific acceptance criteria for a device in the traditional sense of a medical diagnostic or AI-powered device. Instead, it describes a software system, RayStation 11.0, which is a radiation therapy treatment planning system. The document focuses on demonstrating substantial equivalence to a predicate device (RayStation 10.1) and validating its functionality for a new feature: planning for CyberKnife linear accelerator treatment units.
Therefore, the "acceptance criteria" discussed are more about the software's performance meeting its specifications and being suitable for clinical use, rather than quantitative performance metrics for a diagnostic output.
Based on the provided text, here's an attempt to answer your questions by interpreting "acceptance criteria" as the measures taken to validate the software's functionality and safety:
Table of Acceptance Criteria and Reported Device Performance
Given that RayStation is a treatment planning system and not a diagnostic device with specific output metrics like sensitivity/specificity, the "acceptance criteria" are related to the accuracy and safety of its dose calculations and overall system functionality for treatment planning. The document broadly states the "validation shows that the dose computation is suited for clinical use" and "adequate for clinical use."
| Acceptance Criteria Category | Reported Device Performance |
|---|---|
| Dose Computation Accuracy | CyberKnife specific photon dose engine validation (collapsed cone and photon Monte Carlo) performed. Validation shows "dose computation is suited for clinical use" and "adequate for clinical use." |
| System Functionality | "System Tests of RayStation" passed. |
| "Risk analysis-based tests for use error mitigation verification" passed. | |
| Software Quality | "Unit and subsystem testing for low-level testing" passed. |
| "Reviews of design, code and Master Labeling" passed. | |
| User Validation | "User validation in cooperation with cancer clinics" completed successfully. |
Study that proves the device meets the acceptance criteria:
The provided document describes the verification and validation activities performed for RayStation 11.0. This is not a single "study" in the sense of a clinical trial with a defined sample size of patients with a specific condition. Instead, it's a comprehensive software development and quality assurance process.
1. Sample sizes used for the test set and the data provenance:
- Test Set Sample Size: The document does not specify a numerical "sample size" for a typical test dataset as one would expect in an AI/diagnostic device study. Instead, it refers to various testing activities:
- CyberKnife Validation: "All node sets available for patient treatments are included in the validation." This implies comprehensive testing across different configurations of the CyberKnife treatment unit. It doesn't quantify the number of patient plans or variations tested.
- System Tests, Unit/Subsystem Tests, Risk Analysis-based Tests: These are internal software testing activities, and their "sample size" would be defined by the number of test cases executed, which is not detailed.
- User Validation: No specific number of cases or users is given.
- Data Provenance: Not explicitly stated for specific test cases. Given it's a software for radiation therapy planning, the "data" would consist of simulated patient anatomies, treatment plans, and test configurations. The "user validation in cooperation with cancer clinics" implies real-world clinical context, but details are not provided. It's implicitly retrospective as it's part of a software release validation. No country of origin is specified for the data used in testing.
2. Number of experts used to establish the ground truth for the test set and their qualifications:
- CyberKnife Validation: The validation was performed internally by RaySearch Laboratories, likely by their physicists and engineers. The document does not specify the number of external experts or their qualifications for establishing a "ground truth" for the dose calculations. The "ground truth" for dose calculations would be established through established physics models and comparisons to known phantoms or reference data, not necessarily expert consensus in the typical sense of image interpretation.
- User Validation: Conducted "in cooperation with cancer clinics." This indicates involvement of clinical experts (e.g., radiation oncologists, medical physicists) who would evaluate the usability and clinical appropriateness of the plans generated. The number and qualifications of these experts are not specified.
- General: The document does not mention the use of experts to establish 'ground truth' in the context of diagnostic interpretation, which is common for AI-powered image analysis tools. For a treatment planning system, 'ground truth' relates to the accuracy of physics calculations and the clinical utility of the generated plans.
3. Adjudication method for the test set:
- No explicit adjudication method (e.g., "2+1" or "3+1") is described, as the validation is not focused on subjective interpretation or a diagnostic outcome requiring such consensus. For dose calculations, discrepancies would be resolved by identifying and correcting calculation errors. For user validation, feedback would be gathered and addressed.
4. If a multi-reader multi-case (MRMC) comparative effectiveness study was done, and its effect size:
- No MRMC study was performed or described. The study type is focused on validating the software's internal performance and its capability to generate treatment plans, not on how human readers (e.g., radiologists) improved their performance using the software. RayStation is a planning tool, not an AI-assisted diagnostic tool for human readers.
5. If a standalone (i.e., algorithm only without human-in-the-loop performance) was done:
- Yes, standalone performance was assessed for core functionalities. The "Dose engine validation" for the CyberKnife is a standalone assessment of the algorithm's accuracy in calculating dose. "System Tests," "Unit and subsystem testing," and "Risk analysis-based tests" also represent assessments of the algorithm's performance without direct human intervention during the test execution. The "user validation" component does involve humans in the loop to assess usability and clinical utility.
6. The type of ground truth used:
- For Dose Calculations: The ground truth for dose engine validation would be based on physical principles, established dose measurement techniques (e.g., phantom measurements), and validated physics models. It's about the objective accuracy of numerical calculations.
- For System Functionality: The ground truth for system tests would be the expected output or behavior based on the software's design specifications and requirements.
- For Clinical Appropriateness/Usability (User Validation): This would be based on expert clinical judgment from the cancer clinics involved, but it's not a "ground truth" for a diagnostic outcome, rather an assessment of the tool's utility.
7. The sample size for the training set:
- Not applicable / Not provided. RayStation is a physics-based treatment planning system, not typically an AI/machine learning model in the sense of being "trained" on a large dataset of patient images or outcomes to learn patterns. While it might contain machine learning features (the document mentions "Related to machine learning, there is no change compared to the predicate device," implying some existing ML, but not as the primary function or new feature for the 11.0 update), the core CyberKnife planning feature validated here is based on physics algorithms, not a trained model. Therefore, there's no "training set" in the context of supervised machine learning.
8. How the ground truth for the training set was established:
- Not applicable / Not provided. As explained above, the core functionality validated (CyberKnife planning) is algorithm-driven, not learned from a training set. If there are existing ML components, the document doesn't detail their training or ground truth establishment.
§ 892.5050 Medical charged-particle radiation therapy system.
(a)
Identification. A medical charged-particle radiation therapy system is a device that produces by acceleration high energy charged particles (e.g., electrons and protons) intended for use in radiation therapy. This generic type of device may include signal analysis and display equipment, patient and equipment supports, treatment planning computer programs, component parts, and accessories.(b)
Classification. Class II. When intended for use as a quality control system, the film dosimetry system (film scanning system) included as an accessory to the device described in paragraph (a) of this section, is exempt from the premarket notification procedures in subpart E of part 807 of this chapter subject to the limitations in § 892.9.