(420 days)
RayStation is a software system for radiation therapy and medical oncology. Based on user input, RayStation proposes treatment plans. After a proposed treatment plan is reviewed and approved by authorized intended users, RayStation may also be used to administer treatments.
The system functionality can be configured based on user needs.
RayStation is a software system for radiation therapy and medical oncology. Based on user input, RayStation proposes treatment plans. After a proposed treatment plan is reviewed and approved by authorized intended users, RayStation may also be used to administer treatments.
The system functionality can be configured based on user needs.
RayStation consists of multiple applications:
- The main RayStation application is used for treatment planning.
- The RayPhysics application is used for commissioning of treatment machines to make them available for treatment planning and used for commissioning of imaging systems.
The devices to be marketed, RayStation/RayPlan 2024A SP3, 2024A and 2023B, contain modified features compared to last cleared version RayStation 12A including:
- Improved sliding window VMAT (Volumetric Modulated Arc Therapy) sequencing
- Higher dose grid resolution for proton PBS (Pencil Beam Scanning)
- Automated field in field planning
- LET optimization (Linear Energy Transfer)
These applications are built on a software platform, containing the radiotherapy domain model and providing GUI, optimization, dose calculation and storage services. The platform uses three Microsoft SQL databases for persistent storage of the patient, machine and clinic settings data.
As a treatment planning system, RayStation aims to be an extensive software toolbox for generating and evaluating various types of radiotherapy treatment plans. RayStation supports a wide variety of radiotherapy treatment techniques and features an extensive range of tools for manual or semi-automatic treatment planning.
The RayStation application is divided in modules, which are activated through licensing. A simplified license configuration of RayStation is marketed as RayPlan.
The provided document is a 510(k) clearance letter for the RayStation/RayPlan 2024A SP3, 2024A, and 2023B devices. It discusses the substantial equivalence of these devices to a predicate device (RayStation 12A).
However, the document does not contain specific acceptance criteria tables nor detailed study results for a single, comprehensive study proving the device meets acceptance criteria in the format typically requested (e.g., a specific clinical validation study with explicitly defined acceptance metrics like sensitivity, specificity, or AUC, and corresponding reported performance values).
Instead, the document describes a broad software verification and validation process, stating that the software underwent:
- Unit Testing
- Integration Testing
- System Level Testing
- Cybersecurity Testing
- Usability Testing (Validation in a clinical environment)
- Regression Testing
For several "Added/updated functions," the document provides a description of the verification and validation data used to demonstrate substantial equivalence and simply states "Yes" under the "Substantially Equivalent?" column if the validation was "successful." The acceptance criteria for these tests are described narratively within the text, not in a consolidated table format with numerical performance outcomes.
Therefore, I cannot generate the requested table of "acceptance criteria and the reported device performance" as a single, consolidated table with numerical results for the entire device's performance against specific, pre-defined acceptance criteria for a single study. The document describes a process of demonstrating substantial equivalence through various verification and validation activities rather than a single, large-scale study with quantitative acceptance criteria for the overall device performance.
However, I can extract the information related to the validation activities for specific features and the general approach to proving substantial equivalence.
Here's a breakdown of the requested information based on the provided document, addressing each point to the best of my ability given the available details:
Acceptance Criteria and Device Performance (Based on provided verification and validation descriptions)
As noted, a single, consolidated table of quantitative acceptance criteria and overall device performance is not provided. Instead, the document describes various verification and validation activities with implicit or explicit pass criteria for individual features or system aspects to demonstrate substantial equivalence to the predicate device.
Below are examples of how some "acceptance criteria" (pass criteria) and "reported performance" are described for specific features. These are not aggregated performance metrics for the entire device but rather success criteria for sub-components or changes.
| Feature/Aspect Tested | Acceptance Criteria (Pass Criteria) Described | Reported Device Performance (as stated in the document) |
|---|---|---|
| Dose compensation point computation for Tomo Synchrony | 1. Calculated values for the center point coordinates are equal to values from the version used in Accuray validation. |
- Calculated values are numerically equal to values obtained from calling the method (regression test).
- Calculated values are exported correctly from RayStation to DICOM (equality between calculated and exported point, only for Helical Tomo Synchrony plans, only in correct DICOM item).
- Calculated values are converted correctly from DICOM to Accuray's system format (equality of point coordinates, only for relevant plan types). | "The successful validation of this feature demonstrates that the device is as safe and effective as the predicate device." (Implies all pass criteria were met). |
| Point-dose optimization in brachy plans | 1. Position from the correct image set is used for point-dose objectives/constraints. - Possible to add optimization objective/constraint to a point, referring to the correct point.
- When adding objective/constraint, selection of function type and dose level is possible and reflected in description.
- Saving and loading an optimization function template containing point objectives/constraints works correctly (loaded functions are same as saved).
- Results from single/multiple point optimization are as expected (dose in point(s) should be equal to specified dose in objective(s)). | "The successful validation of this feature demonstrates that the device is as safe and effective as the predicate device." (Implies all pass criteria were met). |
| Electron Monte Carlo dose engine improvements | Comparing calculated doses with:
- Measured doses obtained from clinics,
- Doses computed in independent, well-established TPS,
- Doses computed with earlier versions of RayStation,
- Doses computed in BEAMnrc/egs++
using Gamma evaluation criteria. | "The successful validation of this feature demonstrates that the device is as safe and effective as the predicate device." (Implies adequate agreement based on Gamma criteria). |
| Evaluation on converted CBCT images for protons | For proton MC/PB dose computation: - Gamma 2%/2mm pass rate above 90%
- Gamma 3%/3mm pass rate above 95% | "The successful validation of this feature demonstrates that the device is as safe and effective as the predicate device." (Implies specified Gamma pass rates were achieved). |
| Overall Device (Software Verification/Validation) | Software specifications conform to user needs and intended uses, and particular requirements implemented through software can be consistently fulfilled. Conformance to applicable requirements and specifications. Successful outcome of unit, integration, system, cybersecurity, usability, and regression testing. Safety and effectiveness validated. | "RayStation/RayPlan 2024A SP3, 2024A and 2023B have met specifications and are as safe, as effective and perform as well as the legally marketed predicate devices." All general software tests (unit, integration, system, cybersecurity, usability, regression) were acceptable/successful. |
Study Details (Based on the document)
Given the nature of the 510(k) submission for a treatment planning system, the "study" is primarily a comprehensive software verification and validation effort to demonstrate substantial equivalence, rather than a single, standalone clinical trial or diagnostic accuracy study.
-
Sample sizes used for the test set and the data provenance:
- Test Set Sample Sizes: Not explicitly stated as a single numerical value for a global "test set." Testing was conducted at multiple levels (unit, integration, system, usability, regression) across various features.
- For "Evaluation on converted CBCT images for protons," it mentions "Test cases consist of CBCTs from the MedPhoton imaging ring on a Mevion S250i system, as well as the on-board CBCT systems on a Varian ProBeam and an IBA P1," implying a set of patient or phantom imaging data, but the exact number of cases/patients is not specified.
- For other features, it refers to "tests," "validation data," or "computed doses" but doesn't quantify the number of distinct data points/cases used.
- Data Provenance:
- Country of Origin: Not specified in the document. Likely internal RaySearch data and potentially data from collaboration with clinical sites, but no specific countries are mentioned.
- Retrospective or Prospective: Not explicitly stated. The verification and validation activities appear to be primarily retrospective (using existing data, phantom measurements, or simulated scenarios) as part of the software development lifecycle, rather than prospective clinical data collection for a specific study.
- Test Set Sample Sizes: Not explicitly stated as a single numerical value for a global "test set." Testing was conducted at multiple levels (unit, integration, system, usability, regression) across various features.
-
Number of experts used to establish the ground truth for the test set and the qualifications of those experts:
- This information is not provided in the document. The document refers to "measured doses obtained from clinics" and "doses computed in independent, well-established TPS" as part of the validation for dose engine improvements, suggesting some form of external or expert-derived ground truth, but the number and qualifications of experts involved are not detailed. For "Evaluation on converted CBCT images for protons," it states "For each case, a ground truth CT image has been prepared to serve as ground truth," implying expert or established reference standard, but again, no details on experts.
-
Adjudication method (e.g., 2+1, 3+1, none) for the test set:
- This information is not provided. The document focuses on computational and functional verification rather than multi-reader clinical assessment.
-
If a multi reader multi case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance:
- No, an MRMC comparative effectiveness study was not explicitly done or reported in this document. The device, RayStation/RayPlan, is a treatment planning system that assists users in creating treatment plans, not primarily an AI-driven image interpretation or diagnostic aid where human reader performance improvement is typically measured. The AI-related feature mentioned is "deep learning segmentation," but the document states, "(The model training is performed offline on clinical CT and structure data.)" It does not detail an MRMC study related to its performance or impact on human readers.
-
If a standalone (i.e. algorithm only without human-in-the-loop performance) was done:
- Yes, standalone (algorithm-only) performance was central to the validation. The document describes extensive "Unit Testing," "Integration Testing," "System Level Testing," and "Dose engine validation" which are all a form of standalone algorithmic evaluation. For example, the Gamma evaluation criteria for dose calculations or the numerical equality checks for dose compensation points are purely algorithmic performance assessments.
-
The type of ground truth used (expert consensus, pathology, outcomes data, etc):
- The ground truth varied depending on the feature being validated:
- "Measured doses" from clinics / Independent TPS computations / BEAMnrc/egs++ calculations: For dose engine validation. This represents a highly accurate, often physical measurement or well-established computational standard.
- "Ground truth CT image": For evaluation of converted CBCT images for protons. This implies a high-quality reference image.
- Internal "expected results" and "specifications": For functional and system-level tests (e.g., for point-dose optimization, the expected result was that the dose in the point should equal the dose specified in the objective).
- "Clinical objectives": Used for plan comparisons (e.g., in segment weight optimization validation), likely representing desired dose distributions defined by clinical experts.
- The ground truth varied depending on the feature being validated:
-
The sample size for the training set:
- The document mentions "deep learning segmentation" and states that "The model training is performed offline on clinical CT and structure data." However, the sample size for this training set is not provided.
-
How the ground truth for the training set was established:
- For "deep learning segmentation," the ground truth for training would implicitly be the "clinical CT and structure data" mentioned. This typically means expert-delineated structures (ROIs) on clinical CT images, but the exact method (e.g., single expert, consensus, specific software tools) is not detailed.
§ 892.5050 Medical charged-particle radiation therapy system.
(a)
Identification. A medical charged-particle radiation therapy system is a device that produces by acceleration high energy charged particles (e.g., electrons and protons) intended for use in radiation therapy. This generic type of device may include signal analysis and display equipment, patient and equipment supports, treatment planning computer programs, component parts, and accessories.(b)
Classification. Class II. When intended for use as a quality control system, the film dosimetry system (film scanning system) included as an accessory to the device described in paragraph (a) of this section, is exempt from the premarket notification procedures in subpart E of part 807 of this chapter subject to the limitations in § 892.9.