(100 days)
The Monaco system is used to make treatment plans for patients with prescriptions for external beam radiation therapy. The system calculates dose for photon treatment plans and displays, on-screen and in hard-copy, two- or three-dimensional radiation dose distributions inside patients for given treatment plan set-ups.
The Monaco product line is intended for use in radiation treatment planning. It uses generally accepted methods for:
- contouring
- image manipulation
- simulation
- image fusion
- plan optimization
- QA and plan review
Monaco is a radiation treatment planning system that first received FDA clearance in 2007 (K071938). The modified system received clearance in 2009, when Volumetric Modulated Arc Therapy (VMAT) planning capability was added (K091179). The Monaco system accepts patient diagnostic imaging data and "source" dosimetry data from a linear accelerator. The system then permits the user to display and define (contour) the target volume to be treated and critical structures which must not receive above a certain level of radiation on these diagnostic images.
Based on the prescribed dose, the user, a Dosimetrist or Medical Physicist, can create multiple treatment scenarios involving the number, position(s) and energy of radiation beams and the use of a multileaf collimator (MLC) between the source of radiation and the patient to shape the beam. The Monaco system then produces a display of radiation dose distribution within the patient, indicating doses to the target volume and surrounding structures. The "best" plan satisfying the prescription is then selected, one that maximizes dose to the target volume while minimizing dose to surrounding healthy volumes.
The Monaco RTP System is a radiation treatment planning system. Here's a breakdown of its acceptance criteria and the supporting study:
1. Table of Acceptance Criteria and Reported Device Performance
The provided summary does not explicitly list distinct, quantifiable acceptance criteria with corresponding performance metrics in a readily extractable table format for dose calculation or planning accuracy. Instead, it states that verification tests were "written and executed to ensure that the system is working as designed" and that "Pass/fail requirements and results of this testing can be found in section 18 of this submission." However, Section 18 is not included in the provided text.
Based on the available information, the general performance criteria can be inferred as:
| Acceptance Criteria (Inferred from intended use and testing descriptions) | Reported Device Performance |
|---|---|
| Accurate dose calculation for photon treatment plans | "Algorithm testing was performed to compare calculated against measured doses to ensure dose calculation accuracy." The system "successfully passed verification testing." |
| Capability for contouring | Yes |
| Capability for image manipulation | Yes |
| Capability for simulation | Yes (CT Simulation) |
| Capability for image fusion | Yes |
| Capability for plan optimization | Yes |
| Capability for QA and plan review | Yes |
| Support for Dynamic Conformal capability | Yes, as a new feature of the Monaco RTP System. The system supports dynamic delivery methods. |
| Overall system functionality as designed | "Verification tests were written and executed to ensure that the system is working as designed... Monaco successfully passed verification testing." The product was "deemed fit for clinical use." |
2. Sample Size Used for the Test Set and the Data Provenance
The summary states that "Clinical trials were not performed as part of the development of this product." Instead, "Algorithm testing was performed to compare calculated against measured doses," and "clinically oriented validation test cases were written and executed in-house by CMS customer support personnel."
Therefore:
- Test Set Sample Size: Not specified in terms of number of patient cases. The testing involved "algorithm testing" (comparing calculated vs. measured doses) and an unspecified number of "clinically oriented validation test cases."
- Data Provenance: Not explicitly stated regarding origin (e.g., country). However, the testing was "in-house" by the manufacturer (Computerized Medical Systems, Inc., USA). This implies the data used for the algorithm and validation tests would be internally generated or sourced. The context suggests it was not patient data from clinical settings.
- Retrospective/Prospective: The testing appears to be retrospective in the sense that it did not involve prospective human subjects but rather validation against pre-existing data (measured doses) or simulated/representative cases for the "clinically oriented validation test cases."
3. Number of Experts Used to Establish the Ground Truth for the Test Set and the Qualifications of Those Experts
- Number of Experts: Not explicitly stated. The "clinically oriented validation test cases" were "written and executed in-house by CMS customer support personnel."
- Qualifications of Experts: The personnel were "CMS customer support personnel." While they handled "clinically oriented" test cases, their specific clinical qualifications (e.g., medical physicist, dosimetrist, or specific years of experience) are not provided. The summary also notes that plans are "reviewed and approved by qualified clinicians" in a clinical setting, but this refers to post-approval clinical use, not the ground truth establishment for the premarket testing.
4. Adjudication Method for the Test Set
The document does not describe an adjudication method for establishing ground truth for the test set. Since the testing involved "algorithm testing" comparing calculated against measured doses, and "clinically oriented validation test cases" executed in-house, it is unlikely a multi-expert adjudication method was employed in the traditional sense. The "ground truth" for algorithmic accuracy would be established by the physical measurements, and for validation cases, by adherence to predefined clinical expectations or specifications.
5. If a Multi Reader Multi Case (MRMC) Comparative Effectiveness Study Was Done, If so, What Was the Effect Size of How Much Human Readers Improve with AI vs Without AI Assistance?
No, a Multi Reader Multi Case (MRMC) comparative effectiveness study was not done. The device is a radiation treatment planning system, not an AI-assisted diagnostic tool for human readers. Its primary function is to calculate dose and aid in plan creation, not to improve human reader performance in interpreting images or making diagnoses.
6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) Was Done
Yes, a form of standalone performance assessment was done. "Algorithm testing was performed to compare calculated against measured doses to ensure dose calculation accuracy." This directly evaluates the algorithm's output (calculated dose) against an objective standard (measured dose) without a human-in-the-loop decision-making process. The "clinically oriented validation test cases" also assessed the system's ability to produce acceptable plans based on defined criteria.
7. The Type of Ground Truth Used
- For Algorithm Testing: The ground truth was measured doses. The summary states "Algorithm testing was performed to compare calculated against measured doses." This implies physical measurements were used as the gold standard.
- For "Clinically Oriented Validation Test Cases": The ground truth was based on predefined clinical expectations/specifications or internal standards established by the CMS customer support personnel who wrote and executed these cases.
8. The Sample Size for the Training Set
The document does not specify a separate "training set" sample size. The Monaco system is a radiation treatment planning system that calculates dose and optimizes plans based on established physics models and algorithms. It does not appear to be a machine learning model that requires a distinct "training set" in the common understanding of AI devices. Its development would involve calibration, verification, and validation, rather than a training process on a large dataset of patient images or outcomes.
9. How the Ground Truth for the Training Set Was Established
Since a "training set" in the context of machine learning is not mentioned or implied for this device, the method for establishing its ground truth is not applicable/not provided. The system's foundational accuracy would stem from its underlying physical models and their calibration, which would involve experimental data and established physics principles, rather than a labeled training dataset.
§ 892.5050 Medical charged-particle radiation therapy system.
(a)
Identification. A medical charged-particle radiation therapy system is a device that produces by acceleration high energy charged particles (e.g., electrons and protons) intended for use in radiation therapy. This generic type of device may include signal analysis and display equipment, patient and equipment supports, treatment planning computer programs, component parts, and accessories.(b)
Classification. Class II. When intended for use as a quality control system, the film dosimetry system (film scanning system) included as an accessory to the device described in paragraph (a) of this section, is exempt from the premarket notification procedures in subpart E of part 807 of this chapter subject to the limitations in § 892.9.