(154 days)
The Monaco system is used to make treatment plans for patients with prescriptions for external beam radiation therapy. The system calculates dose for photon, proton, and electron treatment plans and displays, on-screen and in hard-copy, two- or threedimensional radiation dose distributions inside patients for given treatment plan setups.
The Monaco product line is intended for use in radiation treatment planning. It uses generally accepted methods for:
- · contouring
- · image manipulation
- · simulation
- · image fusion
- · plan optimization
- QA and plan review
Monaco is a radiation treatment planning system that first received FDA clearance in 2007 (K071938). The modified system received clearance in 2009. when Volumetric Modulated Arc Therapy (VMAT) planning capability was added (K091179), again when Dynamic Conformal Arc planning was added (K110730), and electron planning, support for stereotactic cones, and SUV calculation were added (K132971). Specialty image creation was added in 2015 (K151233), and adaptive planning and dose calculation in the presence of a magnetic field (e.g., MR-Linac) was added in 2018 (K183037). A 510(k) was filed in 2017 for the addition of carbon ion planning. The 510(k) was withdrawn because there was no hardware cleared for the US market capable of delivering carbon ion plans. Monaco's carbon ion planning functionality remains licensed off and inaccessible to US users.
The Monaco system accepts patient imaging data and "source" dosimetry data from a linear accelerator. The system then permits the user to display and define (contour) the target volume to be treated and critical structures which must not receive above a certain level of radiation on these diagnostic images.
Based on the prescribed dose, the user, a Dosimetrist or Medical Physicist, can create multiple treatment scenarios involving the number, position(s) and energy of radiation beams and the use of a beam modifier (MLC, block, etc.) between the source of radiation and the patient to shape the beam. The Monaco system then produces a display of radiation dose distribution within the patient, indicating doses to the target volume and surrounding structures. The "best" plan satisfying the clinican prescription is then selected, one that maximizes dose to the target volume while minimizing dose to surrounding healthy volumes.
Monaco 6.00 supports Proton Pencil Beam Scanning (Proton PBS) planning for IBA Proteus®ONE and Proteus®PLUS delivery systems (Ion Beam Applications S.A.).
The provided text is a 510(k) summary for the Elekta Monaco RTP System. It describes the device, its intended use, and a comparison to predicate devices, but it explicitly states that "Clinical trials were not performed as part of the development of this product. Clinical testing on patients is not advantageous in demonstrating substantial equivalence or safety and effectiveness of the device since testing can be performed such that no human subjects are exposed to risk. Validation testing involved simulated clinical workflows using actual patient data, such as patient images. Pre-defined pass/fail criteria were also equivalent to that of the previous version of Monaco."
Therefore, I cannot provide a detailed answer to your request regarding acceptance criteria and a study proving the device meets them in the way you've outlined, as there was no clinical study (MRMC, standalone, etc.) involving human readers or a traditional test set/ground truth establishment as would be done for an AI-based diagnostic device.
The study referenced is a non-clinical verification study that focuses on software functionality, safety, and effectiveness compared to an existing predicate device, primarily through regression testing and verification of new functionalities. The acceptance criteria are "pre-defined pass/fail criteria" that were equivalent to those used for the previous version of Monaco.
However, based on the information provided, I can infer and summarize what was done:
1. A table of acceptance criteria and the reported device performance:
The document broadly states that "Pre-defined pass/fail criteria were also equivalent to that of the previous version of Monaco." and "Conformity to the same pass/fail criteria as the predicate version of Monaco indicated that Monaco 6.00 was substantially equivalent in safety and effectiveness."
While specific numeric acceptance criteria and performance metrics are not detailed in this summary, the overall acceptance criterion was Substantial Equivalence to the predicate devices. The performance reported is that "Monaco 6.00 was deemed safe and effective for its intended use" based on the non-clinical testing.
| Acceptance Criteria (Internal Software Validation/Verification) | Reported Device Performance (Summary) |
|---|---|
| Equivalence to previous Monaco version's pass/fail criteria | Deemed safe and effective, and substantially equivalent |
| Verification of new product functionality (e.g., Proton PBS) | "System is working as designed" |
| Risk mitigations functioning as intended | Ensured continued safety and effectiveness |
| Regression tests to ensure continued safety and effectiveness | Ensured continued safety and effectiveness for existing functionality |
| Conformity to FDA Quality System Regulation (21 CFR §820) | Met regulations |
| Conformity to ISO 13485 Quality Management System standard | Met standards |
| Conformity to IEC 62304 Software Life Cycle standard | Met standards |
| Conformity to ISO 14971 Risk Management Standard | Met standards |
2. Sample size used for the test set and the data provenance:
- Sample Size for Test Set: The document mentions "Over 600 test procedures were executed" and "Validation testing involved simulated clinical workflows using actual patient data, such as patient images." The exact number of patient datasets or specific test cases within those 600 procedures is not specified.
- Data Provenance: "actual patient data, such as patient images." No specific country of origin is mentioned, but "simulated clinical workflows" suggests internally generated or existing de-identified data. The testing was retrospective in the sense that it used pre-existing patient data for simulation, not prospective patient enrollment.
3. Number of experts used to establish the ground truth for the test set and the qualifications of those experts:
- This wasn't a ground truth establishment study for a diagnostic AI. The "ground truth" for this system's validation was primarily the expected output of the algorithms (e.g., dose calculations, plan optimizations) as per documented specifications and comparisons to known good results from the predicate device.
- The document states that "Once completed, plans are reviewed and approved by qualified clinicians and may be subject to quality assurance practices before treatment actually takes place." This implies that the system is used by "Dosimetrist or Medical Physicist" and reviewed by "qualified clinicians" in a clinical setting, but these roles were not part of a formal "ground truth" establishment for the validation study itself.
4. Adjudication method (e.g., 2+1, 3+1, none) for the test set:
- Not applicable. This was a software verification and validation against specified requirements and predicate performance, not a clinical adjudication of diagnostic findings.
5. If a multi-reader multi-case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance:
- No MRMC comparative effectiveness study was performed. The document explicitly states: "Clinical trials were not performed as part of the development of this product." and "Clinical testing on patients is not advantageous in demonstrating substantial equivalence or safety and effectiveness of the device since testing can be performed such that no human subjects are exposed to risk."
6. If a standalone (i.e. algorithm only without human-in-the-loop performance) was done:
- The testing described is essentially a form of "standalone" algorithm verification in a simulated environment, focused on the software's functional correctness for tasks like dose calculation and plan optimization, rather than a diagnostic AI's performance. The product itself is a "Radiation Treatment Planning System," which is inherently a human-in-the-loop device, where the software outputs are reviewed and approved by clinicians before implementation. The verification ensured the software's outputs were correct according to its specifications and predicate performance.
7. The type of ground truth used (expert consensus, pathology, outcomes data, etc.):
- The "ground truth" for this validation was primarily derived from:
- Validated algorithms/physical models: For dose calculation, the "ground truth" is based on established physics principles and validated dose calculation algorithms (Monte Carlo, Collapsed Cone, Pencil Beam).
- Predicate device performance: Functional equivalence and similar calculation results to the previously cleared Monaco RTP System (K190178) and RayStation 8.1 (K190387).
- Pre-defined pass/fail criteria: These would be based on engineering specifications, clinical requirements for accuracy in dose distribution, and comparison to known good results for test cases.
- "Simulated clinical workflows using actual patient data": This implies that for these simulated workflows, the expected correct outcome (e.g., the accurate dose distribution for a given patient anatomy and treatment plan) served as the reference.
8. The sample size for the training set:
- This device is not an AI/ML model that undergoes a "training" phase in the typical sense (i.e., learning from annotated data). It's a software system built on established algorithms for radiation treatment planning. Therefore, there is no "training set" as would be applicable to a deep learning model.
9. How the ground truth for the training set was established:
- Refer to point 8. Not applicable for this type of device.
§ 892.5050 Medical charged-particle radiation therapy system.
(a)
Identification. A medical charged-particle radiation therapy system is a device that produces by acceleration high energy charged particles (e.g., electrons and protons) intended for use in radiation therapy. This generic type of device may include signal analysis and display equipment, patient and equipment supports, treatment planning computer programs, component parts, and accessories.(b)
Classification. Class II. When intended for use as a quality control system, the film dosimetry system (film scanning system) included as an accessory to the device described in paragraph (a) of this section, is exempt from the premarket notification procedures in subpart E of part 807 of this chapter subject to the limitations in § 892.9.