Search Results

Contour+ (MVision Al Segmentation) is a software system for image analysis algorithms to be used in radiation therapy treatment planning workflows. The system includes processing tools for automatic contouring of CT and MR images using machine learning based algorithms. The produced segmentation templates for regions of interest must be transferred to appropriate image visualization systems as an initial template for a medical professional to visualize, review, modify and approve prior to further use in clinical workflows.

The system creates initial contours of pre-defined structures of common anatomical sites, i.e., Head and Neck, Brain, Breast, Lung and Abdomen, Male Pelvis, and Female Pelvis.

Contour+ (MVision Al Segmentation) is not intended to detect lesions or tumors. The device is not intended for use with real-time adaptive planning workflows.

Contour+ (MVision Al Segmentation) is a software-only medical device (software system) that can be used to accelerate region of interest (ROI) delineation in radiotherapy treatment planning by automatic contouring of predefined ROIs and the creation of segmentation templates on CT and MR images.

The Contour+ (MVision Al Segmentation) software system is integrated with a customer IT network and configured to receive DICOM CT and MR images, e.g., from a CT or MRI scanner or a treatment planning system (TPS). Automatic contouring of predefined ROIs is performed by pre-trained, locked, and static models that are based on machine learning using deep artificial neural networks. The models have been trained on several anatomical sites, including the brain, head and neck, bones, breast, lung and abdomen, male pelvis, and female pelvis using hundreds of scans from a diverse patient population. The user does not have to provide any contouring atlases. The resulting segmentation structure set is connected to the original DICOM images and can be transferred to an image visualization system (e.g., a TPS) as an initial template for a medical professional to visualize, modify and approve prior to further use in clinical workflows.

The provided text does not include a table of acceptance criteria and the reported device performance, nor does it specify the sample sizes used for the test set, the number of experts for ground truth, or details on comparative effectiveness studies (MRMC).

However, based on the available information, here is a description of the acceptance criteria and study details:

Acceptance Criteria and Study for Contour+ (MVision AI Segmentation)

The study evaluated the performance of automatic segmentation models by comparing them to ground truth segmentations using Dice Score (DSC) and Surface-Dice Score (S-DSC@2mm) as metrics. The acceptance criteria were based on a "set level of minimum agreement against ground truth segmentations determined through clinically relevant similarity metrics DSC and S-DSC@2mm." While specific numerical thresholds for these metrics are not provided, the submission states that the device fulfills "the same acceptance criteria" as the predicate device.

It's important to note that the provided document is an FDA 510(k) clearance letter and not the full study report. As such, it summarizes the findings and affirms the device's substantial equivalence without detailing every specific test result or acceptance threshold.

1. A table of acceptance criteria and the reported device performance

2. Sample sized used for the test set and the data provenance (e.g. country of origin of the data, retrospective or prospective)

• Sample Size for Test Set: The exact sample size for the test (golden) dataset is not specified, but it's referred to as "various subsets of the golden dataset" and chosen to "achieve high granularity in performance evaluation tests."
• Data Provenance: The datasets originate from "multiple EU and US clinical sites (with over 50% of data coming from US sites)." It is described as containing "hundreds of scans from a diverse patient population," ensuring representation of the "US population and medical practice." The text does not explicitly state if the data was retrospective or prospective, but the description of "hundreds of scans" from multiple sites suggests it is likely retrospective.

3. Number of experts used to establish the ground truth for the test set and the qualifications of those experts (e.g. radiologist with 10 years of experience)

The number of experts used to establish the ground truth for the test set is not specified in the provided text. The qualifications are vaguely mentioned as "radiotherapy experts" who performed "Performance validation of machine learning-based algorithms for automatic segmentation." No specific years of experience or board certifications are detailed.

4. Adjudication method (e.g. 2+1, 3+1, none) for the test set

The adjudication method for establishing ground truth on the test set is not specified in the provided text. The text only states that the auto-segmentations were compared to "ground truth segmentations."

5. If a multi reader multi case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance

A multi-reader multi-case (MRMC) comparative effectiveness study focusing on the improvement of human readers with AI assistance versus without AI assistance is not explicitly described in the provided text.

The text states: "Performance validation of machine learning-based algorithms for automatic segmentation was also carried out by radiotherapy experts. The results show that Contour+ (MVision AI Segmentation) assists in reducing the upfront effort and time required for contouring CT and MR images, which can instead be devoted by clinicians on refining and reviewing the software-generated contours." This indicates that experts reviewed the output and perceived a benefit in efficiency, but it does not detail a formal MRMC study comparing accuracy or time, with a specific effect size.

6. If a standalone (i.e. algorithm only without human-in-the-loop performance) was done

Yes, a standalone performance evaluation of the algorithm was conducted. The primary performance metrics (DSC and S-DSC@2mm) were calculated by directly comparing the "produced auto-segmentations to ground truth segmentations," which is a standalone assessment of the algorithm's output. The statement "Performance verification and validation results for various subsets of the golden dataset show the generalizability and robustness of the device" further supports this.

7. The type of ground truth used (expert consensus, pathology, outcomes data, etc)

The ground truth used was expert consensus segmentations. The text repeatedly refers to comparing the device's output to "ground truth segmentations" established by "radiotherapy experts." There is no mention of pathology or outcomes data being used for ground truth.

8. The sample size for the training set

The exact sample size for the training set is not specified, but the models were "trained on several anatomical sites... using hundreds of scans from a diverse patient population."

9. How the ground truth for the training set was established

The text states that the machine learning models were "trained on several anatomical sites... using hundreds of scans from a diverse patient population." While it doesn't explicitly detail the process for establishing ground truth for the training set, it is implied to be through expert contouring/segmentation, as the validation uses "ground truth segmentations" which are established by "radiotherapy experts." Given the extensive training data required for machine learning, it's highly probable that these "hundreds of scans" also had expert-derived segmentations as their ground truth for training.

Ask a specific question about this device

MVision AI Segmentation is a software system for image analysis algorithms to be used in radiation therapy treatment planning workflows. The system includes processing tools for automatic contouring of CT images using machine learning based algorithms. The produced segmentation templates for regions of interest must be transferred to appropriate image visualization systems as an initial template for a medical professional to visualize, review, modify and approve prior to further use in clinical workflows.

The system creates initial contours of pre-defined structures of common anatomical sites, i.e. Head and Neck, Brain, Breast, Lung and Abdomen, Male Pelvis, and Female Pelvis in adult patients.

MVision AI Segmentation is not intended to detect lesions or tumors. The device is not intended for use with real-time adaptive planning workflows.

MVision AI Segmentation is a software only medical device which can be used to accelerate region of interest (ROI) delineation in radiotherapy treatment planning by creating automatic segmentation templates on CT images for these ROIs.

The segmentations are produced by pre-trained, locked, and static models that are based on deep artificial neural networks. The produced structure is intended to be used as a template for medical professionals to visualize, modify and approve prior to further use in clinical workflows.

The system is integrated with the customer IT network to receive DICOM images. CT images from, for example, a scanner or a treatment planning system (TPS) are exported to the device. A structure set is created in the device, and the created segmentation results are connected to the original images. These data are sent to the destination DICOM import folder to import the data to, for example, a treatment planning system. The produced structures can then be used as a template for manual ROI editing, review and approval workflow. The segmentations are produced by pre-trained and locked models that are based on deep artificial neural networks. To take the device into use, the user does not have to provide any contouring atlases. The models have been trained with the order of hundreds of scans, depending on the ROI in question. The MVision AI Segmentation device creates initial contours of pre-defined structures of common anatomical sites, i.e. Head and Neck, Brain, Breast, Lung and Abdomen, Male Pelvis, and Female Pelvis.

Here's a breakdown of the acceptance criteria and study information for the MVision AI Segmentation device, based on the provided text:

1. Table of Acceptance Criteria and Reported Device Performance

The provided text does not explicitly state specific numerical acceptance criteria for evaluation metrics (e.g., a minimum Dice Similarity Coefficient (DSC) or Hausdorff Distance (HD)). Instead, it generally states that the device's performance will "reflect the real clinical performance" and that it produces "usable contours" that "save clinicians' time."

Therefore, I will extract relevant performance statements and structure them as best as possible, acknowledging the lack of specific thresholds.

2. Sample Sizes Used for the Test Set and Data Provenance

• Test Set ("Golden Dataset") Sample Size: The exact number of cases in the test set is not explicitly stated. The document mentions "various subsets of the golden dataset."
• Data Provenance: The data originates from "multiple different sources" to ensure generalizability. It is collected to reflect "the US patient population and US medical practice." The text does not specify countries of origin beyond "US patient population." The data type is implied to be CT images for use in radiotherapy. The text does not specify if the data is retrospective or prospective.

3. Number of Experts Used to Establish the Ground Truth for the Test Set and Their Qualifications

• Number of Experts: The number of experts used to establish ground truth for the test set is not explicitly stated.
• Qualifications of Experts: The ground truth for the test set was established by "radiotherapy experts." No further specific qualifications (e.g., years of experience, specific subspecialty) are provided.

4. Adjudication Method for the Test Set

• The text does not describe a specific adjudication method (e.g., 2+1, 3+1). It only states that the ground truth was established by "radiotherapy experts."

5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study

• No MRMC study comparing human readers with and without AI assistance was reported. The document focuses on the device's performance and its ability to "assist in reducing the upfront effort and time" for clinicians, implying an improvement in efficiency, but not a formal MRMC study demonstrating a quantified effect size of human improvement with AI vs without.

6. Standalone Performance Study (Algorithm Only)

• Yes, a standalone performance evaluation was clearly done. The entire "Performance Evaluation Summary" Section (Pages 7-8) describes the evaluation of the "model performance" and "machine learning based algorithms" on "training and test sets (golden dataset)." The results refer to the device producing contours and assisting in reducing effort, indicating an algorithm-only evaluation.

7. Type of Ground Truth Used

• The ground truth used is expert consensus, established by "radiotherapy experts" following "segmentation consensus guidelines."

8. Sample Size for the Training Set

• The models were trained with "the order of hundreds of scans, depending on the ROI in question."

9. How the Ground Truth for the Training Set Was Established

• The ground truth for the training set was established following "segmentation consensus guidelines" as the models were "trained to comply with" these guidelines. This implies expert-derived ground truth, consistent with the test set's ground truth methodology.

Ask a specific question about this device

AI Segmentation uses CT images to segment patient anatomy for use in radiation therapy treatment planning. AI Segmentation utilizes a pre-defined set of organ structures in the following regions: head and neck, thorax, pelvis, abdomen. Segmentation results are subject to review and editing by qualified, expert radiation therapy treatment planners. Results of AI Segmentation are utilized in the Eclipse Treatment Planning System where it is the responsibility of a qualified physician to further review, edit as needed, and approve each structure.

Al Segmentation is a web-based application, running in the cloud, that provides a combined deep learning and classical-based approach for automated segmentation of organs at risk, along with tools for structure visualization. This software medical device product is used by trained medical professionals and consists of a web application user interface where the results from the automated segmentation can be reviewed, edited, and selected for export into the compatible treatment planning system. Al Segmentation is not intended to provide clinical decisions, medical advice, or evaluations of radiation plans or treatment procedures.

Here's an analysis of the acceptance criteria and study detailed in the provided text:

1. Table of Acceptance Criteria and Reported Device Performance

The text doesn't provide a direct, explicit table of acceptance criteria with corresponding performance metrics for all AI models. Instead, it describes a general approach for evaluating performance, focusing on the DICE similarity index for automated contouring and a qualitative expert assessment.

2. Sample Size Used for the Test Set and Data Provenance

The document does not explicitly state the sample size (number of patients or scans) used for the test set. It mentions "non-clinical performance tests for automated contouring AI models" but lacks the specific number of cases.

• Data Provenance: The document does not specify the country of origin of the data. It indicates that the study was a non-clinical performance evaluation, implying retrospective data was likely used, but this is not explicitly stated.

3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications

The document states: "Clinical experts also evaluated the performance of these AI models during validation testing." However, it does not specify:

• The exact number of experts used.
• The specific qualifications of these experts (e.g., "radiologist with 10 years of experience"). It generally refers to them as "qualified, expert radiation therapy treatment planners" and "qualified physicians" in the Indications for Use, which implies relevant expertise.

4. Adjudication Method for the Test Set

The document mentions that "Each AI model was assessed using the DICE similarity index as a comparative measure of the auto-generated contours against ground truth contours for a given structure." and that "Clinical experts also evaluated the performance of these AI models during validation testing."

However, it does not explicitly detail the adjudication method used for establishing the ground truth or resolving expert discrepancies (e.g., 2+1, 3+1). The primary comparison seems to be against "ground truth contours" rather than against potentially varying expert opinions that would necessitate an adjudication method for the ground truth itself. The qualitative expert assessment seems to be a separate evaluation of the AI output against the target of "acceptable with minor or no adjustments," rather than a process to establish the ground truth.

5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study

• No MRMC study was done. The document explicitly states: "No animal studies or clinical tests have been included in this pre-market submission." and "The predicate device was cleared based only on non-clinical testing, and no animal or clinical studies were performed for the subject device."
• Therefore, there is no reported effect size of how much human readers improve with AI vs. without AI assistance. The device is intended to be reviewed and edited by human experts, but a comparative study on this effect was not performed for this submission.

6. Standalone (Algorithm Only Without Human-in-the-Loop Performance) Study

• Yes, a standalone performance study was done. The performance evaluation focused on the AI models' ability to generate contours, as measured by the DICE similarity index against ground truth and qualitative expert assessment of the auto-generated contours. This indicates a standalone assessment of the algorithm's output before human editing.

7. Type of Ground Truth Used

The ground truth used was expert consensus / expert-generated contours. The text states:

• "Each AI model was assessed using the DICE similarity index as a comparative measure of the auto-generated contours against ground truth contours for a given structure."
• The Indications for Use also mention that "Segmentation results are subject to review and editing by qualified, expert radiation therapy treatment planners" and "it is the responsibility of a qualified physician to further review, edit as needed, and approve each structure," which implies that the ground truth would be established by such experts.

8. Sample Size for the Training Set

The document does not specify the sample size used for the training set of the AI models.

9. How the Ground Truth for the Training Set Was Established

The document does not specify how the ground truth for the training set was established. It describes the evaluation of the AI models but not their development. However, given it's an AI segmentation tool for radiation therapy, it's highly probable that the training data ground truth was also established through expert contouring.

Ask a specific question about this device

AI Segmentation uses CT images to segment patient anatomy for use in radiation therapy treatment planning. AI Segmentation utilizes a pre-defined set of organ structures in the following regions: head and neck, thorax, pelvis, abdomen. Segmentation results are subject to review and editing by qualified, expert radiation therapy treatment planners. Results of AI Segmentation are utilized in the Eclipse Treatment Planning System where it is the responsibility of a qualified physician to further review, edit as needed, and approve each structure.

AI segmentation is a web-based application, running in the cloud, that provides a combined deep learning and classical-based approach for automated segmentation of organs at risk, along with tools for structure visualization. This software medical device product is used by trained medical professionals and consists of a web application user interface where the results from the automated segmentation can be reviewed and selected for export into the compatible treatment planning system. AI Segmentation is not intended to provide clinical decisions, medical advice, or evaluations of radiation plans or treatment procedures.

The provided text describes that the AI Segmentation device does not include clinical data in its premarket submission. The document explicitly states: "No animal studies or clinical tests have been included in this pre-market submission." Therefore, it is not possible to provide acceptance criteria or a study proving the device meets the acceptance criteria using the requested information (e.g., sample size for the test set, number of experts, adjudication method, MRMC study, standalone performance, type of ground truth for test and training sets, and training set size).

Instead, the submission for AI Segmentation focused on non-clinical data, specifically software verification and validation testing, and conformance to relevant standards.

Here is what can be extracted from the document regarding the non-clinical evaluation:

1. Table of Acceptance Criteria and Reported Device Performance:

Since no clinical study data is available, a table of acceptance criteria and reported device performance in terms of clinical accuracy (e.g., Dice score, sensitivity, specificity) cannot be provided. The performance data presented is focused on software quality and adherence to regulatory standards.

2. Sample size used for the test set and the data provenance: Not applicable, as no clinical test set data from patients was submitted. The evaluation was based on software testing.

3. Number of experts used to establish the ground truth for the test set and the qualifications of those experts: Not applicable, as no clinical test set requiring expert ground truth was submitted.

4. Adjudication method (e.g., 2+1, 3+1, none) for the test set: Not applicable, as no clinical test set requiring adjudication was submitted.

5. If a multi reader multi case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance: No MRMC study was done, as explicitly stated that no clinical tests were included.

6. If a standalone (i.e. algorithm only without human-in-the loop performance) was done: The document states, "Segmentation results are subject to review and editing by qualified, expert radiation therapy treatment planners. Results of AI Segmentation are utilized in the Eclipse Treatment Planning System where it is the responsibility of a qualified physician to further review, edit as needed, and approve each structure." This indicates the device is intended for human-in-the-loop use. However, no clinical performance data (standalone or otherwise) was provided. The software verification and validation would have tested the algorithm's output without human intervention, but these were functional tests, not clinical performance studies.

7. The type of ground truth used (expert consensus, pathology, outcomes data, etc.): Not applicable for clinical ground truth, as no clinical studies were submitted. For software verification, ground truth would be against predetermined functional requirements and expected outputs established during software development.

8. The sample size for the training set: Not provided in the document.

9. How the ground truth for the training set was established: Not provided in the document.

Ask a specific question about this device