(181 days)
ROMA Calculation Tool Using Elecsys Assays (RCTUEA) is a qualitative test for serum and plasma (K2-EDTA, K3-EDTA and Li-Heparin) that combines the results of the Elecsys HE4 assay, Elecsys CA 125 II assay and menopausal status into a numerical score.
RCTUEA is intended to aid in assessing whether a premenopausal woman who presents with an ovarian adnexal mass is at high or low likelihood of finding malignancy on surgery. RCTUEA is indicated for women who meet the following criteria: over age 18; ovarian adnexal mass present for which surgery is planned, and not yet referred to an oncologist. RCTUEA must be interpreted in conjunction with an independent clinical and radiological assessment. The test is not intended as a screening or stand-alone diagnostic assay.
The electrochemiluminescence immunoassay "ECLIA" is intended for use on Elecsys and cobas e immunoassay analyzers.
PRECAUTION: RCTUEA should not be used without an independent clinical/ radiological evaluation and is NOT intended to be a screening test or to determine whether a patient should proceed to surgery. Incorrect use of RCTUEA carries the risk of unnecessary testing, surgery and/or delayed diagnosis.
ROMA Calculation Tool Using Elecsys Assays (RCTUEA) is a qualitative test for serum and plasma (K2-EDTA, K3-EDTA and Li-Heparin) that combines the results of the Elecsys HE4 assay, Elecsys CA 125 II assay and menopausal status into a numerical score. ROMA was developed using separate logistic regression equations for premenopausal and postmenopausal women:
Pre menopausal:
Predictive Index (PI) = - 12.0 + 2.38 x LN[HE4] + 0.0626 x LN[CA 125] Post menopausal:
Predictive Index (PI) = - 8.09 + 1.04 x LN[HE4] + 0.732 x LN[CA 125] RCTUEA value = exp (PI) / [1 + exp(PI)b)] x 10
RCTUEA is used to stratify women into likelihood groups for finding cancer on surgery. In order to provide a specificity level of 75 %, a cutoff point of ≥ 1.14 was used for premenopausal women and ≥ 2.99 was used for postmenopausal women who present with an ovarian adnexal mass. Women with RCTUEA results above these cutoff points are at high likelihood of finding malignancy on surgery.
The immunoassays used in RCTUEA are:
Elecsys HE4: an electrochemiluminescence immunoassay for the quantitative determination of HE4 in human serum and plasma
Elecsys CA 125 II: an electrochemiluminescence immunoassay for the quantitative determination of OC 125 reactive determinants in human serum and plasma.
Here's a breakdown of the acceptance criteria and study information for the ROMA Calculation Tool Using Elecsys Assays (RCTUEA), based on the provided document:
1. Table of Acceptance Criteria and Reported Device Performance
The acceptance criteria are implied by the performance goals for sensitivity, specificity, and predictive values. The study reports these values for different cohorts. The primary focus for the device's utility is its ability to stratify women into high or low likelihood groups for finding malignancy on surgery, especially its adjunctive use with Initial Cancer Risk Assessment (ICRA).
| Metric | Premenopausal (EOC Only) (Target/Achieved) | Postmenopausal (EOC Only) (Target/Achieved) | All Cancers & LMP Tumors (Adjunctive Use) (Target/Achieved) | Notes |
|---|---|---|---|---|
| Sensitivity | 100% (9/9) (66.4% - 100% CI) | 89.5% (34/38) (75.2% - 97.1% CI) | Increased from 76.9% to 90.8% (81.0% - 96.5% CI) | The document states RTCUEA was used to "aid in assessing whether a premenopausal woman...is at high or low likelihood of finding malignancy... In order to provide a specificity level of 75%". This implicitly sets the specificity as the primary acceptance criterion, with other metrics assessed relative to this. For standalone RCTUEA, the specificity was targeted at 75%. For adjunctive use, the goal appeared to be demonstrating a statistically significant increase in NPV and an increase in sensitivity, even if accompanied by a decrease in specificity and PPV. The significant increase in NPV (P=0.0000) for adjunctive use is a key finding supporting its effectiveness in ruling out cancer. |
| Specificity | 77.6% (177/228) (71.7% - 82.9% CI) | 82.5% (118/143) (75.3% - 88.4% CI) | Decreased from 84.4% to 70.4% (65.4% - 75.0% CI) | |
| Positive Predictive Value (PPV) | 15.0% (9/60) (7.1% - 26.6% CI) | 57.6% (34/59) (44.1% - 70.4% CI) | Decreased from 46.3% to 34.9% (27.8% - 42.6% CI) | |
| Negative Predictive Value (NPV) | 100.0% (177/177) (97.9% - 100% CI) | 96.7% (118/122) (91.8% - 99.1% CI) | Increased from 95.4% to 97.8% (95.2% - 99.2% CI) | |
| TP-FP | 77.6% (72.1% - 83.2% CI) | 72.0% (60.2% - 83.8% CI) | 61.1% (52.5% - 69.7% CI) |
2. Sample Size and Data Provenance
- Test Set Sample Size:
- Total Evaluated: 455 women
- Premenopausal (Primary EOC analysis): 237
- Postmenopausal (Primary EOC analysis): 181
- Adjunctive Use Analysis: 436 (65 Malignant, 371 No Malignancy by Pathology)
- Data Provenance: Prospective, multi-center clinical trial. The document does not specify the countries of origin for the data, but it mentions samples were tested at "three US testing sites," which implies at least some data is from the US. The "multi-center" nature suggests other locations as well, but these are not named.
3. Number of Experts and Qualifications for Ground Truth
- The document does not specify the number of experts used to establish the ground truth for the test set.
- Qualifications of Experts: The ground truth was based on "histopathology reports collected after surgery." This implies pathologists are the experts, but their specific qualifications (e.g., years of experience, board certification) are not provided. An "Initial Cancer Risk Assessment (ICRA)" was completed by a "nurse practitioner, physician assistant or a non-gynecological oncologist," but this was a baseline assessment, not the definitive ground truth.
4. Adjudication Method for the Test Set
- The document does not describe a specific adjudication method (e.g., 2+1, 3+1, none) for resolving discrepancies in the ground truth. It simply states that "histopathology reports were collected after surgery," implying the single report served as the definitive ground truth.
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
- The document describes a study comparing the performance of the RCTUEA alone versus the performance of an Initial Cancer Risk Assessment (ICRA) alone, and then versus the adjunctive use of RCTUEA with ICRA.
- Effect Size of Human Improvement with AI vs. without AI assistance:
- This is not a traditional MRMC study where human readers interpret cases with and without AI assistance to measure reader improvement. Instead, it compares a clinical assessment (ICRA, performed by clinicians) to the device's output, and then the combination.
- For "all malignancies including EOC, non-epithelial ovarian cancer, other gynecologic and non-gynecologic cancers," when RCTUEA was used adjunctively with ICRA:
- Sensitivity increased from 76.9% (ICRA alone) to 90.8% (ICRA + RCTUEA). This represents an increase of 13.9 percentage points.
- Specificity decreased from 84.4% (ICRA alone) to 70.4% (ICRA + RCTUEA). This represents a decrease of 14 percentage points.
- NPV increased from 95.4% (ICRA alone) to 97.8% (ICRA + RCTUEA), which was statistically significant (P=0.0000). This indicates an improvement in the ability to rule out cancer.
6. Standalone Performance Study (Algorithm Only)
- Yes, a standalone performance study was done. The performance metrics for RCTUEA alone (without ICRA) are reported for both premenopausal and postmenopausal women with Epithelial Ovarian Cancer (EOC) only.
- Premenopausal (EOC): Sensitivity 100%, Specificity 77.6%
- Postmenopausal (EOC): Sensitivity 89.5%, Specificity 82.5%
- Performance of RCTUEA for "all malignancies and LMP tumors" is also shown:
- Sensitivity: 86.2%
- Specificity: 79.5%
7. Type of Ground Truth Used
- Pathology: The primary and definitive ground truth used was "histopathology reports collected after surgery."
8. Sample Size for the Training Set
- The document does not explicitly state the sample size used for the training set. It describes the RCTUEA calculation (logistic regression equations) as "developed using separate logistic regression equations," but does not provide details about the data used for this development phase. The reported clinical trial is for effectiveness determination (a test set).
9. How the Ground Truth for the Training Set Was Established
- The document states that the ROMA algorithm was "developed using separate logistic regression equations." However, it does not provide information on how the ground truth for the training data used to develop these equations was established. It is reasonable to assume it would also have been based on histopathology, but this is not explicitly stated for the training phase.
§ 866.6050 Ovarian adnexal mass assessment score test system.
(a)
Identification. An ovarian/adnexal mass assessment test system is a device that measures one or more proteins in serum or plasma. It yields a single result for the likelihood that an adnexal pelvic mass in a woman, for whom surgery is planned, is malignant. The test is for adjunctive use, in the context of a negative primary clinical and radiological evaluation, to augment the identification of patients whose gynecologic surgery requires oncology expertise and resources.(b)
Classification. Class II (special controls). The special control for this device is FDA's guidance document entitled “Class II Special Controls Guidance Document: Ovarian Adnexal Mass Assessment Score Test System.” For the availability of this guidance document,see § 866.1(e).(c)
Black box warning. Under section 520(e) of the Federal Food, Drug, and Cosmetic Act these devices are subject to the following restriction: A warning statement must be placed in a black box and must appear in all advertising, labeling, and promotional material for these devices. That warning statement must read: