Search Results

The OSDI HER-2/neu Assay is an in vitro, diagnostic device intended for use in the quantitative determination of serum HER-2/neu in women with metastatic breast cancer who have an initial value of 15 ng/ml or greater. HER-2/neu values obtained may be used in the follow-up and monitoring of patients with metastatic breast cancer. HER-2/neu values should be used in conjunction with information available from clinical and other diagnostic procedures in the management of breast cancer. The clinical utility of the serum measurement of HER-2/neu as a prognostic indicator for early recurrence and in the management of patients on immunotherapy regiments has not been fully established.

Device Description

The HER-2/neu ELISA is a sandwich enzyme immunoassay, which utilizes two monoclonal antibodies to quantitate the extracellular domain (ECD) of the HER-2/neu protein in serum. The capture antibody (NB-3) has been immobilized on the interior surface of the microtiter plate wells. To perform the assay, an appropriate volume of serum is incubated in the coated well to allow binding of the antigen by the capture antibody. The captured HER-2/neu ECD is then reacted with a different anti-HER-2/neu antibody designated TA-1. The TA-1 antibody is biotinylated. The detection of the ECD of HER-2/neu is with a streptavidin horseradish peroxidase conjugate, which then catalyzes the conversion of the chromogenic substrate o-phenylenediamine into a colored product. The colored reaction product is quantitated by spectrophotometry (read absorbance at 490 mm) and is related to the amount of the HER-2/neu ECD in the serum sample. Six prepared HER-2/neu standards (0, 2.5, 7.5, 15, 25 and 35 ng/ml) allow construction of a standard curve for subsequent quantification of HER-2/neu in serum samples. Also available from OSDI is a set of HER-2/neu controls (10-CVX) designed to be used in conjunction with the ELISA for quality monitoring of assay performance. This control set consists of three (3) control levels at 2.9, 9.1 and 24.0 ng/mL HER-2/neu protein.

AI/ML Overview

The Oncogene Science Diagnostics, Inc. (OSDI) Manual HER-2/neu Microtiter ELISA is an in vitro diagnostic assay intended for the quantitative measurement of the extracellular domain of HER-2/neu protein in human serum. It is used in the follow-up and monitoring of patients with metastatic breast cancer whose initial serum level of HER-2/neu is 15 ng/mL or greater, in conjunction with other clinical and diagnostic procedures.

Here's an analysis of the acceptance criteria and the study proving the device meets those criteria:

1. Table of Acceptance Criteria and Reported Device Performance

The acceptance criteria are not explicitly stated in a quantitative table within the document but can be inferred from the performance metrics described for various aspects of the assay. The "reported device performance" are the results from the non-clinical and clinical studies.

Acceptance Criteria (Inferred)	Reported Device Performance
Specificity: Interference	"None of the potential endogenous or exogenous interferents demonstrated any significant interfering effects on HER-2/neu recovery." (Page 6)
Specificity: Cross-Reactivity	"The maximum effect seen with [Human Epidermal Growth Factor Receptor (HER-1)] cross-reactant was not significant (≤1%)... no cross-reactivity with HER-3." (Page 7)
Specificity: Heterophilic Antibodies and HAMA Interference	"HAMA and rheumatoid factors do not cause false negative or false positive results in the present assay format" when detector diluent contained Normal Mouse Serum (NMS). (Page 7)
Linearity	"When observed HER-2/neu concentration was plotted versus expected concentration, the slope and correlation coefficient of each data set were close to the optimum of 1, indicating linearity within the dynamic range of the assay." (Page 7)
Spike and Recovery (Matrix Effect)	"The total average percent recovery of all spiked serum samples in all three kit lots was 103%, indicating no effect of matrix on the ability of the ELISA to accurately measure HER-2/neu protein in serum." (Page 8)
Hook Effect (Antigen Excess)	"Separation of sample and reagent addition by washes prevents high-dose hook effect in this device." (Page 8)
Parallelism (Dilution Studies)	"Recoveries were evaluated at each of the dilution points for mean and standard deviation and were comparable for all three ELISA kit lots. Serum samples diluted in kit Sample Diluent result in accurate recovery of analyte at a range of concentrations in human samples." (Page 8)
End-to-End Variation (Robustness to timing)	"The expected variance in timing during normal use by skilled individuals should fall well short of these extended times tested and recoveries should be unaffected." (Page 9 - small increase/decrease (15%) observed only when times extended well beyond normal usage)
Plate Coating Variations	"Serum sample recovery, and therefore antibody coating, is consistent throughout plates and kit lots with 4.7-7.6 % CV." (Page 9)
Reproducibility (Intra- and Inter-assay)	OSDI Site, All Microtiter ELISA Lots Combined (Table 4.3-1): - Control 1 (24.2 ng/mL): Within Run %CV 3.8, Total %CV 7.8 - Control 2 (9.5 ng/mL): Within Run %CV 4.7, Total %CV 9.1 - Control 3 (2.9 ng/mL): Within Run %CV 7.8, Total %CV 10.6 - Control 4 (9.2 ng/mL): Within Run %CV 4.5, Total %CV 8.3 Three Sites and Three Kit Lots Combined (Table 4.3-2): - Control 1 (24.5 ng/mL): Within Run %CV 6.0, Total %CV 10.7 - Control 2 (9.8 ng/mL): Within Run %CV 7.0, Total %CV 10.4 - Control 3 (3.3 ng/mL): Within Run %CV 10.2, Total %CV 17.7 - Control 4 (9.5 ng/mL): Within Run %CV 6.9, Total %CV 10.8 "Within run CV's of 6-7% and total CV's of 10-11% for clinically relevant controls... well within acceptable limits for an assay of this type and for its intended use." (Page 9)
Sensitivity (Detection Limit)	"An MDD of 1.5 ng/mL was observed when assaying 232 replicates of Sample Diluent alone in three HER-2/neu ELISA kit lots. This MDD is acceptable for the intended use of this assay, where the lowest normal patient was 4.23 ng/ml and the cutoff determined in this study for elevated HER-2/neu values was 1.5 ng/ml." (Page 10)
Clinical Utility (Correspondence between HER-2/neu changes and clinical course)	Table 4.4-1: Correspondence of Metastatic Breast Cancer Patient Disease Status with Changes in Serum HER-2/neu: Visit-by-Visit - Change in HER-2/neu ≥20% ↑: 52 cases of Progression, 33 cases of No Progression (Total 85) - Change in HER-2/neu <20% ↑: 55 cases of Progression, 96 cases of No Progression (Total 151) Overall: "Data collected from this study show that changes in serum HER-2/neu concentrations over time in initially elevated metastatic breast cancer patients reflect changes in clinical status such as response to therapy or progression of disease." (Page 13-14)
Clinical Sensitivity and Specificity for Elevated HER-2/neu Cutoff	"The Upper Limit of Normals in this study with the ELISA was 15 ng/ml, which was confirmed as an appropriate cutoff by ROC analysis to give 95% specificity." (Page 12)

2. Sample size used for the test set and the data provenance

Test Set Sample Size:
- Non-clinical studies: The exact number of samples for each non-clinical test (interference, cross-reactivity, linearity, spike and recovery, hook effect, parallelism, end-to-end variation, plate coating variations, reproducibility, sensitivity) is not always explicitly stated as a single total number. For example:
  - Reproducibility: 130-131 runs and 385-390 replicates combined across three sites and three kit lots for 4 controls (Table 4.3-2).
  - Sensitivity (MDD): 232 replicates of Sample Diluent alone.
  - Interference: Not specified, but involved numerous compounds tested.
  - Linearity: Two controls combined to produce five samples.
- Clinical studies:
  - Longitudinal monitoring: 56 clinically evaluable metastatic breast cancer patients with initial elevated HER-2/neu (≥15 ng/mL). There were 236 visit-by-visit comparisons (Table 4.4-1).
  - Upper Limit of Normals determination: 38 healthy women (for longitudinal variability of 20% criterion).
  - Clinical sensitivity/specificity to establish 15 ng/ml cutoff: "patients with breast cancer," "patients with benign breast diseases," "other non-malignant diseases," and "normal healthy individuals." Specific numbers for each group are not provided for this specific analysis.
Data Provenance: Retrospective (Page 10, 12).
Country of Origin: The clinical evaluations were conducted at "two US clinical trial sites" (Page 2, 3), and studies were performed at OSDI, Cambridge, Massachusetts (Page 3).

3. Number of experts used to establish the ground truth for the test set and the qualifications of those experts

Number of Experts: Not explicitly stated as a specific number.
Qualifications of Experts: For the clinical studies, the "Clinical course or status was determined by the physician" (Page 10). It is implied these are qualified medical professionals, but specific qualifications (e.g., "radiologist with 10 years of experience") are not detailed.

4. Adjudication method for the test set

The document implies that "Clinical course or status was determined by the physician" (Page 10). This suggests individual physician assessment rather than a detailed, multi-reader, adjudicated ground truth process for each patient case, at least as explicitly described for the clinical status.
For the visit-by-visit analysis (Table 4.4-1), changes greater than or equal to 20% in HER-2/neu values were considered reflective of disease progression, while less than 20% increase reflected lack of progression. This 20% criterion was internally derived from the longitudinal variability in normal patients (six serial samples from 38 healthy women), suggesting an objective, pre-defined threshold rather than expert adjudication for each change.

5. If a multi reader multi case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance

No, a multi-reader multi-case (MRMC) comparative effectiveness study was not done. This device is an immunoassay (ELISA), not an AI-powered image analysis tool that would typically involve human readers.

6. If a standalone (i.e. algorithm only without human-in-the-loop performance) was done

Yes, the device was evaluated in a standalone manner. The ELISA assay itself provides quantitative values for HER-2/neu. The non-clinical studies (e.g., interference, linearity, reproducibility, sensitivity) directly assess the performance of the assay independent of human interpretation or intervention beyond performing the laboratory procedure correctly. The clinical studies then correlated the values generated by the ELISA with the clinical course of the disease, but the device itself is a measurement tool, not an interpretive algorithm that a human would then refine.

7. The type of ground truth used

Non-clinical studies:
- Defined Antigen: The standard antigen (p105) itself is a characterization of the "ground truth" for what the assay is supposed to detect (Page 3).
- Spiked Samples/Known Concentrations: For analytical performance like linearity, spike and recovery, and hook effect, samples with known, defined concentrations of HER-2/neu were used as ground truth.
- Clinical Studies:
  - Clinical Course/Status Determined by Physician: For the metastatic patient serial monitoring, the ground truth for disease progression or lack thereof (stable/responding disease) was the "clinical course or status determined by the physician" (Page 10). This would be based on a combination of medical assessments and other diagnostic procedures.
  - Retrospective Sample Storage: The ground truth relied on patient samples collected and stored in specimen banks, with the assumption that HER-2/neu concentration remained stable during storage (Page 10).
  - Statistical Threshold: The 20% change criterion for correlating HER-2/neu values with clinical status was established based on longitudinal variability in normal patients, acting as a statistical ground truth for what constitutes a "significant" change (Page 12).
  - ROC Analysis: Used to confirm 15 ng/ml as an appropriate cutoff for 95% specificity, likely against a diagnosed patient population as ground truth (Page 12).

8. The sample size for the training set

The document does not explicitly describe a separate "training set" in the context of typical machine learning models. Instead, the non-clinical studies (which involve extensive characterization of the assay's performance) and the selection of assay parameters (like antibody choice or standard curve concentrations) can be thought of as the "development" phase that would precede a formal clinical validation. The standard development of an ELISA kit does not typically involve a "training set" in the same way an AI/ML model does.

9. How the ground truth for the training set was established

As a traditional immunoassay, there isn't a "training set" in the machine learning sense. The assay's "ground truth" during its development would have been established through:
- Characterization of the Antigen and Antibodies: (Pages 3-4) Understanding the specific extracellular domain of HER-2/neu (p105) and developing highly specific monoclonal antibodies (NB-3 and TA-1) that bind to it. This involves biological and biochemical validation.
- Standard Curve Development: Using prepared HER-2/neu standards of known concentrations (0, 2.5, 7.5, 15, 25, and 35 ng/ml) to establish accurate quantification (Page 2).
- Analytical Validation: Extensive non-clinical studies to demonstrate performance characteristics (specificity, linearity, sensitivity, reproducibility) using samples with known properties (e.g., spiked samples, controls derived from cell lines or serum pools). (Pages 3-9)

Ask a Question

Ask a specific question about this device

Page 1 of 1