(281 days)
The QCA device is intended to detect and classify cells of clinical interest based on recognition of cellular areas of particular color and chromatic intensity. In this software application, the QCA device is intended to measure and quantitate the percentage and intensity of positively stained nuclei in formalin-fixed, paraffin-embedded tissue specimens immunohistochemically stained for estrogen receptors.
It is indicated for use as an aid in the management, prognosis and prediction of therapy outcomes of breast cancer when used with reagents validated for those indications.
The QCA system is an adjunctive computer-assisted methodology to assist the reproducibility of a qualified pathologist in the acquisition and measurement of images from microscopic slides of breast cancer specimens stained for the presence of estrogen (ER) nuclear receptor protein. The accuracy of the test result depends upon the quality of immunohistochemical staining. It is the responsibility of a qualified pathologist to employ appropriate morphological studies and controls to assure the validity of the QCA ER scores.
QCA is a standalone, automated intelligent cell assessment software device that analyzes digital images of cells of interest by pixel color attributes and pixel area detection algorithms. The software system utilizes a pathologist's own personal computer, light microscope, digital camera, printer, and Internet connection.
The Cell Analysis QCA (Quantitative Cellular Assessment) system is a videomicroscopy software system designed for quantitative estrogen receptor immunohistochemistry. It aims to provide objective measurements of estrogen receptor nuclear antigens in breast cancer specimens, addressing the subjectivity and inter-observer variability of manual assessment.
Here's an analysis of its acceptance criteria and the study proving its performance:
1. Table of Acceptance Criteria and Reported Device Performance
The acceptance criteria are not explicitly stated in a quantitative manner (e.g., "R-value must be > 0.90"). Instead, the study aims to show "excellent" correlations and "substantial equivalence" to manual assessment and a predicate device. The performance is reported through regression analysis results.
| Metric | Acceptance Criteria (Implied) | Reported Device Performance (vs. Manual) | Reported Device Performance (vs. Predicate) |
|---|---|---|---|
| ER % Positivity Correlation (QCA vs. Manual) | Excellent correlation (regression analysis) | Pathologist #1: R=0.957, Slope=0.92, Intercept=6.18, SE=7.09 | Not applicable |
| Pathologist #2: R=0.934, Slope=0.97, Intercept=0.15, SE=8.81 | |||
| Pathologist #3: R=0.925, Slope=0.92, Intercept=4.62, SE=7.36 | |||
| ER Score Correlation (QCA vs. Manual) | Excellent correlation (regression analysis) | Pathologist #1: R=0.849, Slope=0.98, Intercept=-0.01, SE=0.37 | Not applicable |
| Pathologist #2: R=0.854, Slope=0.92, Intercept=-0.02, SE=0.36 | |||
| Pathologist #3: R=0.880, Slope=0.90, Intercept=-0.01, SE=0.34 | |||
| ER % Positivity Correlation (QCA vs. Predicate Device) | Substantial equivalence (regression analysis) | Not applicable | R=0.897, Slope=0.88, Intercept=21.3, SE=11.8 |
| Qualitative Agreement (QCA vs. Manual) | High agreement at specified cut-offs | At ≥1.0% positivity: 149 True Positives, 0 False Positives, 0 False Negatives, 3 True Negatives. | Not applicable |
| Inter-microscope Variability (QCA vs. Manual) | High correlation across different microscopes | R-values for %Positivity: 0.964 to 0.988. R-values for Score: 0.962 to 0.989. | Not applicable |
| Within-Image Reproducibility | Consistent results across repeated captures | Every result within each set of 10 images was absolutely identical. | Not applicable |
2. Sample Size Used for the Test Set and Data Provenance
- Test Set Description:
- Main Comparative Study (QCA vs. Manual and QCA vs. Predicate): 32 invasive breast carcinoma tissue specimens from archival material at Lake Forest Hospital. For manual and QCA comparisons, 192 images were used (3 representative images from each of the 32 cases, captured by 2 pathologists).
- Qualitative Percent Positivity Comparison Study: An additional 120 cases were added to the initial 32, totaling 152 cases/slides, including both invasive and in-situ breast cancers.
- Inter-microscope Variability Study: 32 breast cancers (four images per case).
- Within-Image Reproducibility Study: 10 different breast cancer cases/slides.
- Data Provenance: Retrospective, derived from archival material at Lake Forest Hospital. The origin is implied to be the USA.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications of Those Experts
- Number of Experts:
- Image Capture: Two pathologists "based on professional judgment acquired through pathology training and experience" captured images.
- Manual Ground Truth: Three "highly trained pathologists" performed independent manual inspection and derived scores.
- Qualifications of Experts: Described as "highly trained pathologists" and having "professional judgment acquired through pathology training and experience". Specific years of experience are not provided.
4. Adjudication Method for the Test Set
- Manual assessment: "Three different pathologists performed independent manual inspection and derived scores for 192 randomly mixed images. Manual inspection results were kept blind." This suggests independent assessment without an explicit adjudication or consensus method described for deriving a single "ground truth" per image from the three pathologists. The analysis then compares QCA results against each individual pathologist's manual results.
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
- A formal MRMC comparative effectiveness study, designed to quantify AI assistance on human reader performance, was not explicitly conducted as a primary outcome.
- The study compares the QCA system's performance against individual pathologists (without AI assistance) and against a predicate device.
- The QCA system is described as an "adjunctive computer-assisted methodology to assist the reproducibility of a qualified pathologist," implying a human-in-the-loop scenario. However, the performance metrics provided are for the standalone QCA against manual/predicate results. There is no reported effect size of how much human readers improve with AI vs. without AI assistance. The QCA evaluation described involves one pathologist using the QCA software "without manually overriding the program's scores," which is more indicative of a standalone performance test when compared to manual.
6. Standalone Performance
- Yes, a standalone performance study was implicitly done. The analysis of QCA "without manually overriding the program's scores" when compared against manual assessments for % positivity and ER scores, and against the predicate device, represents the standalone performance of the algorithm.
7. Type of Ground Truth Used
- Expert Consensus / Expert Observation: The primary ground truth for the device's performance was established through the manual assessment of three "highly trained pathologists." In the context provided, "manual inspection of immunohistochemistry special stains remains the method of choice for this analysis in the vast majority of all pathology laboratories, and must therefore be considered the standard to which other systems should be compared."
8. Sample Size for the Training Set
- The document does not explicitly state a separate training set or its sample size. The description focuses on the validation study using 32 (and later 152) cases for testing. It's possible that the device's algorithms were developed or refined using other data not detailed in this submission, or that development and testing were intertwined in an earlier phase.
9. How the Ground Truth for the Training Set Was Established
- Since a separate training set and its sample size are not explicitly mentioned, the method for establishing its ground truth is also not detailed.
- The "assessment algorithms have been designed to mimic visual observations by highly trained health care professionals," which suggests that expert observations would have been crucial during the development phase, even if not formally presented as a "training set ground truth" in this submission.
§ 864.1860 Immunohistochemistry reagents and kits.
(a)
Identification. Immunohistochemistry test systems (IHC's) are in vitro diagnostic devices consisting of polyclonal or monoclonal antibodies labeled with directions for use and performance claims, which may be packaged with ancillary reagents in kits. Their intended use is to identify, by immunological techniques, antigens in tissues or cytologic specimens. Similar devices intended for use with flow cytometry devices are not considered IHC's.(b)
Classification of immunohistochemistry devices. (1) Class I (general controls). Except as described in paragraphs (b)(2) and (b)(3) of this section, these devices are exempt from the premarket notification requirements in part 807, subpart E of this chapter. This exemption applies to IHC's that provide the pathologist with adjunctive diagnostic information that may be incorporated into the pathologist's report, but that is not ordinarily reported to the clinician as an independent finding. These IHC's are used after the primary diagnosis of tumor (neoplasm) has been made by conventional histopathology using nonimmunologic histochemical stains, such as hematoxylin and eosin. Examples of class I IHC's are differentiation markers that are used as adjunctive tests to subclassify tumors, such as keratin.(2) Class II (special control, guidance document: “FDA Guidance for Submission of Immunohistochemistry Applications to the FDA,” Center for Devices and Radiologic Health, 1998). These IHC's are intended for the detection and/or measurement of certain target analytes in order to provide prognostic or predictive data that are not directly confirmed by routine histopathologic internal and external control specimens. These IHC's provide the pathologist with information that is ordinarily reported as independent diagnostic information to the ordering clinician, and the claims associated with these data are widely accepted and supported by valid scientific evidence. Examples of class II IHC's are those intended for semiquantitative measurement of an analyte, such as hormone receptors in breast cancer.
(3) Class III (premarket approval). IHC's intended for any use not described in paragraphs (b)(1) or (b)(2) of this section.
(c)
Date of PMA or notice of completion of a PDP is required. As of May 28, 1976, an approval under section 515 of the Federal Food, Drug, and Cosmetic Act is required for any device described in paragraph (b)(3) of this section before this device may be commercially distributed. See § 864.3.