(287 days)
The APAS Compact is an in vitro diagnostic system comprised of an instrument for automated imaging of agar culture plates and a software analysis module for the following use:
The APAS Compact, when using its urine analysis module, automates urine culture plate imaging and interpretation to detect the presence of microbial growth on sheep blood and MacConkey agar culture plates that are inoculated with a 1uL sample volume. The APAS Compact, when using its urine analysis module, provides a semiquantitative assessment of colony counts that are used as an aid in the diagnosis of urinary tract infection. All urine culture plates that are identified as positive for growth by the APAS Compact, when using its urine analysis module, must be reviewed by a trained microbiologist.
The APAS Compact with Urine Analysis Module is an instrumented system that is designed for screening of urine culture plates for the presence of microbial growth. The device comprises an imaging station for capture of digital images of the culture plates, together with software for analysis of the images, the determination and enumeration of microbial growth and reporting of results.
This document outlines the acceptance criteria and study findings for the APAS Compact with Urine Analysis Module.
1. Acceptance Criteria and Reported Device Performance
| Criteria | Acceptance Criteria | Reported Device Performance (Overall) |
|---|---|---|
| Detection of Growth on Blood Agar (Clinical Study) | False negative results for growth detection should be acceptably low. | 99.0% (95% CI: 98.7-99.2%) correct designation of growth. False negative rate for growth detection ranged from 0% to 2.9% depending on colony count. |
| Detection of Growth on MacConkey Agar (Clinical Study) | False negative results for growth detection should be acceptably low. | 99.5% (95% CI: 99.2-99.7%) correct designation of growth. False negative rate for growth detection ranged from 0% to 1.3% depending on colony count. |
| Colony Count Accuracy (Analytical Study) | Low counts obtained by APAS Compact should be within 1-log10 of manual reference. APAS Compact should not incorrectly designate any cultures with growth as "Negative". | All low counts were within 1-log10 of manual reference. APAS Compact did not incorrectly designated any of the cultures with growth as "Negative". Instances where APAS Compact designated growth while reference method reported no growth were acceptable as these require microbiologist review. |
| Reproducibility of Colony Counts | Acceptable reproducibility of colony counts between instruments and rotations. | Demonstrated acceptable reproducibility (detailed in Tables 5 and 6, with %CVs generally low, though some higher %CVs for very low counts or specific dilutions). |
| Analytical Specificity (Expected Colony Morphology - Pure Cultures) | High agreement for detection of expected colony morphology. | Blood Agar: Generally 100% agreement, with exceptions for Aerococcus urinae (98.1%), Lactobacillus rhamnosus (22.2%), Pseudomonas aeruginosa (98.0%), and Staphylococcus epidermidis (70.4%). All classifications were "Positive" or "Review" for all but A. urinae and L. rhamnosus. |
| MacConkey Agar: Generally 100% agreement, with Serratia marcescens at 77.8% (but 100% at 20 & 22 hrs). All classifications were "Positive". | ||
| Analytical Specificity (Expected Colony Morphology - Mixed Cultures) | High agreement for detection of both expected colony types. | Blood Agar (1:1 ratio): 100% for all combinations. |
| Blood Agar (1:10 ratio): 100% for E. coli combinations, but S. agalactiae in presence of E. faecalis was 44.4%. All classifications were "Positive". | ||
| MacConkey Agar: Generally 100% agreement, with one image for E. coli/M. morganii (1:10 ratio) at 98.8%. All classifications were "Positive". | ||
| Daily Quality Control Performance | Expected results for positive controls; acceptable rate for negative controls. | 100% of positive controls yielded expected results on both blood and MacConkey agar. 98.4% of negative controls (blood agar) and 100% (MacConkey agar) yielded expected results. Failed negative controls were due to single contaminating colonies or artifacts. |
| Detection Limit (LOD) | Demonstrated limits of colony size for reliable detection. | Provided specific LODs in mm (95% CI) for various organisms on blood and MacConkey agar (Tables 10 and 11). |
2. Sample Sizes and Data Provenance
-
Test Set (Clinical Study):
- Total Enrolled Samples: 10,100 urine samples (5835 from Site 1, 2117 from Site 2, 2148 from Site 3).
- Included in Performance Analysis: 9,224 samples (5634 from Site 1, 1769 from Site 2, 1821 from Site 3).
- Data Provenance: Three clinical sites: one in the US (Site 1) and two ex-US (Sites 2 and 3). The study used remnant urine samples from standard of care culture. This indicates a retrospective data collection approach for the clinical study.
-
Training Set: The document does not explicitly state the sample size for the training set.
3. Number of Experts and Qualifications for Ground Truth (Test Set)
- Number of Experts: Three microbiologists.
- Qualifications: "Trained microbiologist" is mentioned multiple times. For the clinical study, each microbiologist was "trained to read the urine culture plates in a standard fashion." Specific years of experience are not provided.
4. Adjudication Method (Test Set)
- The document states that for the reference method in the clinical study, "each microbiologist... was blinded to the results from the other panel members and to those obtained by the APAS Compact."
- The ground truth for the clinical study was established by a "reference microbiologist panel," meaning the consensus among these three microbiologists. The exact method of achieving consensus (e.g., 2+1, 3+1, none) is not explicitly detailed, but it implies a shared understanding to form the "reference method."
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
- No, a MRMC comparative effectiveness study was not done. The clinical study compared the APAS Compact's performance against a panel of microbiologists (serving as the reference standard), not against human readers with and without AI assistance to measure improvement. The primary goal of the APAS Compact is to automate initial screening to reduce microbiologist workload, rather than directly augment human reading for every plate.
6. Standalone Performance Study
- Yes, a standalone performance study was done. The entire clinical study (Section L.3) and analytical performance studies (Section L.1) evaluate the APAS Compact's performance as an algorithm-only device (without human-in-the-loop during the initial assessment and designation phase). The performance tables (e.g., Tables 19-24) directly report the APAS Compact's designations and colony counts compared to the reference standard. While the device's indications for use emphasize that growth-positive plates "must be reviewed by a trained microbiologist," the reported performance metrics are for the algorithm's initial assessment before this mandatory human review.
7. Type of Ground Truth Used
- Clinical Study: Expert Consensus (Microbiologist Panel): The ground truth for the clinical study was established by "an independent panel of three microbiologists," who read the urine culture plates in a "standard fashion" and were blinded to other results. Their combined assessment served as the reference method.
- Analytical Study (Linearity/Assay Reportable Range): Expert Consensus (Manual Colony Counts): The mean of two independent manual colony counts performed by two microbiologists was used as the reference result for each plate.
8. Sample Size for the Training Set
- The document does not explicitly state the sample size for the training set. It describes the "APAS Controller Software" and "Urine Analysis Module Software" functions, implying that these algorithms were developed and likely trained, but the specific training data volume is not provided in this excerpt.
9. How Ground Truth for the Training Set Was Established
- The document does not explicitly describe how the ground truth for the training set was established. While it details the methods for establishing ground truth in the analytical and clinical validation studies (using expert microbiologist consensus), it does not provide information specific to the data used for training the algorithms.
§ 866.2190 Automated image assessment system for microbial colonies on solid culture media.
(a)
Identification. An automated image assessment system for microbial colonies on solid culture media is a system that is intended to assess the presence or absence of microbial colonies on solid microbiological culture medium, and to interpret their number, and phenotypic and morphologic characteristics through analysis of two dimensional digital images as an aid in diagnosis of infectious disease.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Premarket notification submissions must include a detailed description of the device, including the technology employed, components and software modules, as well as a detailed explanation of the result algorithms and any expert rules that are used to assess colony characteristics and enumerate colonies from image capture through end result.
(2) Premarket notification submissions must include detailed documentation of the analytical studies performed to characterize device performance to support the intended use, as appropriate.
(3) Premarket notification submissions must include detailed documentation from clinical studies performed on a population that is consistent with the intended use population.
(i) The clinical studies must establish the device performance based on comparison to results obtained by an acceptable reference method, as appropriate.
(ii) The clinical study documentation must include the study protocol with a predefined statistical analysis plan and the final report documenting support for the Indications for Use and the results of the statistical analysis, as appropriate.
(4) Premarket notification submissions must include detailed documentation for device software, including but not limited to software applications and hardware based components that incorporate software, and any decision-making thresholds used to generate results for the device. If a part of a Total Laboratory Automation System, the premarket notification submission must include detailed documentation addressing the instrument and software system integration.
(5) Premarket notification submissions must include detailed documentation of appropriate instructions for use regarding the intended user's device quality control procedures for the instrument system and components, as appropriate.
(6) The 21 CFR 809.10 compliant device labeling must include:
(i) Detailed user instructions to mitigate the risk of failure to operate the instrument correctly.
(ii) A detailed explanation of the interpretation of results and limitations regarding the need for review of culture plates by a qualified microbiologist, as appropriate.
(iii) A summary of performance data obtained from the analytical studies used to support device performance, as appropriate.
(iv) A summary of performance data obtained from clinical studies performed on a population that is consistent with the intended use population, as appropriate.
(7) Under 21 CFR 820.30 compliant design control, device manufacturers must, as appropriate:
(i) Conduct human factors/usability validation testing with the final version of the labeling and related materials to adequately mitigate the risk of failure to operate the instrument correctly.
(ii) Document a device training program that will be offered to the end user to adequately mitigate the risk of failure to operate the instrument correctly.