INTENDED USE: The BBLCRYSTAL™ Gram Positive (GP) Identification (ID) System is a miniaturized identification method employing modified conventional, fluorogenic, and chromogenic substrates. It is intended for the identification of frequently isolated aerobic gram-positive bacteria from clinical specimens.

INDICATIONS FOR USE: Use of the BBLCRYSTAL™ Gram Positive Identification System is indicated when the aerobic gram-positive organisms described in the attached table have been isolated in pure culture from clinical specimens in a clinical laboratory, and identification of the microorganisms is desired.

Device Description

The main component of the BBLCRYSTAL™ GP ID System is the BBLCRYSTAL GP panel assembly, consisting of the CRYSTAL base and lid. The BBLCRYSTAL lid consists of 29 dehydrated biochemical/chromogenic/ fluorogenic substrates and one fluorogenic negative control, on the ends of plastic prongs. The CRYSTAL base consists of 30 matching wells; its design allows inoculation of all 30 wells in a single step by pouring the suspension of pure culture into the target area and tilting the base. The test inoculum rehydrates the dried substrates and initiates test reactions.

The pure culture suspension is prepared by picking several small colonies of the same morphology from media such as Trypticase® Soy Agar with 5% Sheep Blood or Columbia Agar with 5% Sheep Blood, or alternatively selective media such as Phenylethyl Alcolol Agar with 5% Sheep Blood or Columbia CNA Agar with 5% Sheep Blood. A standardized suspension of this culture is prepared in the BBLCRYSTAL™ ANR, GP, RGP, N/H ID Inoculum Fluid provided. The suspension is added to the target area of the panel base, which the use then rocks back and forth to inoculate all the wells contained in the base.

After the base/lid assembly has been incubated for 4 hours at 35-37°C, the assembly is placed on the BBLCRYSTAL Panel Viewer and the color reactions are visually compared to the BBLCRYSTAL GP Color Chart provided. Each reaction is scored as a positive (+) or negative (-) and recorded on the BBLCRYSTAL GP Report Form.

After all results are read, a 10-digit numerical profile is calculated by assigning a value of 4, 2, or 1 to each positive reaction. (Negative reactions are scored as "O".) The values for each column are then added together to obtain the 10 digit Profile Number.

The BBLCRYSTAL ID System Electronic Codebook is loaded into the user's PC and the appropriate database is selected. Then the Profile Number and results of any off-line tests are entered, and the Codebook gives one of the following three results:

(a) a definitive ID;
(b) a tie between two or more species; or
no ID possible with data submitted. (c)

In the case of a definitive ID or a tie between two or more potential ID's, the user can access the statistics for that ID as well as background information for the species identified.

AI/ML Overview

Here's a breakdown of the acceptance criteria and study details for the BBLCRYSTAL™ Gram Positive ID System, based on the provided text:

1. Acceptance Criteria and Reported Device Performance

The document doesn't explicitly state formal "acceptance criteria" with numerical thresholds. However, based on the clinical correlation study results, we can infer the performance targets that were likely considered acceptable for an identification system:

Acceptance Criteria Category	Reported Device Performance (BBLCRYSTAL™ GP ID System)
Correct Identification Rate	90% (668/735) including supplemental testing
Incorrect Identification Rate	7.6% (56/735)
"No Identification" Rate	1.5% (11/735)
Overall Reproducibility	96.7%
Individual Substrate Reaction Reproducibility	79.2% to 100%
Individual QC Organism Reaction Reproducibility	91.4% to 99.8%
Individual Site Reproducibility	95.5% to 97.8%

2. Sample Size and Data Provenance for the Test Set

Sample Size for Test Set: 735 gram positive aerobic isolates.
Data Provenance: The isolates were a mix of:
- "Fresh, routine isolates arriving in the clinical laboratory."
- "Previously identified isolates of the clinical trial sites' choice."
- The study was conducted at "four clinical laboratories," implying data from multiple locations. The country of origin is not specified but is implicitly the United States, given the FDA submission.
Retrospective or Prospective: The "fresh, routine isolates" suggest a prospective component, while "previously identified isolates" suggest a retrospective component. The overall study appears to be a hybrid.

3. Number of Experts and Qualifications for Ground Truth - Test Set

The document states the BBLCRYSTAL™ system's performance was evaluated "against a combination of the API Staph, API 20 Strep, and API Coryne Identification Systems, and conventional methodologies."
While it doesn't explicitly mention "experts" establishing ground truth in the same way one might for image interpretation, the "conventional methodologies" and predicate devices (API systems) served as the current gold standard. This implies that the ground truth was established by standard laboratory practices and the accepted performance of these established systems, which would involve trained microbiology laboratory personnel.
No specific number or qualifications of individual "experts" (like radiologists with X years of experience) are provided, as the ground truth was based on established laboratory methods rather than subjective human interpretation.

4. Adjudication Method for the Test Set

The document implies that the ground truth was established by comparing the BBLCRYSTAL™ results to the results obtained from the predicate API systems and "conventional methodologies."
There is no mention of a specific "adjudication method" like 2+1 or 3+1 for resolving discrepancies. The conventional methodologies and predicate devices implicitly served as the authoritative reference for identification. If there were discrepancies, the conventional methods would likely be the arbiter, as the new device's performance was being measured against them.

5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study

This device is an automated identification system for microorganisms, not an imaging device requiring human interpretation in the same way.
Therefore, an MRMC comparative effectiveness study comparing human readers with and without AI assistance is not applicable and was not conducted. The "human-in-the-loop" aspect primarily involves preparing the sample, inoculating the panel, and entering results, not interpreting complex outputs.

6. Standalone (Algorithm Only) Performance Study

Yes, the performance study described is essentially a standalone evaluation of the BBLCRYSTAL™ GP ID System. The "system" (BBLCRYSTAL™) generates an identification, which is then compared to the ground truth established by other validated methods.
While human users prepare the inoculum, read the reactions, and enter the profile number, the core "identification" logic (the electronic codebook) operates independently to provide the final result. The reported percentages (90% correct, 7.6% incorrect, 1.5% no ID) are indicative of the algorithm's standalone performance within the context of its intended use.

7. Type of Ground Truth Used

The ground truth was established using a combination of:
- Predicate device results: BioMerieux Vitek, Inc., API Staph, API 20 Strep, and API Coryne.
- Conventional methodologies: This would typically include established biochemical tests, phenotypic characteristics, and potentially genetic methods used in clinical microbiology for definitive identification.

8. Sample Size for the Training Set

The document does not specify the sample size used for the training set. It focuses entirely on the performance evaluation cohort (the test set). The "Electronic Codebook" implies a database was built, but details on its creation are not provided.

9. How Ground Truth for the Training Set Was Established

The document does not provide details on how the ground truth for the training set (i.e., the database building for the BBLCRYSTAL™ Electronic Codebook) was established. It describes the product itself and its performance evaluation. Typically, such databases are built using well-characterized, often type strains and a large collection of clinical isolates identified by a comprehensive battery of conventional and molecular methods.

Summary

{0}------------------------------------------------

KA61968

FEB - 4 1997

May 17, 1996

SUMMARY OF SAFETY AND EFFECTIVENESS

SUBMITTED BY:

Virginia C. Weinknecht Becton Dickinson Microbiology Systems P O Box 243 Cockevsville, MD 21030-0243

NAME OF DEVICE:

Trade Name:	BBLCRYSTAL™ Gram Positive ID System
Common Name/Description:	Miniaturized Microorganism ID System
Classification Name:	Microbiology - Discs, Strips, and Reagents,Microorganism Differentiation

PREDICATE DEVICES:

BioMerieux Vitek, Inc., API Staph (K813614) BioMerieux Vitek, Inc., API 20 Strep (K813610) BioMerieux Vitek, Inc., API Coryne (K910304)

DEVICE DESCRIPTION:

{1}------------------------------------------------

TABLE B-I: Taxa List for BBLCRYSTAL™ Gram Positive Identification System

Actinomyces pyogenes Aarococcus species includes A. urinee and A. viridans) Aarococcus urinae Aarococcus viridans Alfoiococcus otitidis Arcanobacterium haemolyticum

Bacillus brevis Bacillus cereus Bacillus circulans Bacillus coaculans Recilius lichaniformis Bacillus megatarium Becillus pumilus Bacillus species lincludes B. brevis, B. circulans, B. coagulans, B. licheniformis, B. megaterium, B. pumilus and B. sphaericus, P. alvei, P. macerans) Bacillus sphaericus Bacillus subtilis

Corvnebacterium aquaticum Corvnebacterium bovis Corvnebectarium diotheries linctudes C. dintharies sea arevis. C. diptheriee ssp mitis and C. diptheriae sap intermedius Corynabacterium genitalium Corvnebacterium jelkalum Corynebacterium kutscheri Corynebecterium propinguum Corynebacterium pseudodiotheriticum Corvnebectarium pseudogenitalium Corvnebacterium pseudotuberculosis Corynebacterium renele group Corynebacterium species (includes C. aquaticum, C. bovis, C. kutacheri, C. propinquum, c. psaudodiphthariticum, C. pseudotuberculosis, C. renale group, C. striatum and C. ulcerans) Corvnebacterium striatum Corvnebacterium ulcerans Enterococcus avium

Enterococcus casselifievus/gallinerum Enterocuccus durans Enterococcus faccall: Enterococcus faecium Enterococcus hime Enterococcus raffinosus Enterocuccus solitanus Ervsioelothrix rhusiopathias

Gardnerella vaginalis Gemells hasmolysans Gemella morbillorum Gamella species (includes G. haemolysans, and G. morbillorum) Globicatelle sanguis

Helcococcus kunzil

Lactococcus garviese Lactococcus lectis sep cremoris Lactococcus lectis sap hordnied Lactococcus lectis sap lactis Lactococcus reffinoloctis Lactococcus spacies lincludes L. lactis sao cremoris. L. lactis sso hordnise. I., loctis sap lactis and L. reffinolactis) Lauconostoc citreum Leuconostoc lactis Leuconostoc mesenteroides ssp mesenteroides Leuconostoc psaudomesenteroides Leuconostoc species (includes L. citreum, L. lectis. L. mesenteroides sap mezenteroides and L. psaudomesenteroides Listeria gravi Listeria ivanovii sso ivanovii Listeria monocytogenes Listeria murravi

Micrococcus kristinae Micrococcus luteus Micrococcus Iylee Micrococcus roseus Micrococcus sedentarius Micrococcus species lincludes M. kristinae, M. lutous. M. M. Mee, M. roseus, M. sedentarius)

Oerskovia species lincludes O. turbate, and O. xanthineolytical

Peonibacillus alvai Peonibacitus mecarans Pediococcus demnasus Pediococcus pervukis Pediococcus pentoseceus Pediococcus species (includes P.demnosus, P. pervulus, and P. pentosecous)

Rhodococcus equi Rothia dentocariosa

Staphylococcus aurous Staphylococcus suricularis Staphylococcus capitis lincludes S. capitis ssp capitis and S. capitis sup ureolyticus) Staphylococcus capras Staphylococcus camosus Staphylococcus cohnii lincludes S. cohnii seo cohnii and S. cohnii sap ures/vticum) Staphylococcus cohnil sap cohnii Staphylococcus cohnil sep urealyticum Staphylococcus epidermidia Staphylococcus equonim Staphylococcus felis Staphylococcus gallinerum Staphylococcus heemolyticus Staph viococcus hominis Staphylococcus intermedius

Staphylococcus kloosii Staphylococcus lantus Staphylococcus lugdunensis Staphylococcus pasteuri Staphylococcus seccharolyticus Staphylococcus saprophyticus Staphylococcus schleiferi (includes S. schleiferi ssp cagulans and S. schleiferi ssp schleiferi Staphylococcus sciuri Staphylococcus simulans Staphylococcus vitulus Staphylococcus werner Staphylococcus xylosus Stometococcus muciloginosus Streptococcus acidominimus Streptococcus agalecties Streptococcus anginosus Streptococcus bovis lincludes S. bovis I and S. bovis II) Streptococcus constellatus Streptococcus cricetus Streptococcus crists Streptococcus dvegalacties Streptococcus equi lincludes S. equi sap equi and S. equi ssp zooepidemicus) Streptococcus equi ssp equi Streptococcus equi ssp zooepidemicus Streatacaccus equinus Streptococcus gordonii Streptococcus internedius Streptococcus milleri group lincludes S. anginosus, S. constellatus and S. intermedius) Streptococcus mitis Streptococcus mitis group (includes S. mitis and S. oralis) Streptococcus mutans Streptococcus mutans group lincludes S. cricetus, S. mutans and S. sobrinus) Streptococcus oralis Straptococcus parasanguis Streptococcus pneumonies Streptococcus porcinus Streptococcus pyogenes Streptococcus salvanus Streptococcus saliverius group (includes S. saliverius and S. vastibularis) Streptococcus sanguis Streptococcus sanguis group lincludes S. crista, S. gordonii, S. perasenguis and S. sanguis) Streptococcus sobrinus Streptococcus uberis Streptoccus vestibularis

Turicella otitidis

' These taxa have < 10 unique CRYSTAL profiles in the current database.

{2}------------------------------------------------

PRODUCT DESCRIPTION:

(a) a definitive ID;
(b) a tie between two or more species; or
no ID possible with data submitted. (c)

In the case of a definitive ID or a tie between two or more potential ID's, the user can access the statistics for that ID as well as background information for the species identified.

{3}------------------------------------------------

In the case where no ID is possible, the Codebook suggests that the user perform a purity check of the test isolate. If culture purity has been confirmed, then it is likely that (i) the test isolate is producing atypical Crystal reactions (which may also be caused by procedural errors), (ii) the test species is not part of the intended taxa, or (iii) the system is unable to identify the test isolate with the required level of confidence. Once user error has been ruled out, the Codebook suggests that additional testing must be done to establish an identification.

PERFORMANCE DATA:

Clinical Correlation:

In a study conducted at four clinical laboratories, the performance of the BBLCRYSTAL Gram Positive ID System was evaluated against a combination of the API Staph, API 20 Strep, and API Coryne Identification Systems, and conventional methodologies. Fresh, routine isolates arriving in the clinical laboratory, as well as previously identified isolates of the clinical trial sites" choice were utilized in the study. A total of 735 gram positive aerobic isolates were tested; 90% (668/735) of these isolates were correctly identified (including supplemental testing) using the BBLCRYSTAL™ GP ID System; 7.6% (56/735) were incorrectly identified; and 1.5% (11/735) vielded a "No Identification" result.

Reproducibility:

At the same four clinical laboratories, reproducibility of the BBLCRYSTAL™ GP ID System was established by testing ten (10) Quality Control strains in triplicate on three days. Evaluations were performed of individual and overall reproducibility of substrate reactions, of QC organism reactions, and of interand intra-laboratory reproducibility.

Overall reproducibility was calculated as 96.7%. Reproducibility of individual substrate reactions ranged from 79.2 to 100%; individual QC organism reaction reproducibility ranged from 91.4 to 99.8%; and individual site reproducibility from 95.5 to 97.8%.

Regulation Number and Section

§ 866.2660 Microorganism differentiation and identification device.

(a)
Identification. A microorganism differentiation and identification device is a device intended for medical purposes that consists of one or more components, such as differential culture media, biochemical reagents, and paper discs or paper strips impregnated with test reagents, that are usually contained in individual compartments and used to differentiate and identify selected microorganisms. The device aids in the diagnosis of disease.(b)
Classification. Class I (general controls). The device is exempt from the premarket notification procedures in subpart E of part 807 of this chapter subject to § 866.9.