Search Results

The Colibrí is an automated in vitro diagnostic specimen preparation system for use with WASPLab to prepare MALDI-TOF targets for the bioMérieux VITEK MS systems or Bruker MALDI Biotyper CA mass spectrometry systems for qualitative identification and microbial suspension for the bioMérieux VITEK 2 systems or Beckman Coulter MicroScan WalkAway Antimicrobial Susceptibility Testing (AST) systems for qualitative testing of isolated colonies of gram-negative and gram-positive bacterial species grown on solid culture media.

The Colibrí is an automated pre-analytical processor that picks isolated colonies designated by the operator and uses a pipetting system to prepare MALDI-TOF MS (Matrix-Assisted Laser Desorption/lonization-Time of Flight Mass Spectrometry) target slides for bacterial identification and microbial suspension at known concentration for Antimicrobial Susceptibility Testing and purity assessment.

The Colibrí software records the identity of each sample and its position on the target slide and communicates this information electronically to the MALDI-TOF MS analyzers.

Bacterial suspensions for AST and purity plates are identified by barcode label.

The Colibrí is intended for use by trained healthcare professionals in clinical laboratories in conjunction with other clinical and laboratory findings, including Gram staining, to aid in the diagnosis of bacterial infections.

The Colibrí has not been validated for use in the identification or processing of yeast species, molds, Nocardia, or mycobacteria.

The Colibrí is an instrument which automates the picking of selected colonies from plated media and prepares MALDI target slides for the bioMérieux VITEK MS systems or the Bruker MALDI Biotyper CA systems that are used in clinical laboratories for identification and differentiation of organisms grown on plated media by Matrix-Assisted Laser Desorption/Jonization Time-of Flight Mass Spectrometry (MALDI-TOF MS). The Colibri automates the preparation of microbial suspensions at known concentration for bioMérieux VITEK 2 systems and Beckman Coulter MicroScan WalkAway systems that are used in clinical laboratories for AST analyses. Moreover, the Colibrí is used for Purity Plates preparation for purity assessments.

The Colibrí includes the following components:

• Colibrí instrument and software with on-board pipetting system and nephelometer .
• Colibrí Primary Tubes
• Colibrí Spreader
• Colibrí Daily Verification kit.

Colibri is designed to be used in conjunction with the WASPLab device for culture plate incubation and image analysis. After appropriate plate incubation, the operator selects the colonies from a digital image of culture media plate streaked with microbiological human specimen, available through WebApp software, the WASPLab User Interface.

The operator assigns the automatic ID or AST tasks to the isolated colonies to be processed. Then, the operator loads the plates in the Collbri where colonies are automatically picked, spotted on the target slide and overlayed with the matrix or suspended into the dedicated solution for the preparation of the microbial suspension for AST purposes (Secondary Tube).

When used in conjunction with the bioMérieux VITEK MS systems, the Colibri can prepare the 48-spot target slides by performing the direct spotting of colonies. The calibrator used for quality control is manually applied by the operator at the end of the automated colony spotting. When used in conjunction with the Bruker MALDI Biotyper CA systems, the Colibri can prepare either reusable 48-spot or disposable 96-spot targets by performing the Direct Transfer Sample Procedure. The BTS used for quality control is manually applied by the operator at the end of the automated colony spotting.

When used in conjunction with the bioMérieux VITEK 2 systems or the Beckman Coulter MicroScan WalkAway systems, the Colibri can prepare the microbial suspension at the proper concentration by direct colony suspension method. The onboard nephelometer allows the preparation of Secondary Tubes (AST suspensions) at the correct concentration and the Colibri Spreader is used for Purity Plates preparation.

The Colibrí software records the identity of each sample and its position on the target slide and communicates this information electronically to the MALDI-TOF MS analyzers.

The traceability of prepared Secondary Tube and Purity Plates is maintained by dedicated labels applications.

Colibrí requires four different calibrations, one on the nephelometer, three on the cameras. None of these calibration activities require user intervention if not in terms of periodical cleaning of the mechanical component as described in the dedicated section of the User Manual. The Set-up calibration of nephelometer and camera units are performed during the device initial setup. Auto-calibration is performed at the end of the initial set-up and periodically during the preventive maintenance to check that all the mechanical references can be found inside the positioning tolerances, that the I/Os are responsive. Runtime calibration is performed during the normal usage to automatically check the proper functioning of the Colibrí.

Colibrí requires a daily nephelometer verification to check the proper reading of suspensions at different turbidity values.

The Colibrí device is an automated in vitro diagnostic specimen preparation system. The provided text describes the acceptance criteria and the study that proves the device meets these criteria for preparing microbial suspensions for Antimicrobial Susceptibility Testing (AST) using Beckman Coulter MicroScan WalkAway systems.

Here's a breakdown of the requested information:

1. A table of acceptance criteria and the reported device performance:

The acceptance criteria are implicitly derived from the successful outcomes of the analytical studies. The performance is reported as the percentage of successful outcomes for each metric.

• E. coli ATCC 25922: 3-7 x 105 CFU/mL
• Other bacteria: 2-8 x 105 CFU/mL | 98.5% of prepared suspensions had microbial concentration within acceptable limits.
• E. coli: 100% (36/36)
• Pseudomonas aeruginosa: 96.7% (29/30)
• Staphylococcus aureus: 97.6% (41/42)
• Enterococcus faecalis: 100% (30/30) |
  | AST Challenge Test (Agreement with Manual Preparation) | Essential Agreement (EA) of MICs: High agreement
  Category Agreement (CA): High agreement
  Discrepancies (vmj, maj): Low/none | Overall EA: 100% (1232/1232 evaluable MIC results within 1 two-fold dilution)
  Overall CA: 98.4% (4187/4254 SIR categorizations in agreement)
  Very Major discrepancy (vmj): 0
  Major discrepancy (maj): 0 |
  | Reproducibility Study | Best-case reproducibility: ≥95% (implied)
  Worst-case reproducibility: ≥89% (implied) | Best-case reproducibility: ≥99.8% (all panels combined)
  Worst-case reproducibility: ≥94.3% (all panels combined) |
  | Sample preparation for Quality Control | 100% of MIC values within CLSI/panel IFU QC range | 100% (all tested organisms and antimicrobial agents) |
  | Purity Plates Evaluation (Cross-contamination) | Absence of cross-contamination (100% monomicrobial growth) | 100% (453/453 Purity Plates showed monomicrobial growth) |

2. Sample sizes used for the test set and the data provenance:

• Preparation of Microbial Suspensions for AST:

  • Test Set Size: 132 microbial suspensions (36 E. coli, 30 Pseudomonas aeruginosa, 42 Staphylococcus aureus, 30 Enterococcus faecalis).
  • Data Provenance: Not explicitly stated (e.g., country of origin). The study involved three Colibrí instruments, suggesting internal validation. Retrospective or prospective is not specified, but the nature of the validation suggests prospective testing.

• AST Challenge Test:

  • Test Set Size: Different species: Enterobacterales (n=50 isolates), Staphylococcus (n=20 isolates), Streptococcus (n=12 isolates), Enterococcus (n=18 isolates), non-fermenters (n=10 isolates). Each processed by three Colibrí instruments, yielding varying numbers of MIC results and SIR categorizations across different panels (e.g., 2454 total MIC results for Enterobacterales on NM-NF50 panel).
  • Data Provenance: Not explicitly stated (e.g., country of origin). The study design implies prospective testing within the manufacturer's validation process.

• Reproducibility Study:

  • Test Set Size: 9 gram-positive and 9 gram-negative strains, processed on 3 Colibrí instruments over 3 days, with each condition tested in triplicate (total of 27 replicates for each strain-antimicrobial agent combination).
  • Data Provenance: Not explicitly stated (e.g., country of origin, retrospective or prospective). Implied prospective.

• Sample preparation for Quality Control:

  • Test Set Size: CLSI-recommended reference strains (Escherichia coli ATCC 25922, Pseudomonas aeruginosa ATCC 27853, Staphylococcus aureus ATCC 29213, Enterococcus faecalis ATCC 29212). The "No. MIC within QC range" indicates a total of 30 or 36 or 42 tests for each drug-organism combination, per three instruments.
  • Data Provenance: Not explicitly stated. Implied prospective.

• Purity Plates Evaluation:

  • Test Set Size: 453 purity plates (150 from AST Challenge, 162 from AST Reproducibility, 141 from Quality Control studies).
  • Data Provenance: Not explicitly stated. Implied prospective.

3. Number of experts used to establish the ground truth for the test set and the qualifications of those experts:

• The ground truth for AST results (MICs and SIR categories) generally refers to the results obtained from a reference method. In this case, "Manual suspension was used as comparative method" for the AST Challenge Test. This implies that manually prepared suspensions, processed by the MicroScan WalkAway, served as the reference standard.
• The text does not specify the number of experts or their qualifications for establishing this manual ground truth. It mentions that three different technicians operated the Colibrí machines, but it doesn't detail the personnel for the manual comparative method or for interpreting the results as ground truth beyond the "FDA-Recognized Antimicrobial Susceptibility Test Interpretive Criteria" and CLSI guidelines.

4. Adjudication method (e.g. 2+1, 3+1, none) for the test set:

• The document does not describe any expert adjudication process for the test set results. The comparison is made against a "manual result" (ground truth). The discrepancies (vmj, maj, min) are simply categorized and reported, implying a direct comparison without further expert review for resolving initial disagreements.

5. If a multi-reader multi-case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance:

• No, an MRMC study was not conducted. This device (Colibrí) is an automated system for sample preparation and does not involve human "readers" or "AI assistance" in the typical sense of image analysis for diagnosis. Its role is to automate a laboratory process, and the performance is measured against reference methods, not human interpretation.

6. If a standalone (i.e. algorithm only without human-in-the-loop performance) was done:

• The studies presented are primarily standalone (algorithm only without human-in-the-loop performance) in terms of the Colibrí device's automated functions. The device picks colonies, prepares suspensions, and records data automatically. The performance metrics (inoculum density, MIC accuracy, reproducibility, purity) assess the device's output against established standards and manual methods.
• While an operator designates colonies for picking, the act of preparation itself is automated and evaluated for its accuracy. The "manual suspension" used for comparison acts as the reference for the "algorithm only" performance of the Colibrí in producing the suspension.

7. The type of ground truth used (expert consensus, pathology, outcomes data, etc):

• The ground truth for the analytical studies combines reference methods/standards and established guidelines:
  • For microbial suspension concentration: Viable cell count (CFU/mL) against CLSI and FDA guidelines.
  • For AST results: Manual suspension preparation as the comparative method, and comparison of MICs and SIR categories against FDA-Recognized Antimicrobial Susceptibility Test Interpretive Criteria and CLSI guideline M07.
  • For reproducibility: Comparison to the "mode result" (most frequent MIC value) and established reproducibility criteria (e.g., within one doubling dilution).
  • For Quality Control: CLSI-recommended QC ranges and MicroScan panel IFU values.
  • For Purity Plates: Visual assessment (implied) to confirm monomicrobial growth.

8. The sample size for the training set:

• This document describes performance validation studies for a medical device (Colibrí), not a machine learning model. Therefore, there is no "training set" in the context of data used to train an AI algorithm. The Colibrí is an automated instrument with pre-programmed functions, not a learning algorithm that requires a training dataset.

9. How the ground truth for the training set was established:

• As explained above, there is no training set for this device in the context of AI/ML.

Ask a specific question about this device

The Collbri™ System is an in vitro diagnostic device comprised of the Collbri™ Preparation Station for use with the bioMérieux VITEK® MS or Bruker MALDI Biotyper® CA mass spectrometry systems for qualitative identification and with the bioMérieux VITEK® 2 Antimicrobial Susceptibility Testing (AST) system for qualitative testing of isolated colonies of gram-negative and gram-positive bacterial species grown on solid culture media. The Collbri™ System is a semi-automated pre-analytical processor that picks isolated colonies designated by the operator and uses a pipetting system to prepare MALDI-TOF MS (Matrix-Assisted Laser Desorption/Jonization-Time of Flight Mass Spectrometry) target slides for bacterial identification and microbial suspension at known concentration for Antimicrobial Susceptibility Testing and purity assessment.

The Colibrí software records the identity of each sample and its position on the target slide and communicates this information electronically to the MALDI-TOF MS analyzers.

Bacterial suspensions for AST and purity plates are identified by barcode label.

The Colibr™ System is intended for use by trained healthcare professionals in clinical laboratories in conjunction with other clinical and laboratory finding Gram staining, to aid in the diagnosis of bacterial infections.

The Collbri™ System has not been validated for use in the identification or processing of yeast species, Mocardia, or mycobacteria.

The Copan Colibrí System is designed to be used as an accessory of the downstream MALDI-TOF MS and antimicrobial susceptibility testing (AST) analyzers automating various manual steps in the workflow for the preparation of samples for the identification of isolated colonies and for AST of isolated colonies of gram-negative and gram-positive bacterial species grown on solid culture media.

The Colibrí System automates the preparation of MALDI target slides for the bioMérieux VITEK MS or the Bruker MALDI Biotyper CA System that are used in clinical laboratories for identification (ID) of organisms grown on plated media by Matrix-Assisted Laser Desorption/Jonization Time-of Flight Mass Spectrometry (MALDI-TOF MS). The Colibri System automates the preparation of microbial suspensions at known concentration for bioMérieux VITEK 2 System that is used in clinical laboratories for AST analyses. Moreover, the Colibri System is used for Purity Plates preparation for purity assessments.

The Colibrí System comprises the Colibrí Vision System and Colibrí Preparation Station hardware modules and pipette tips, Primary Tubes, Spreader and nephelometer Verification Kit as consumables. After appropriate plate incubation, the operator using the graphical User Interface (Image Reading Interface) chooses the plates exhibiting adequate growth and selects the isolated colonies to be processed assigning the automatic ID or AST tasks. By using the Colibrí Vision System, specific colonies to be picked are designated by the operator on a digital plate. The Operator manually loads the plates in the Colibri Preparation Station where colonies are automatically picked, spotted on the target slide and overlayed with the matrix or suspended into the dedicated solution for the preparation of the microbial suspension for AST purposes (Secondary Tube).

When used in conjunction with the bioMérieux VITEK MS, the Colibrí System can prepare the 48spot target slides by performing the direct spotting of colonies. The calibrator used for quality control is manually applied by the operator at the end of the automated colony spotting. When used in conjunction with the Bruker MALDI Biotyper CA System, the Colibrí System can prepare either reusable 48-spot or disposable 96-spot targets by performing the Direct Transfer Sample Procedure. The BTS used for quality control is manually applied by the operator at the automated colony spotting.

When used in conjunction with the bioMérieux VITEK 2, the Colibrí System can prepare the microbial suspension at the proper concentration by direct colony suspension method. The onboard nephelometer allows the preparation of Secondary Tubes (AST suspensions) at the correct concentration and the Colibrí Spreader is used for Purity Plates preparation.

Copan WASP S.r.l., Traditional 510(k)- Colibrí System

The Colibrí software records the identity of each sample and its position on the target slide and communicates this information electronically to the MALDI-TOF MS analyzers.

The traceability of prepared Secondary Tube and Purity Plates is maintained by dedicated labels applications.

Colibri System requires four different calibrations, one on the nephelometer, three on the cameras. None of these calibration activities require user intervention if not in terms of periodical cleaning of the mechanical component as described in the dedicated section of the User Manual. The Set-up calibration of nephelometer and camera units positioned on the Colibrí Vision System and on the Colibrí Preparation Station are performed during the device initial setup. Auto-calibration is performed at the end of the initial set-up and periodically during the preventive maintenance to check that, in the Collbrí Preparation all the mechanical references can be found inside the positioning tolerances, that the I/Os are responsive. Run-time calibration is performed during the normal usage to automatically check the proper functioning of the Colibrí Vision System and the Colibrí Preparation Station.

Colibrí System requires a daily nephelometer verification to check the proper reading of suspensions at different turbidity values.

The provided text describes the performance data for the Colibrí System, an in vitro diagnostic device. The acceptance criteria and performance are primarily focused on its ability to accurately prepare microbial suspensions for Antimicrobial Susceptibility Testing (AST) and for MALDI-TOF MS identification, compared to manual methods.

Here's an attempt to extract the requested information. Please note that the document is a 510(k) summary, which often provides summarized performance data rather than detailed study protocols. Therefore, some information might be explicitly stated as "not applicable" or inferred based on common practices for such device clearances.

1. A table of acceptance criteria and the reported device performance

The document outlines several analytical studies. While explicit "acceptance criteria" are not always presented as target percentages, the performance results are given, implying that these results met the internal pre-defined acceptance thresholds for substantial equivalence.

2. Sample sizes used for the test set and the data provenance

• Nephelometer Calibration Verification: 300 suspensions were prepared (20 suspensions for each of 5 concentrations, across 3 operators and 3 Colibrí systems, though the calculation isn't directly 2053*3).
• Pipettor Trueness and Precision: 10 measurements for each of 4 volumes, across 3 Colibrí System pipettors (10 * 4 * 3 = 120 measurements).
• E. coli Suspensions Preparation Verification: The exact number of suspensions isn't provided, but it states "A variable number of colonies was selected... to create different suspensions... Three Colibrí Systems run by three different operators were included."
• Colony Picking and Microbial Suspensions for AST: 6 bacterial species (3 Gram-Negative and 3 Gram-Positive) grown in 2 polymicrobial mixtures on different culture media.
• AST Challenge Test:
  • Total Tested: 5991 (across various antibiotics and organism groups within the challenge test).
  • Evaluated MIC results: 1883 evaluable MIC results.
  • Organism groups: Enterobacterales (n=62 strains), Staphylococcus (n=16 strains), Streptococcus (n=30 strains), Enterococcus (n=16 strains), and non-fermenters (n=32 strains).
  • Strains: Both susceptible and resistant strains, exhibiting a range of on-scale MIC values.
  • Media: Trypticase Soy Agar + 5% Sheep Blood, MacConkey Agar, and Columbia agar + 5% sheep blood.
  • Incubation times: Varied (e.g., 14h, 24h for Enterobacterales/Non-fermenters, 18h for Staphylococcus/Enterococcus/Streptococcus).
  • Data Provenance: Not explicitly stated (e.g., country of origin, retrospective/prospective). However, given it's a 510(k) submission for a medical device manufacturer (Copan WASP S.r.l., Italy), it's highly likely to be internal, prospective studies conducted at their facilities or collaborator sites.
• Reproducibility Study: Each microorganism was tested with the appropriate antibiotic panel, with each condition tested in triplicate, for a total of 81 replicates for each combination strain-antimicrobial agent across three Colibrí Systems and three operators over three days.
• Purity Plate Growth: 2,364 purity plates.
• QC Sample Preparation: Conducted daily at the beginning of the working session on each instrument involved in the Analytical Studies.

3. Number of experts used to establish the ground truth for the test set and the qualifications of those experts

• This information is not explicitly provided in the document.
• For the AST Challenge Test, the "ground truth" (or comparator method) was the MICs obtained by the bioMérieux VITEK 2 using manual sample preparation, interpreted according to FDA-Recognized Antimicrobial Susceptibility Test Interpretive Criteria. This implies the ground truth relies on established, validated laboratory methods and interpretations, rather than subjective expert consensus.
• For other analytical studies (e.g., nephelometer accuracy, pipettor precision), the ground truth generally relies on quantitative measurements using calibrated instruments and standard protocols (e.g., viable cell counts for McFarland turbidity verification, gravimetric measurements for pipettor accuracy).

4. Adjudication method for the test set

• This is not applicable in the context of this device's performance validation. The device automates a pre-analytical step. The performance is assessed by comparing its output (prepared microbial suspensions) to a reference method (manual preparation for VITEK 2). It's not a diagnostic AI system requiring expert adjudication of image interpretations or clinical diagnoses.

5. If a multi reader multi case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance

• No, an MRMC comparative effectiveness study was not done. This type of study is relevant for AI systems that assist human readers in interpreting medical images (e.g., radiology AI). The Colibrí System is a laboratory automation device for preparing samples. It does not involve human "readers" of AI outputs in a diagnostic context that would require such a study design.

6. If a standalone (i.e. algorithm only without human-in-the-loop performance) was done

• The performance data are essentially standalone (algorithm/device only) in terms of its ability to perform the physical process of colonial picking and suspension preparation. The device's output is then fed into other IVD analyzers (VITEK 2, bioMérieux VITEK MS, Bruker MALDI Biotyper CA System) for identification and AST.
• The comparison in the AST Challenge Test is between the Colibrí System's automated preparation and manual preparation which is the existing standard. So, it's comparing automated device output to a manual, human-executed process, where the subsequent analysis (VITEK 2) is the same. The data provided (EA, CA, error rates) represent the performance of the Colibrí system's prepared samples, which is a form of standalone performance.

7. The type of ground truth used (expert consensus, pathology, outcomes data, etc.)

• Comparatative Ground Truth/Reference Method: For the AST Challenge Test, the ground truth was derived from the MICs obtained by the bioMérieux VITEK 2 using manual sample preparation, interpreted according to FDA-Recognized Antimicrobial Susceptibility Test Interpretive Criteria. This is a well-established and standardized laboratory reference method.
• Quantitative Ground Truth: For other analytical studies, the ground truth was based on quantitative laboratory measurements, such as:
  • Viable cell counts (CFU/mL) for confirming bacterial concentration for nephelometry.
  • Gravimetric measurements for pipetting accuracy.
  • Visual inspection of purity plates and accepted microbiological methods to ensure monomicrobial suspensions and absence of contamination.

8. The sample size for the training set

• The document describes performance validation studies, not product development or AI model training. Therefore, information about a "training set" (in the context of machine learning) is not applicable or provided. This device automates a physical process, not a machine learning model that needs training data in the traditional sense. The "training" of the device likely refers to physical calibration and quality control.

9. How the ground truth for the training set was established

• As per point 8, the concept of a "training set" for an AI/machine learning model is not applicable to the description of this device's validation. The device's mechanics and software are validated against established engineering and microbiology standards.

Ask a specific question about this device

The Colibrí System is an in vitro diagnostic device comprised of the Colibrí Vision System and Collbrí Preparation Station for use with the bioMérieux VITEK MS or Bruker MALDI Biotyper CA mass spectrometry systems for qualitative identification of isolated colonies of Gram-negative bacterial species grown on solid culture media. The Collbri System is a semi-automated pre-analytical processor that picks isolated by the operator and uses a pipetting system to prepare MALDI-TOF MS (Matrix-Assisted Laser Desorption/Ionization-Time Of Flight Mass Spectrometry) target slides. The Colibri software records the identity of each sample and its position on the target slide and communicates this information electronically to the MALDI-TOF MS analyzer.

The Colibrí System is intended for use by trained healthcare professionals in clinical laboratories in conjunction with other clinical and laboratory finding Gram staining, to aid in the diagnosis of bacterial infections.

The Colibrí System has not been validated for use in identification of yeast species.

The Copan Colibrí System is designed to be used as an accessory of the downstream MALDI-TOF analyzers automating various manual steps in the workflow for the preparation of samples for the identification of isolated colonies of microorganisms cultured from the human body.

The Colibrí System automates the preparation of MALDI target slides for the bioMérieux VITEK MS or the Bruker MALDI Biotyper CA System that are used in clinical laboratories for identification and differentiation of organisms grown on plated media by Matrix-Assisted Laser Desorption/Ionization Time-of Flight Mass Spectrometry (MALDI-TOF MS). The system comprises the Colibrí Vision System and Colibrí Preparation Station and pipette tips as consumables. After appropriate plate incubation, the operator using the graphical User Interface (Image Reading Interface) chooses the plates exhibiting adequate growth and selects the isolated colonies to be processed assigning the automatic ID tasks. By using the Collbrí Vision System, specific colonies to be picked are designated by the operator on a digital plate. The Operator manually loads the plates in the Colibri Preparation Station where colonies are automatically picked, spotted on the target slide and overlayed with the matrix.

When used in conjunction with the bioMérieux VITEK MS, the Colibrí System can prepare the 48spot target slides by performing the direct spotting of colonies. The calibrator used for quality control is manually applied by the operator at the end of the automated colony spotting. When used in conjunction with the Bruker MALDI Biotyper CA System, the Colibrí System can prepare either reusable 48-spot or disposable 96-spot targets by performing the Direct Transfer Sample Procedure. The BTS used for quality control is manually applied by the operator at the and of the automated colony spotting.

The Colibrí software records the identity of each sample and its position on the target slide and communicates this information electronically to the MALDI-TOF MS analyzers.

Colibri System requires three different calibrations. None of these calibration activities require user intervention if not in terms of periodical cleaning of the mechanical component as described in the dedicated section of the User Manual. Set-up calibration is performed during the device initial setup for the camera units positioned on the Colibrí Vision System and on the Colibrí Preparation Station. Auto-calibration is performed at the end of the initial set-up and periodically during the preventive maintenance to check that, in the Colibri Preparation Station, all the mechanical references can be found inside the positioning tolerances, that the I/Os are responsive. Run-time calibration is performed during the normal usage to automatically check the proper functioning of the Colibrí Vision System and the Colibri Preparation Station.

The provided text describes the performance data for the Colibrí System, an in vitro diagnostic device. The acceptance criteria are implicitly defined by the performance observed in various analytical studies, with the goal of demonstrating substantial equivalence to predicate devices for qualitative identification of isolated colonies of Gram-negative and Gram-positive bacterial species.

Here's an analysis of the acceptance criteria and study proving the device meets them:

1. Table of Acceptance Criteria (Implicit) and Reported Device Performance

The document does not explicitly state pre-defined acceptance criteria in a dedicated table with specific thresholds. However, based on the studies conducted, the implicit acceptance criteria appear to be:

• High agreement with expected identification (ground truth).
• Absence of wrong identifications (false positives).
• Maintenance of performance across different conditions (e.g., positional effects, culture age, different MALDI-TOF MS systems and target types, different operators, different Colibrí systems).
• Comparable performance to manual preparation.
• No cross-contamination.

Here's a table summarizing the reported device performance, which serves as the evidence that these implicit criteria were met. The percentages below represent the agreement with the expected ID (ground truth) for high-confidence results where available, or overall agreement as calculated by the study.

2. Sample Sizes Used for the Test Set and Data Provenance

• Test Set (Analytical Studies):

  • Colony Picking: 1390 spots (VITEK MS) and 1690 spots (Bruker MALDI Biotyper CA).
  • Positional Effect: 864 spots (VITEK MS), 864 spots (Bruker MALDI Biotyper CA US IVD 48 Spot), 1692 spots (Bruker MALDI Biotyper CA MBT Biotarget 96 US IVD).
  • Inclusivity: 392 spots/strains (VITEK MS) and 468 spots/strains for each Bruker MALDI Biotyper CA system (US IVD 48 Spot and MBT Biotarget 96 US IVD).
  • Specificity: 20 spots (VITEK MS) and 20 spots (Bruker MALDI Biotyper CA).
  • Reproducibility: 1800 spots (VITEK MS) and 1800 spots (Bruker MALDI Biotyper CA).
  • Cross-Contamination: 572 spots (VITEK MS) and 686 spots (Bruker MALDI Biotyper CA total for both systems).
  • Colony Stability: 576 spots (VITEK MS) and 1440 spots (Bruker MALDI Biotyper CA).

• Data Provenance: The document generally indicates "on-panel" and "off-panel" strains, often referring to common clinical isolates in the US. No explicit country of origin for the clinical samples/strains themselves is stated beyond generic "human specimens" or "commonly isolated Gram-positive and Gram-negative species in the US". The studies were conducted by a single operator (Inclusivity, Specificity, Cross-Contamination, Colony Stability) or two operators (Reproducibility) on multiple Colibrí systems (Reproducibility, Colony Picking, Inclusivity, Cross-Contamination) in laboratory settings. The studies are prospective as they are designed experiments to evaluate the device performance.

3. Number of Experts Used to Establish the Ground Truth for the Test Set and Their Qualifications

• The document implies that the ground truth for identification was established by the expected strain identity of the stock cultures used in the analytical studies. This means the identities of the bacterial strains were known beforehand (e.g., ATCC strains or well-characterized lab strains).
• For the performance results themselves, identification was based on the output of the VITEK MS or Bruker MALDI Biotyper CA systems, which generate confidence values or log scores. The interpretation of these scores (e.g., what constitutes a "correct single choice") is intrinsic to the performance specifications of these established MALDI-TOF MS systems.
• No human experts (e.g., radiologists) were used to establish the ground truth for these microbial identification studies. The "ground truth" is the presumed identity of the bacterial strain used in the experiment.

4. Adjudication Method for the Test Set

• Not applicable in the conventional sense. Since the ground truth for the test set was the expected strain identity of known bacterial cultures, there was no need for human expert adjudication to resolve discrepancies in interpretation. The output of the MALDI-TOF MS system for each sample was compared directly to the known identity of the inoculated strain. Discrepancies (low confidence, no ID, wrong ID) were noted and analyzed against the overall performance.

5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was Done

• No. An MRMC study is typically performed for image-based diagnostic aids where multiple human readers interpret cases with and without AI assistance. This study involves microorganism identification, a laboratory process, not direct human interpretation of images for diagnosis.
• The study compares the performance of the automated Colibrí System to manual preparation (the standard of care, as listed in the predicate device comparison tables), which is a different type of comparison than human readers with/without AI assistance.
• Effect size of human reader improvement: Not applicable, as no human reader study was conducted. The comparison is between an automated sample preparation method (Colibrí) and a manual sample preparation method for laboratory identification.

6. If a Standalone (i.e. algorithm only without human-in-the loop performance) was Done

• Yes, indirectly. The Colibrí System is a semi-automated pre-analytical processor. While an operator designates the colony, the picking, spotting, and matrix application are automated.
• The performance metrics provided (e.g., % agreement, no wrong ID) are for the Colibrí System's ability to prepare samples for subsequent MALDI-TOF MS analysis, which then generates the identification result. The Colibrí software records location and transmits information to the MALDI-TOF MS analyzer.
• The study evaluated the effectiveness of the automated sample preparation step compared to manual preparation, which directly impacts the downstream MALDI-TOF MS identification. However, the identification itself is performed by the MALDI-TOF MS analyzer, not the Colibrí system's "algorithm" in isolation, except for tracking sample positions. The Colibrí's key "algorithm" is in its vision system and robotic picking/spotting, which impacts the quality of the sample for the MALDI-TOF MS.

7. The Type of Ground Truth Used

• Expected Strain Identity: The ground truth for the analytical studies was the known, expected identity of the bacterial strains used. These were presumably well-characterized laboratory strains (e.g., ATCC cultures) with a confirmed identity. This is a highly controlled form of ground truth.

8. The Sample Size for the Training Set

• Not explicitly stated in terms of a "training set" for an AI/machine learning model used within Colibrí's core functions. The Colibrí System is described as a "semi-automated pre-analytical processor" that uses "a pipetting system" and "Colibrí software records the identity of each sample and its position". It's unclear if the "Colibrí Vision System" employs complex machine learning that would require a dedicated training set beyond basic image processing for colony detection and localization.
• The document implies that the system is automating a manual process, meaning its "training" pertains more to engineering and calibration rather than a machine learning model for diagnosis. The performance studies focus on the system's accuracy in physical manipulation and compatibility with existing MALDI-TOF MS systems.

9. How the Ground Truth for the Training Set Was Established

• Not applicable as a distinct ML training set is not described. If the "Colibrí Vision System" used internal algorithms that were "trained" to recognize colony morphology for picking, the ground truth would likely have been established by manually annotating colonies on images or physical plates as "pickable" or "not pickable" based on expert microbiological judgment, but this is not mentioned in the provided text. The current text suggests the operator "chooses the plates exhibiting adequate growth and selects the isolated colonies to be processed assigning the automatic ID tasks" on a "digital plate", implying human-in-the-loop for key decisions.

Ask a specific question about this device