Search Filters

Search Results

Found 3 results

510(k) Data Aggregation

    K Number
    K232756
    Device Name
    Colibrí
    Manufacturer
    Date Cleared
    2023-12-27

    (110 days)

    Product Code
    Regulation Number
    866.1645
    Reference & Predicate Devices
    Why did this record match?
    Applicant Name (Manufacturer) :

    Copan WASP S.r.l.

    AI/MLSaMDIVD (In Vitro Diagnostic)TherapeuticDiagnosticis PCCP AuthorizedThirdpartyExpeditedreview
    Intended Use

    The Colibrí is an automated in vitro diagnostic specimen preparation system for use with WASPLab to prepare MALDI-TOF targets for the bioMérieux VITEK MS systems or Bruker MALDI Biotyper CA mass spectrometry systems for qualitative identification and microbial suspension for the bioMérieux VITEK 2 systems or Beckman Coulter MicroScan WalkAway Antimicrobial Susceptibility Testing (AST) systems for qualitative testing of isolated colonies of gram-negative and gram-positive bacterial species grown on solid culture media.

    The Colibrí is an automated pre-analytical processor that picks isolated colonies designated by the operator and uses a pipetting system to prepare MALDI-TOF MS (Matrix-Assisted Laser Desorption/lonization-Time of Flight Mass Spectrometry) target slides for bacterial identification and microbial suspension at known concentration for Antimicrobial Susceptibility Testing and purity assessment.

    The Colibrí software records the identity of each sample and its position on the target slide and communicates this information electronically to the MALDI-TOF MS analyzers.

    Bacterial suspensions for AST and purity plates are identified by barcode label.

    The Colibrí is intended for use by trained healthcare professionals in clinical laboratories in conjunction with other clinical and laboratory findings, including Gram staining, to aid in the diagnosis of bacterial infections.

    The Colibrí has not been validated for use in the identification or processing of yeast species, molds, Nocardia, or mycobacteria.

    Device Description

    The Colibrí is an instrument which automates the picking of selected colonies from plated media and prepares MALDI target slides for the bioMérieux VITEK MS systems or the Bruker MALDI Biotyper CA systems that are used in clinical laboratories for identification and differentiation of organisms grown on plated media by Matrix-Assisted Laser Desorption/Jonization Time-of Flight Mass Spectrometry (MALDI-TOF MS). The Colibri automates the preparation of microbial suspensions at known concentration for bioMérieux VITEK 2 systems and Beckman Coulter MicroScan WalkAway systems that are used in clinical laboratories for AST analyses. Moreover, the Colibrí is used for Purity Plates preparation for purity assessments.

    The Colibrí includes the following components:

    • Colibrí instrument and software with on-board pipetting system and nephelometer .
    • Colibrí Primary Tubes
    • Colibrí Spreader
    • Colibrí Daily Verification kit.

    Colibri is designed to be used in conjunction with the WASPLab device for culture plate incubation and image analysis. After appropriate plate incubation, the operator selects the colonies from a digital image of culture media plate streaked with microbiological human specimen, available through WebApp software, the WASPLab User Interface.

    The operator assigns the automatic ID or AST tasks to the isolated colonies to be processed. Then, the operator loads the plates in the Collbri where colonies are automatically picked, spotted on the target slide and overlayed with the matrix or suspended into the dedicated solution for the preparation of the microbial suspension for AST purposes (Secondary Tube).

    When used in conjunction with the bioMérieux VITEK MS systems, the Colibri can prepare the 48-spot target slides by performing the direct spotting of colonies. The calibrator used for quality control is manually applied by the operator at the end of the automated colony spotting. When used in conjunction with the Bruker MALDI Biotyper CA systems, the Colibri can prepare either reusable 48-spot or disposable 96-spot targets by performing the Direct Transfer Sample Procedure. The BTS used for quality control is manually applied by the operator at the end of the automated colony spotting.

    When used in conjunction with the bioMérieux VITEK 2 systems or the Beckman Coulter MicroScan WalkAway systems, the Colibri can prepare the microbial suspension at the proper concentration by direct colony suspension method. The onboard nephelometer allows the preparation of Secondary Tubes (AST suspensions) at the correct concentration and the Colibri Spreader is used for Purity Plates preparation.

    The Colibrí software records the identity of each sample and its position on the target slide and communicates this information electronically to the MALDI-TOF MS analyzers.

    The traceability of prepared Secondary Tube and Purity Plates is maintained by dedicated labels applications.

    Colibrí requires four different calibrations, one on the nephelometer, three on the cameras. None of these calibration activities require user intervention if not in terms of periodical cleaning of the mechanical component as described in the dedicated section of the User Manual. The Set-up calibration of nephelometer and camera units are performed during the device initial setup. Auto-calibration is performed at the end of the initial set-up and periodically during the preventive maintenance to check that all the mechanical references can be found inside the positioning tolerances, that the I/Os are responsive. Runtime calibration is performed during the normal usage to automatically check the proper functioning of the Colibrí.

    Colibrí requires a daily nephelometer verification to check the proper reading of suspensions at different turbidity values.

    AI/ML Overview

    The Colibrí device is an automated in vitro diagnostic specimen preparation system. The provided text describes the acceptance criteria and the study that proves the device meets these criteria for preparing microbial suspensions for Antimicrobial Susceptibility Testing (AST) using Beckman Coulter MicroScan WalkAway systems.

    Here's a breakdown of the requested information:

    1. A table of acceptance criteria and the reported device performance:

    The acceptance criteria are implicitly derived from the successful outcomes of the analytical studies. The performance is reported as the percentage of successful outcomes for each metric.

    MetricAcceptance Criteria (Implied)Reported Device Performance
    Preparation of Microbial Suspensions for ASTMicrobial concentration within acceptable limits:
    • E. coli ATCC 25922: 3-7 x 105 CFU/mL
    • Other bacteria: 2-8 x 105 CFU/mL | 98.5% of prepared suspensions had microbial concentration within acceptable limits.
    • E. coli: 100% (36/36)
    • Pseudomonas aeruginosa: 96.7% (29/30)
    • Staphylococcus aureus: 97.6% (41/42)
    • Enterococcus faecalis: 100% (30/30) |
      | AST Challenge Test (Agreement with Manual Preparation) | Essential Agreement (EA) of MICs: High agreement
      Category Agreement (CA): High agreement
      Discrepancies (vmj, maj): Low/none | Overall EA: 100% (1232/1232 evaluable MIC results within 1 two-fold dilution)
      Overall CA: 98.4% (4187/4254 SIR categorizations in agreement)
      Very Major discrepancy (vmj): 0
      Major discrepancy (maj): 0 |
      | Reproducibility Study | Best-case reproducibility: ≥95% (implied)
      Worst-case reproducibility: ≥89% (implied) | Best-case reproducibility: ≥99.8% (all panels combined)
      Worst-case reproducibility: ≥94.3% (all panels combined) |
      | Sample preparation for Quality Control | 100% of MIC values within CLSI/panel IFU QC range | 100% (all tested organisms and antimicrobial agents) |
      | Purity Plates Evaluation (Cross-contamination) | Absence of cross-contamination (100% monomicrobial growth) | 100% (453/453 Purity Plates showed monomicrobial growth) |

    2. Sample sizes used for the test set and the data provenance:

    • Preparation of Microbial Suspensions for AST:

      • Test Set Size: 132 microbial suspensions (36 E. coli, 30 Pseudomonas aeruginosa, 42 Staphylococcus aureus, 30 Enterococcus faecalis).
      • Data Provenance: Not explicitly stated (e.g., country of origin). The study involved three Colibrí instruments, suggesting internal validation. Retrospective or prospective is not specified, but the nature of the validation suggests prospective testing.
    • AST Challenge Test:

      • Test Set Size: Different species: Enterobacterales (n=50 isolates), Staphylococcus (n=20 isolates), Streptococcus (n=12 isolates), Enterococcus (n=18 isolates), non-fermenters (n=10 isolates). Each processed by three Colibrí instruments, yielding varying numbers of MIC results and SIR categorizations across different panels (e.g., 2454 total MIC results for Enterobacterales on NM-NF50 panel).
      • Data Provenance: Not explicitly stated (e.g., country of origin). The study design implies prospective testing within the manufacturer's validation process.
    • Reproducibility Study:

      • Test Set Size: 9 gram-positive and 9 gram-negative strains, processed on 3 Colibrí instruments over 3 days, with each condition tested in triplicate (total of 27 replicates for each strain-antimicrobial agent combination).
      • Data Provenance: Not explicitly stated (e.g., country of origin, retrospective or prospective). Implied prospective.
    • Sample preparation for Quality Control:

      • Test Set Size: CLSI-recommended reference strains (Escherichia coli ATCC 25922, Pseudomonas aeruginosa ATCC 27853, Staphylococcus aureus ATCC 29213, Enterococcus faecalis ATCC 29212). The "No. MIC within QC range" indicates a total of 30 or 36 or 42 tests for each drug-organism combination, per three instruments.
      • Data Provenance: Not explicitly stated. Implied prospective.
    • Purity Plates Evaluation:

      • Test Set Size: 453 purity plates (150 from AST Challenge, 162 from AST Reproducibility, 141 from Quality Control studies).
      • Data Provenance: Not explicitly stated. Implied prospective.

    3. Number of experts used to establish the ground truth for the test set and the qualifications of those experts:

    • The ground truth for AST results (MICs and SIR categories) generally refers to the results obtained from a reference method. In this case, "Manual suspension was used as comparative method" for the AST Challenge Test. This implies that manually prepared suspensions, processed by the MicroScan WalkAway, served as the reference standard.
    • The text does not specify the number of experts or their qualifications for establishing this manual ground truth. It mentions that three different technicians operated the Colibrí machines, but it doesn't detail the personnel for the manual comparative method or for interpreting the results as ground truth beyond the "FDA-Recognized Antimicrobial Susceptibility Test Interpretive Criteria" and CLSI guidelines.

    4. Adjudication method (e.g. 2+1, 3+1, none) for the test set:

    • The document does not describe any expert adjudication process for the test set results. The comparison is made against a "manual result" (ground truth). The discrepancies (vmj, maj, min) are simply categorized and reported, implying a direct comparison without further expert review for resolving initial disagreements.

    5. If a multi-reader multi-case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance:

    • No, an MRMC study was not conducted. This device (Colibrí) is an automated system for sample preparation and does not involve human "readers" or "AI assistance" in the typical sense of image analysis for diagnosis. Its role is to automate a laboratory process, and the performance is measured against reference methods, not human interpretation.

    6. If a standalone (i.e. algorithm only without human-in-the-loop performance) was done:

    • The studies presented are primarily standalone (algorithm only without human-in-the-loop performance) in terms of the Colibrí device's automated functions. The device picks colonies, prepares suspensions, and records data automatically. The performance metrics (inoculum density, MIC accuracy, reproducibility, purity) assess the device's output against established standards and manual methods.
    • While an operator designates colonies for picking, the act of preparation itself is automated and evaluated for its accuracy. The "manual suspension" used for comparison acts as the reference for the "algorithm only" performance of the Colibrí in producing the suspension.

    7. The type of ground truth used (expert consensus, pathology, outcomes data, etc):

    • The ground truth for the analytical studies combines reference methods/standards and established guidelines:
      • For microbial suspension concentration: Viable cell count (CFU/mL) against CLSI and FDA guidelines.
      • For AST results: Manual suspension preparation as the comparative method, and comparison of MICs and SIR categories against FDA-Recognized Antimicrobial Susceptibility Test Interpretive Criteria and CLSI guideline M07.
      • For reproducibility: Comparison to the "mode result" (most frequent MIC value) and established reproducibility criteria (e.g., within one doubling dilution).
      • For Quality Control: CLSI-recommended QC ranges and MicroScan panel IFU values.
      • For Purity Plates: Visual assessment (implied) to confirm monomicrobial growth.

    8. The sample size for the training set:

    • This document describes performance validation studies for a medical device (Colibrí), not a machine learning model. Therefore, there is no "training set" in the context of data used to train an AI algorithm. The Colibrí is an automated instrument with pre-programmed functions, not a learning algorithm that requires a training dataset.

    9. How the ground truth for the training set was established:

    • As explained above, there is no training set for this device in the context of AI/ML.
    Ask a Question

    Ask a specific question about this device

    K Number
    K220546
    Device Name
    Colibrí System
    Manufacturer
    Date Cleared
    2022-10-05

    (222 days)

    Product Code
    Regulation Number
    866.1645
    Reference & Predicate Devices
    Why did this record match?
    Applicant Name (Manufacturer) :

    Copan WASP S.r.l.

    AI/MLSaMDIVD (In Vitro Diagnostic)TherapeuticDiagnosticis PCCP AuthorizedThirdpartyExpeditedreview
    Intended Use

    The Collbri™ System is an in vitro diagnostic device comprised of the Collbri™ Preparation Station for use with the bioMérieux VITEK® MS or Bruker MALDI Biotyper® CA mass spectrometry systems for qualitative identification and with the bioMérieux VITEK® 2 Antimicrobial Susceptibility Testing (AST) system for qualitative testing of isolated colonies of gram-negative and gram-positive bacterial species grown on solid culture media. The Collbri™ System is a semi-automated pre-analytical processor that picks isolated colonies designated by the operator and uses a pipetting system to prepare MALDI-TOF MS (Matrix-Assisted Laser Desorption/Jonization-Time of Flight Mass Spectrometry) target slides for bacterial identification and microbial suspension at known concentration for Antimicrobial Susceptibility Testing and purity assessment.

    The Colibrí software records the identity of each sample and its position on the target slide and communicates this information electronically to the MALDI-TOF MS analyzers.

    Bacterial suspensions for AST and purity plates are identified by barcode label.

    The Colibr™ System is intended for use by trained healthcare professionals in clinical laboratories in conjunction with other clinical and laboratory finding Gram staining, to aid in the diagnosis of bacterial infections.

    The Collbri™ System has not been validated for use in the identification or processing of yeast species, Mocardia, or mycobacteria.

    Device Description

    The Copan Colibrí System is designed to be used as an accessory of the downstream MALDI-TOF MS and antimicrobial susceptibility testing (AST) analyzers automating various manual steps in the workflow for the preparation of samples for the identification of isolated colonies and for AST of isolated colonies of gram-negative and gram-positive bacterial species grown on solid culture media.

    The Colibrí System automates the preparation of MALDI target slides for the bioMérieux VITEK MS or the Bruker MALDI Biotyper CA System that are used in clinical laboratories for identification (ID) of organisms grown on plated media by Matrix-Assisted Laser Desorption/Jonization Time-of Flight Mass Spectrometry (MALDI-TOF MS). The Colibri System automates the preparation of microbial suspensions at known concentration for bioMérieux VITEK 2 System that is used in clinical laboratories for AST analyses. Moreover, the Colibri System is used for Purity Plates preparation for purity assessments.

    The Colibrí System comprises the Colibrí Vision System and Colibrí Preparation Station hardware modules and pipette tips, Primary Tubes, Spreader and nephelometer Verification Kit as consumables. After appropriate plate incubation, the operator using the graphical User Interface (Image Reading Interface) chooses the plates exhibiting adequate growth and selects the isolated colonies to be processed assigning the automatic ID or AST tasks. By using the Colibrí Vision System, specific colonies to be picked are designated by the operator on a digital plate. The Operator manually loads the plates in the Colibri Preparation Station where colonies are automatically picked, spotted on the target slide and overlayed with the matrix or suspended into the dedicated solution for the preparation of the microbial suspension for AST purposes (Secondary Tube).

    When used in conjunction with the bioMérieux VITEK MS, the Colibrí System can prepare the 48spot target slides by performing the direct spotting of colonies. The calibrator used for quality control is manually applied by the operator at the end of the automated colony spotting. When used in conjunction with the Bruker MALDI Biotyper CA System, the Colibrí System can prepare either reusable 48-spot or disposable 96-spot targets by performing the Direct Transfer Sample Procedure. The BTS used for quality control is manually applied by the operator at the automated colony spotting.

    When used in conjunction with the bioMérieux VITEK 2, the Colibrí System can prepare the microbial suspension at the proper concentration by direct colony suspension method. The onboard nephelometer allows the preparation of Secondary Tubes (AST suspensions) at the correct concentration and the Colibrí Spreader is used for Purity Plates preparation.

    Copan WASP S.r.l., Traditional 510(k)- Colibrí System

    The Colibrí software records the identity of each sample and its position on the target slide and communicates this information electronically to the MALDI-TOF MS analyzers.

    The traceability of prepared Secondary Tube and Purity Plates is maintained by dedicated labels applications.

    Colibri System requires four different calibrations, one on the nephelometer, three on the cameras. None of these calibration activities require user intervention if not in terms of periodical cleaning of the mechanical component as described in the dedicated section of the User Manual. The Set-up calibration of nephelometer and camera units positioned on the Colibrí Vision System and on the Colibrí Preparation Station are performed during the device initial setup. Auto-calibration is performed at the end of the initial set-up and periodically during the preventive maintenance to check that, in the Collbrí Preparation all the mechanical references can be found inside the positioning tolerances, that the I/Os are responsive. Run-time calibration is performed during the normal usage to automatically check the proper functioning of the Colibrí Vision System and the Colibrí Preparation Station.

    Colibrí System requires a daily nephelometer verification to check the proper reading of suspensions at different turbidity values.

    AI/ML Overview

    The provided text describes the performance data for the Colibrí System, an in vitro diagnostic device. The acceptance criteria and performance are primarily focused on its ability to accurately prepare microbial suspensions for Antimicrobial Susceptibility Testing (AST) and for MALDI-TOF MS identification, compared to manual methods.

    Here's an attempt to extract the requested information. Please note that the document is a 510(k) summary, which often provides summarized performance data rather than detailed study protocols. Therefore, some information might be explicitly stated as "not applicable" or inferred based on common practices for such device clearances.

    1. A table of acceptance criteria and the reported device performance

    The document outlines several analytical studies. While explicit "acceptance criteria" are not always presented as target percentages, the performance results are given, implying that these results met the internal pre-defined acceptance thresholds for substantial equivalence.

    Performance MetricAcceptance Criteria (Implied/Direct)Reported Device Performance
    Nephelometer Calibration VerificationAccurate preparation of microbial suspensions at specific concentrations (0.25, 0.5, 1.0, 2.0, 3.0 McFarland) with acceptable accuracy and nominal microbial content (e.g., 1-2 × 10⁸ CFU/mL for 0.5 McFarland E. coli).Overall, 100% of suspensions contained the correct concentration of bacteria.
    Pipettor Trueness and PrecisionTrueness and reproducibility for four volumes (50uL, 100uL, 500uL, 900uL) within acceptance criteria.Trueness and reproducibility varied according to the volume under testing but always within the acceptance criteria.
    E. coli Suspensions Preparation VerificationCorrect management of Primary Tubes according to turbidity value, and expected number of colonies based on McFarland standard (e.g., 1-2 × 10⁸ CFU/mL for 0.5 McFarland E. coli).All Primary Tubes were correctly managed by Colibrí System according to the turbidity value. 100% of suspensions over the entire working range contained the expected number of colonies.
    Colony Picking Accuracy & Microbial Suspension Preparation (Purity Check)Accurate picking of designated colonies from culture plates without contamination from other microorganisms, demonstrating monomicrobial suspensions. High percentage of prepared suspensions with microbial concentration within acceptable limits.100% of colonies designated by the operator were picked correctly by the Colibrí System (both whole plates and bi-plates). 100% of Purity Plates showed no evidence of microbial contamination. The percentage of prepared suspension with microbial concentration within the acceptable limits was 99.2%, and for each instrument, the result was always >98%. No statistically significant difference among instruments.
    AST Challenge Test (Agreement with Manual Preparation)High agreement (Essential Agreement and Category Agreement) with MICs obtained by VITEK 2 using manual sample preparation. Target percentages for EA and CA for comparability (typically >90% or >95% for Essential Agreement and >90% for Category Agreement and low major/very major error rates as per CLSI guidelines).Overall Essential Agreement (EA) of evaluable MIC results was >99.9%. Overall Category Agreement (CA) was 99.3%. Notably low error rates: 0 Very Major Errors, 1 Major Error (for Cefepime, Non-fermenters), and 49 Minor Errors across all tested combinations. 1882/1883 evaluable MIC results were within one doubling dilution of the comparator method. 5947/5991 SIR categorizations were in agreement.
    Reproducibility (MIC Results)MIC results considered reproducible if they fell within one doubling dilution from the modal value of each combination.Generally very high reproducibility. The "worst case" percentages, representing MIC values that were not within one doubling dilution, were generally very low (e.g., lowest was 96.3% for Tobramycin, Instrument 1/3, and Ciprofloxacin, Instrument 2; lowest combined was 97.5% for Tobramycin and Oxacillin), meaning that a very high percentage were within the acceptable limit. The values provided for "Best case" were often 100%.
    Purity Plate Growth100% of purity plates correctly processed and providing evidence of monomicrobial suspensions, demonstrating no cross-contamination.2,364/2,364 (100%) purity plates were correctly processed, demonstrating that the Colibrí Preparation Station prepares monomicrobial suspensions and prevents cross-contamination.
    QC Sample Preparation100% in-range MIC values for QC organisms compared to established ranges. Purity of all suspensions confirmed.100% in-range MIC values for QC organisms. Purity of all suspensions confirmed by Purity Plates.

    2. Sample sizes used for the test set and the data provenance

    • Nephelometer Calibration Verification: 300 suspensions were prepared (20 suspensions for each of 5 concentrations, across 3 operators and 3 Colibrí systems, though the calculation isn't directly 2053*3).
    • Pipettor Trueness and Precision: 10 measurements for each of 4 volumes, across 3 Colibrí System pipettors (10 * 4 * 3 = 120 measurements).
    • E. coli Suspensions Preparation Verification: The exact number of suspensions isn't provided, but it states "A variable number of colonies was selected... to create different suspensions... Three Colibrí Systems run by three different operators were included."
    • Colony Picking and Microbial Suspensions for AST: 6 bacterial species (3 Gram-Negative and 3 Gram-Positive) grown in 2 polymicrobial mixtures on different culture media.
    • AST Challenge Test:
      • Total Tested: 5991 (across various antibiotics and organism groups within the challenge test).
      • Evaluated MIC results: 1883 evaluable MIC results.
      • Organism groups: Enterobacterales (n=62 strains), Staphylococcus (n=16 strains), Streptococcus (n=30 strains), Enterococcus (n=16 strains), and non-fermenters (n=32 strains).
      • Strains: Both susceptible and resistant strains, exhibiting a range of on-scale MIC values.
      • Media: Trypticase Soy Agar + 5% Sheep Blood, MacConkey Agar, and Columbia agar + 5% sheep blood.
      • Incubation times: Varied (e.g., 14h, 24h for Enterobacterales/Non-fermenters, 18h for Staphylococcus/Enterococcus/Streptococcus).
      • Data Provenance: Not explicitly stated (e.g., country of origin, retrospective/prospective). However, given it's a 510(k) submission for a medical device manufacturer (Copan WASP S.r.l., Italy), it's highly likely to be internal, prospective studies conducted at their facilities or collaborator sites.
    • Reproducibility Study: Each microorganism was tested with the appropriate antibiotic panel, with each condition tested in triplicate, for a total of 81 replicates for each combination strain-antimicrobial agent across three Colibrí Systems and three operators over three days.
    • Purity Plate Growth: 2,364 purity plates.
    • QC Sample Preparation: Conducted daily at the beginning of the working session on each instrument involved in the Analytical Studies.

    3. Number of experts used to establish the ground truth for the test set and the qualifications of those experts

    • This information is not explicitly provided in the document.
    • For the AST Challenge Test, the "ground truth" (or comparator method) was the MICs obtained by the bioMérieux VITEK 2 using manual sample preparation, interpreted according to FDA-Recognized Antimicrobial Susceptibility Test Interpretive Criteria. This implies the ground truth relies on established, validated laboratory methods and interpretations, rather than subjective expert consensus.
    • For other analytical studies (e.g., nephelometer accuracy, pipettor precision), the ground truth generally relies on quantitative measurements using calibrated instruments and standard protocols (e.g., viable cell counts for McFarland turbidity verification, gravimetric measurements for pipettor accuracy).

    4. Adjudication method for the test set

    • This is not applicable in the context of this device's performance validation. The device automates a pre-analytical step. The performance is assessed by comparing its output (prepared microbial suspensions) to a reference method (manual preparation for VITEK 2). It's not a diagnostic AI system requiring expert adjudication of image interpretations or clinical diagnoses.

    5. If a multi reader multi case (MRMC) comparative effectiveness study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance

    • No, an MRMC comparative effectiveness study was not done. This type of study is relevant for AI systems that assist human readers in interpreting medical images (e.g., radiology AI). The Colibrí System is a laboratory automation device for preparing samples. It does not involve human "readers" of AI outputs in a diagnostic context that would require such a study design.

    6. If a standalone (i.e. algorithm only without human-in-the-loop performance) was done

    • The performance data are essentially standalone (algorithm/device only) in terms of its ability to perform the physical process of colonial picking and suspension preparation. The device's output is then fed into other IVD analyzers (VITEK 2, bioMérieux VITEK MS, Bruker MALDI Biotyper CA System) for identification and AST.
    • The comparison in the AST Challenge Test is between the Colibrí System's automated preparation and manual preparation which is the existing standard. So, it's comparing automated device output to a manual, human-executed process, where the subsequent analysis (VITEK 2) is the same. The data provided (EA, CA, error rates) represent the performance of the Colibrí system's prepared samples, which is a form of standalone performance.

    7. The type of ground truth used (expert consensus, pathology, outcomes data, etc.)

    • Comparatative Ground Truth/Reference Method: For the AST Challenge Test, the ground truth was derived from the MICs obtained by the bioMérieux VITEK 2 using manual sample preparation, interpreted according to FDA-Recognized Antimicrobial Susceptibility Test Interpretive Criteria. This is a well-established and standardized laboratory reference method.
    • Quantitative Ground Truth: For other analytical studies, the ground truth was based on quantitative laboratory measurements, such as:
      • Viable cell counts (CFU/mL) for confirming bacterial concentration for nephelometry.
      • Gravimetric measurements for pipetting accuracy.
      • Visual inspection of purity plates and accepted microbiological methods to ensure monomicrobial suspensions and absence of contamination.

    8. The sample size for the training set

    • The document describes performance validation studies, not product development or AI model training. Therefore, information about a "training set" (in the context of machine learning) is not applicable or provided. This device automates a physical process, not a machine learning model that needs training data in the traditional sense. The "training" of the device likely refers to physical calibration and quality control.

    9. How the ground truth for the training set was established

    • As per point 8, the concept of a "training set" for an AI/machine learning model is not applicable to the description of this device's validation. The device's mechanics and software are validated against established engineering and microbiology standards.
    Ask a Question

    Ask a specific question about this device

    K Number
    K193138
    Device Name
    Colibri System
    Manufacturer
    Date Cleared
    2021-12-27

    (775 days)

    Product Code
    Regulation Number
    866.3378
    Reference & Predicate Devices
    N/A
    Why did this record match?
    Applicant Name (Manufacturer) :

    COPAN WASP S.r.l.

    AI/MLSaMDIVD (In Vitro Diagnostic)TherapeuticDiagnosticis PCCP AuthorizedThirdpartyExpeditedreview
    Intended Use

    The Colibrí System is an in vitro diagnostic device comprised of the Colibrí Vision System and Collbrí Preparation Station for use with the bioMérieux VITEK MS or Bruker MALDI Biotyper CA mass spectrometry systems for qualitative identification of isolated colonies of Gram-negative bacterial species grown on solid culture media. The Collbri System is a semi-automated pre-analytical processor that picks isolated by the operator and uses a pipetting system to prepare MALDI-TOF MS (Matrix-Assisted Laser Desorption/Ionization-Time Of Flight Mass Spectrometry) target slides. The Colibri software records the identity of each sample and its position on the target slide and communicates this information electronically to the MALDI-TOF MS analyzer.

    The Colibrí System is intended for use by trained healthcare professionals in clinical laboratories in conjunction with other clinical and laboratory finding Gram staining, to aid in the diagnosis of bacterial infections.

    The Colibrí System has not been validated for use in identification of yeast species.

    Device Description

    The Copan Colibrí System is designed to be used as an accessory of the downstream MALDI-TOF analyzers automating various manual steps in the workflow for the preparation of samples for the identification of isolated colonies of microorganisms cultured from the human body.

    The Colibrí System automates the preparation of MALDI target slides for the bioMérieux VITEK MS or the Bruker MALDI Biotyper CA System that are used in clinical laboratories for identification and differentiation of organisms grown on plated media by Matrix-Assisted Laser Desorption/Ionization Time-of Flight Mass Spectrometry (MALDI-TOF MS). The system comprises the Colibrí Vision System and Colibrí Preparation Station and pipette tips as consumables. After appropriate plate incubation, the operator using the graphical User Interface (Image Reading Interface) chooses the plates exhibiting adequate growth and selects the isolated colonies to be processed assigning the automatic ID tasks. By using the Collbrí Vision System, specific colonies to be picked are designated by the operator on a digital plate. The Operator manually loads the plates in the Colibri Preparation Station where colonies are automatically picked, spotted on the target slide and overlayed with the matrix.

    When used in conjunction with the bioMérieux VITEK MS, the Colibrí System can prepare the 48spot target slides by performing the direct spotting of colonies. The calibrator used for quality control is manually applied by the operator at the end of the automated colony spotting. When used in conjunction with the Bruker MALDI Biotyper CA System, the Colibrí System can prepare either reusable 48-spot or disposable 96-spot targets by performing the Direct Transfer Sample Procedure. The BTS used for quality control is manually applied by the operator at the and of the automated colony spotting.

    The Colibrí software records the identity of each sample and its position on the target slide and communicates this information electronically to the MALDI-TOF MS analyzers.

    Colibri System requires three different calibrations. None of these calibration activities require user intervention if not in terms of periodical cleaning of the mechanical component as described in the dedicated section of the User Manual. Set-up calibration is performed during the device initial setup for the camera units positioned on the Colibrí Vision System and on the Colibrí Preparation Station. Auto-calibration is performed at the end of the initial set-up and periodically during the preventive maintenance to check that, in the Colibri Preparation Station, all the mechanical references can be found inside the positioning tolerances, that the I/Os are responsive. Run-time calibration is performed during the normal usage to automatically check the proper functioning of the Colibrí Vision System and the Colibri Preparation Station.

    AI/ML Overview

    The provided text describes the performance data for the Colibrí System, an in vitro diagnostic device. The acceptance criteria are implicitly defined by the performance observed in various analytical studies, with the goal of demonstrating substantial equivalence to predicate devices for qualitative identification of isolated colonies of Gram-negative and Gram-positive bacterial species.

    Here's an analysis of the acceptance criteria and study proving the device meets them:

    1. Table of Acceptance Criteria (Implicit) and Reported Device Performance

    The document does not explicitly state pre-defined acceptance criteria in a dedicated table with specific thresholds. However, based on the studies conducted, the implicit acceptance criteria appear to be:

    • High agreement with expected identification (ground truth).
    • Absence of wrong identifications (false positives).
    • Maintenance of performance across different conditions (e.g., positional effects, culture age, different MALDI-TOF MS systems and target types, different operators, different Colibrí systems).
    • Comparable performance to manual preparation.
    • No cross-contamination.

    Here's a table summarizing the reported device performance, which serves as the evidence that these implicit criteria were met. The percentages below represent the agreement with the expected ID (ground truth) for high-confidence results where available, or overall agreement as calculated by the study.

    Study TypeImplicit Acceptance CriteriaReported Device Performance and Notes (Colibrí System)
    Colony Picking for Microbial IdentificationHigh accuracy (close to 100%) in picking designated colonies and correct identification with high confidence, with no wrong identifications. Comparable performance across different MALDI-TOF MS systems.VITEK MS: 100% correct colony picking (no wrong colony picked).Overall 98.4% agreement with expected strain identity (1368/1390 spots with high confidence or low discrimination).0 wrong IDs.Bruker MALDI Biotyper CA: 100% correct colony picking.Overall 90.8% agreement with expected strain identity (1534/1690 spots with high confidence or low discrimination).0 wrong IDs.Noted lower proportion of concordant results for Gram-positive species, but no incorrect identifications. Recommended manual repeat testing for low/no ID results.
    Positional Effect StudyNo positional effect observed; consistent identification accuracy across all target slide positions. No wrong identifications.VITEK MS: 100% agreement for E. coli (432/432), 99.8% for S. aureus (431/432).0 wrong IDs.No positional effect detected.Bruker MALDI Biotyper CA (US IVD 48 Spot): 99.8% agreement for E. coli (431/432), 96.8% for S. aureus (418/432).0 wrong IDs.No positional effect detected.Bruker MALDI Biotyper CA (MBT Biotarget 96 US IVD): 99.9% agreement for E. coli (845/846), 95.7% for S. aureus (810/846).0 wrong IDs.No positional effect detected.
    Inclusivity StudyHigh agreement with expected identification for "on-panel" species, with no false identifications. Performance should be comparable to manual preparation.VITEK MS: Overall 97.2% agreement (334/392 high confidence; 47/392 low discrimination included in agreement, total 381/392) with expected ID.0 wrong IDs.Manual preparation agreement: 91.6% (combined for high confidence and low discrimination). Colibrí system performed comparably or better.Bruker MALDI Biotyper CA (US IVD 48 Spot): Overall 93.2% agreement (436/468 high confidence) with expected ID.0 wrong IDs.Manual preparation agreement: 97.2% (combined for high confidence and low discrimination). Performance lower for Gram-positive species.Bruker MALDI Biotyper CA (MBT Biotarget 96 US IVD): Overall 85.7% agreement (401/468 high confidence) with expected ID.0 wrong IDs.Manual preparation agreement: 88.0% (combined for high confidence and low discrimination). Performance lower for Gram-positive species.For both Bruker systems, device labeling will recommend manual repeat testing for low/no ID results, consistent with predicate IFU.
    Specificity Study"Off-panel" species should yield no identification, or results consistent with their "off-panel" status, with no false positives for "on-panel" species.VITEK MS: 100% agreement (20/20) that "off-panel" organisms yielded no identification or uninterpretable results.0 false positives for "on-panel" species.Bruker MALDI Biotyper CA: 100% agreement (20/20) that "off-panel" organisms yielded no identification or uninterpretable results.0 false positives for "on-panel" species.
    Reproducibility StudyHighly repeatable results across multiple Colibrí systems, operators, and replications, with high agreement to expected ID and no wrong IDs.VITEK MS: 99.9% agreement (1799/1800) with expected ID (including low discrimination results that are considered acceptable by the VITEK MS system).0 wrong IDs.Bruker MALDI Biotyper CA: 88.1% agreement (1585/1800) with high confidence ID.0 wrong IDs.Lower agreement for Gram-positive species (76.7% high confidence) compared to Gram-negative (99.4% high confidence), consistent with other studies. Device labeling to recommend manual repeat testing for low/no ID Gram-positive results.
    Cross-Contamination StudiesNo false-positive results due to contamination of adjacent spots.VITEK MS: "On-panel" species: 99.3% correct identification (284/286) with high confidence."Off-panel" species: 100% yielded no identification (286/286).No false positives observed.Bruker MALDI Biotyper CA (US IVD 48 Spot): "On-panel" species: 95.0% correct identification (190/200 high confidence, 10/200 low confidence included in performance evaluation)"Off-panel" species: 100% yielded no identification (200/200).No false positives observed.Bruker MALDI Biotyper CA (MBT Biotarget 96 US IVD): "On-panel" species: 85.3% correct identification (122/143 high confidence, 11/143 low confidence included in performance evaluation)"Off-panel" species: 100% yielded no identification (143/143).Lack of ID not due to cross-contamination, but lower performance for Gram-positives for this system.
    Colony Stability StudyMaintain expected identification performance across various culture ages and media types.VITEK MS: 99.8% agreement overall (575/576) at various time points (18h, 24h, 48h, 72h depending on media type).No false identifications.Bruker MALDI Biotyper CA (MBT Biotarget 96 US IVD): Generally good agreement for Gram-negative species regardless of culture medium or incubation duration. Lower agreement for Gram-positive species.No incorrect identification results reported.Specific note for B. pertussis on Bordet Gengou Agar showing decreased high confidence scores after prolonged ambient temperature holding.
    Spot Stability Prior To and After Matrix DepositionStable identification performance when testing is delayed (before and after matrix application).VITEK MS: Colonies stable up to 60 mins without matrix.Targets stable for 48h at room temp on Colibrí deck, and 72h in original box.Identification performance not different from standard conditions.Bruker MALDI Biotyper CA: Colonies stable up to 60 mins without matrix on both target types.Targets stable for 24h at room temp when held on Colibrí deck and lab bench.Lower agreement for Gram-positive species with 96-spot disposable format noted.

    2. Sample Sizes Used for the Test Set and Data Provenance

    • Test Set (Analytical Studies):

      • Colony Picking: 1390 spots (VITEK MS) and 1690 spots (Bruker MALDI Biotyper CA).
      • Positional Effect: 864 spots (VITEK MS), 864 spots (Bruker MALDI Biotyper CA US IVD 48 Spot), 1692 spots (Bruker MALDI Biotyper CA MBT Biotarget 96 US IVD).
      • Inclusivity: 392 spots/strains (VITEK MS) and 468 spots/strains for each Bruker MALDI Biotyper CA system (US IVD 48 Spot and MBT Biotarget 96 US IVD).
      • Specificity: 20 spots (VITEK MS) and 20 spots (Bruker MALDI Biotyper CA).
      • Reproducibility: 1800 spots (VITEK MS) and 1800 spots (Bruker MALDI Biotyper CA).
      • Cross-Contamination: 572 spots (VITEK MS) and 686 spots (Bruker MALDI Biotyper CA total for both systems).
      • Colony Stability: 576 spots (VITEK MS) and 1440 spots (Bruker MALDI Biotyper CA).
    • Data Provenance: The document generally indicates "on-panel" and "off-panel" strains, often referring to common clinical isolates in the US. No explicit country of origin for the clinical samples/strains themselves is stated beyond generic "human specimens" or "commonly isolated Gram-positive and Gram-negative species in the US". The studies were conducted by a single operator (Inclusivity, Specificity, Cross-Contamination, Colony Stability) or two operators (Reproducibility) on multiple Colibrí systems (Reproducibility, Colony Picking, Inclusivity, Cross-Contamination) in laboratory settings. The studies are prospective as they are designed experiments to evaluate the device performance.

    3. Number of Experts Used to Establish the Ground Truth for the Test Set and Their Qualifications

    • The document implies that the ground truth for identification was established by the expected strain identity of the stock cultures used in the analytical studies. This means the identities of the bacterial strains were known beforehand (e.g., ATCC strains or well-characterized lab strains).
    • For the performance results themselves, identification was based on the output of the VITEK MS or Bruker MALDI Biotyper CA systems, which generate confidence values or log scores. The interpretation of these scores (e.g., what constitutes a "correct single choice") is intrinsic to the performance specifications of these established MALDI-TOF MS systems.
    • No human experts (e.g., radiologists) were used to establish the ground truth for these microbial identification studies. The "ground truth" is the presumed identity of the bacterial strain used in the experiment.

    4. Adjudication Method for the Test Set

    • Not applicable in the conventional sense. Since the ground truth for the test set was the expected strain identity of known bacterial cultures, there was no need for human expert adjudication to resolve discrepancies in interpretation. The output of the MALDI-TOF MS system for each sample was compared directly to the known identity of the inoculated strain. Discrepancies (low confidence, no ID, wrong ID) were noted and analyzed against the overall performance.

    5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was Done

    • No. An MRMC study is typically performed for image-based diagnostic aids where multiple human readers interpret cases with and without AI assistance. This study involves microorganism identification, a laboratory process, not direct human interpretation of images for diagnosis.
    • The study compares the performance of the automated Colibrí System to manual preparation (the standard of care, as listed in the predicate device comparison tables), which is a different type of comparison than human readers with/without AI assistance.
    • Effect size of human reader improvement: Not applicable, as no human reader study was conducted. The comparison is between an automated sample preparation method (Colibrí) and a manual sample preparation method for laboratory identification.

    6. If a Standalone (i.e. algorithm only without human-in-the loop performance) was Done

    • Yes, indirectly. The Colibrí System is a semi-automated pre-analytical processor. While an operator designates the colony, the picking, spotting, and matrix application are automated.
    • The performance metrics provided (e.g., % agreement, no wrong ID) are for the Colibrí System's ability to prepare samples for subsequent MALDI-TOF MS analysis, which then generates the identification result. The Colibrí software records location and transmits information to the MALDI-TOF MS analyzer.
    • The study evaluated the effectiveness of the automated sample preparation step compared to manual preparation, which directly impacts the downstream MALDI-TOF MS identification. However, the identification itself is performed by the MALDI-TOF MS analyzer, not the Colibrí system's "algorithm" in isolation, except for tracking sample positions. The Colibrí's key "algorithm" is in its vision system and robotic picking/spotting, which impacts the quality of the sample for the MALDI-TOF MS.

    7. The Type of Ground Truth Used

    • Expected Strain Identity: The ground truth for the analytical studies was the known, expected identity of the bacterial strains used. These were presumably well-characterized laboratory strains (e.g., ATCC cultures) with a confirmed identity. This is a highly controlled form of ground truth.

    8. The Sample Size for the Training Set

    • Not explicitly stated in terms of a "training set" for an AI/machine learning model used within Colibrí's core functions. The Colibrí System is described as a "semi-automated pre-analytical processor" that uses "a pipetting system" and "Colibrí software records the identity of each sample and its position". It's unclear if the "Colibrí Vision System" employs complex machine learning that would require a dedicated training set beyond basic image processing for colony detection and localization.
    • The document implies that the system is automating a manual process, meaning its "training" pertains more to engineering and calibration rather than a machine learning model for diagnosis. The performance studies focus on the system's accuracy in physical manipulation and compatibility with existing MALDI-TOF MS systems.

    9. How the Ground Truth for the Training Set Was Established

    • Not applicable as a distinct ML training set is not described. If the "Colibrí Vision System" used internal algorithms that were "trained" to recognize colony morphology for picking, the ground truth would likely have been established by manually annotating colonies on images or physical plates as "pickable" or "not pickable" based on expert microbiological judgment, but this is not mentioned in the provided text. The current text suggests the operator "chooses the plates exhibiting adequate growth and selects the isolated colonies to be processed assigning the automatic ID tasks" on a "digital plate", implying human-in-the-loop for key decisions.
    Ask a Question

    Ask a specific question about this device

    Page 1 of 1