(205 days)
VITEK® MS is a mass spectrometry system using matrix-assisted laser desorption/ionization time of flight mass spectrometry (MALDI-TOF MS) for the identification of microorganisms cultured from human specimens.
The VITEK® MS system is a qualitative in vitro diagnostic device indicated for use in conjunction with other clinical and laboratory findings to aid in the diagnosis of bacterial, yeast and mould infections.
The VITEK® MS v3.0 system is a system consisting of kit reagents (VITEK® MS-CHCA, VITEK® MS-FA, VITEK® MS Mycobacterium/Nocardia Kit, VITEK® MS Mould Kit), VITEK® MS-DS target slides, and the VITEK® MS (original equipment VITEK® MS Prep Station, Knowledge Base v3.2.0, software, and the VITEK" MS (original eq
This document focuses on the VITEK® MS system, a mass spectrometry system for microorganism identification, and its performance characteristics. It outlines the acceptance criteria and details of the study conducted to prove the device meets these criteria.
1. Acceptance Criteria and Reported Device Performance
The acceptance criteria are implicitly defined by the reported performance targets which show very high agreement rates for identification. The primary performance metric is the "Total Correct Genus ID (One Choice and Low Discrimination)".
Here's a table summarizing criteria derived from the reported performance, reflecting typical expectations for such devices, and the observed performance for different organism groups.
| Organism Group | Acceptance Criteria (Implied) - Total Correct Genus ID | Reported Device Performance (Total Correct Genus ID) |
|---|---|---|
| Gram-positive bacteria | >95% | 95.8% (3594/3750) |
| Gram-negative bacteria | >95% | 95.2% (5788/6079) |
| Gram-negative bacteria (Brucella only) | >90% | 91.7% (220/240) |
| Yeasts | >95% | 96.8% (1316/1360) |
| Moulds | >90% | 92.7% (1398/1508) |
| Mycobacterium | >95% | 96.5% (777/805) |
| Nocardia | >95% | 97.9% (374/382) |
| Overall (All Organisms Total) | >95% | 95.4% (13247/13884) |
Note: The reported performance also details "One Choice Correct" and "Low Discrimination Correct" breakouts, as well as discordant and no-ID rates, providing a comprehensive view of the device's accuracy. For example, for "All" clinical bacteria and yeast isolates tested, the agreement with reference identification was 98.8% (4189/4241).
2. Sample Size Used for the Test Set and Data Provenance
Test Set Sample Size:
- Overall Performance (Routine fresh and stock strains): 13,884 samples across various organism groups.
- Clinical Studies (Clinical isolates): 4,241 test results.
- Brucella specific testing (ATCC): 240 strains for clinical trial, 180 for reproducibility, 45 for challenge testing.
Data Provenance:
The data was collected from:
- Routine fresh and stock strains from patient cultures in clinical microbiology laboratories and bioMérieux laboratories in the United States and France. This indicates a mix of prospective and retrospective (stock strains) data and geographic diversity.
- Clinical isolates were tested at "one external clinical trial site and one internal site" for bacteria (other than Brucella) and yeasts.
- For Brucella, clinical trial, reproducibility, and challenge testing were performed at "one external clinical trial site" (specifically, ATCC is mentioned for Brucella testing). This suggests a focus on specific, well-characterized strains for Brucella.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications
The document does not explicitly state the number of experts used or their specific qualifications (e.g., "radiologist with 10 years of experience"). However, it mentions that "Samples were sequenced by the appropriate reference laboratory, and if needed, additional analysis was performed internally to obtain a reference identification (GenBank, dendrogram analysis)."
This implies that the ground truth was established by:
- Personnel at "appropriate reference laboratories" (implying qualified microbiologists/scientists).
- Internal personnel at bioMérieux.
- The use of "proficiency panels tested by all participating technologists at the external sites" for the clinical trial suggests that the individuals performing the tests were trained and qualified.
While specific numbers and detailed qualifications of individual experts are not provided, the methodology points to reliance on established laboratory practices and reference techniques for accurate identification.
4. Adjudication Method for the Test Set
The document does not explicitly describe a formal adjudication method (like 2+1 or 3+1 for human readers of images). For microbiological identification, the "adjudication" is inherent in the ground truth establishment process:
- Reference methods (DNA sequencing analysis, supplemental testing) are considered the gold standard.
- If sequencing was insufficient, "additional analysis was performed internally to obtain a reference identification (GenBank, dendrogram analysis)." This suggests a process where discrepancies or difficult identifications are resolved by deeper, more definitive, and comparative analysis rather than a simple consensus among multiple readers.
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
No, a multi-reader multi-case (MRMC) comparative effectiveness study was not done in the traditional sense of comparing human readers with and without AI assistance for interpretation tasks. This device is an automated system for identifying microorganisms, not an AI for image interpretation that assists human diagnosticians. The study focuses on the accuracy of the automated system against a definitive reference method.
6. Standalone (Algorithm Only Without Human-in-the-Loop Performance)
Yes, the study primarily evaluates the standalone performance of the VITEK® MS system. The reported performance metrics (e.g., "Total Correct Genus ID") reflect the accuracy of the device's algorithm in identifying microorganisms when presented with samples.
While the "Interpretation of results and use of the VITEK® MS system require a competent laboratorian" is mentioned in the device description, the performance tables quantify the output of the machine itself compared to the ground truth, effectively reporting its standalone accuracy.
7. The Type of Ground Truth Used
The ground truth for microorganism identification was established using:
- DNA sequencing analysis: This is the primary reference method.
- Supplemental testing (when necessary): This could include biochemical tests, serology, or other definitive microbiological techniques.
- Previously well-characterized strains (ATCC or equivalent): For certain challenge or quality control tests, established strains were used, for which the identity is already known and highly reliable.
- GenBank and dendrogram analysis: Used for additional internal analysis to obtain reference identification when sequencing was not fully conclusive, reinforcing the ground truth.
This indicates a robust, multi-faceted approach to ground truth establishment, relying on genetic and phenotypic gold standards.
8. The Sample Size for the Training Set
The document does not explicitly state the sample size used for the training set for the VITEK® MS Knowledge Base (KB) v3.2.0. It mentions that the "Knowledge Base includes data representing 1316 species and 1158 taxa displayed" and "1095 species of bacteria" and "232 species of fungi." This describes the content of the database used for identification, which is analogous to the "training data" that built the system's ability to recognize these species. However, the exact number of samples or spectra used to build this database is not provided in this excerpt.
9. How the Ground Truth for the Training Set Was Established
The document describes the "Knowledge Base" as having been "developed from spectra of a number of microbial species." While it doesn't detail the training process or the ground truth establishment for the training set specifically, it's highly implied that the same rigorous methods used for the test set (DNA sequencing, supplemental testing, well-characterized strains) would have been used to curate the reference spectra within the Knowledge Base. This ensures that the foundational data used by the device for identification is accurate and reliable. The continuous updates to the "Knowledge Base v3.2.0" suggest an ongoing process of incorporating new and validated spectral data.
§ 866.3378 Clinical mass spectrometry microorganism identification and differentiation system.
(a)
Identification. A clinical mass spectrometry microorganism identification and differentiation system is a qualitative in vitro diagnostic device intended for the identification and differentiation of microorganisms from processed human specimens. The system acquires, processes, and analyzes spectra to generate data specific to a microorganism(s). The device is indicated for use in conjunction with other clinical and laboratory findings to aid in the diagnosis of bacterial and fungal infection.(b)
Classification. Class II (special controls). The special controls for this device are:(1) The intended use statement must include a detailed description of what the device detects, the type of results provided to the user, the clinical indications appropriate for test use, and the specific population(s) for which the device is intended, when applicable.
(2) Any sample collection device used must be FDA-cleared, -approved, or -classified as 510(k) exempt with an indication for in vitro diagnostic use.
(3) The labeling required under § 809.10(b) of this chapter must include:
(i) A detailed device description, including all device components, control elements incorporated into the test procedure, instrument requirements, ancillary reagents required but not provided, and a detailed explanation of the methodology and all pre-analytical methods for processing of specimens, and algorithm used to generate a final result. This must include a description of validated inactivation procedure(s) that are confirmed through a viability testing protocol, as applicable.
(ii) Performance characteristics for all claimed sample types from clinical studies with clinical specimens that include prospective samples and/or, if appropriate, characterized samples.
(iii) Performance characteristics of the device for all claimed sample types based on analytical studies, including limit of detection, inclusivity, reproducibility, interference, cross-reactivity, interfering substances, carryover/cross-contamination, sample stability, and additional studies regarding processed specimen type and intended use claims, as applicable.
(iv) A detailed explanation of the interpretation of test results for clinical specimens and acceptance criteria for any quality control testing.
(4) The device's labeling must include a prominent hyperlink to the manufacturer's website where the manufacturer must make available their most recent version of the device's labeling required under § 809.10(b) of this chapter, which must reflect any changes in the performance characteristics of the device. FDA must have unrestricted access to this website, or manufacturers must provide this information to FDA through an alternative method that is considered and determined by FDA to be acceptable and appropriate.
(5) Design verification and validation must include:
(i) Any clinical studies must be performed with samples representative of the intended use population and compare the device performance to results obtained from an FDA-accepted reference method and/or FDA-accepted comparator method, as appropriate. Documentation from the clinical studies must include the clinical study protocol (including predefined statistical analysis plan, if applicable), clinical study report, and results of all statistical analyses.
(ii) Performance characteristics for analytical and clinical studies for specific identification processes for the following, as appropriate:
(A) Bacteria,
(B) Yeasts,
(C) Molds,
(D) Mycobacteria,
(E) Nocardia,
(F) Direct sample testing (
e.g., blood culture),(G) Antibiotic resistance markers, and
(H) Select agents (
e.g., pathogens of high consequence).(iii) Documentation that the manufacturer's risk mitigation strategy ensures that their device does not prevent any device(s) with which it is indicated for use, including incorporated device(s), from achieving their intended use (
e.g., safety and effectiveness of the functions of the indicated device(s) remain unaffected).(iv) A detailed device description, including the following:
(A) Overall device design, including all device components and all control elements incorporated into the testing procedure.
(B) Algorithm used to generate a final result from raw data (
e.g., how raw signals are converted into a reported result).(C) A detailed description of device software, including validation activities and outcomes.
(D) Acquisition parameters (
e.g., mass range, laser power, laser profile and number of laser shots per profile, raster scan, signal-to-noise threshold) used to generate data specific to a microorganism.(E) Implementation methodology, construction parameters, and quality assurance protocols, including the standard operating protocol for generation of reference entries for the device.
(F) For each claimed microorganism characteristic, a minimum of five reference entries for each organism (including the type strain for microorganism identification), or, if there are fewer reference entries, a clinical and/or technical justification, determined by FDA to be acceptable and appropriate, for why five reference entries are not needed.
(G) DNA sequence analysis characterizing all type strains and at least 20 percent of the non-type strains of a species detected by the device, or, if there are fewer strain sequences, then a clinical and/or technical justification, determined by FDA to be acceptable and appropriate, must be provided for the reduced number of strains sequenced.
(H) As part of the risk management activities, an appropriate end user device training program, which must be offered as an effort to mitigate the risk of failure from user error.