(82 days)
The ARIES® Group A Strep Assay is a real-time polymerase chain reaction (PCR) based qualitative in vitro diagnostic test for the direct detection of Streptococus pyogenes (Group A beta-hemolytic Streptococcus) in throat swab specimens from patients with signs and symptoms of pharyngitis.
The ARIES® Group A Strep Assay can be used as an aid in the diagnosis of Group A Streptococcal pharyngitis. The assay is not intended to monitor treatment for Group A Streptococcus infections.
The ARIES® Group A Strep Assay is indicated for use with ARIES® Systems.
The ARIES® Group A Strep Assay is a polymerase chain reaction (PCR)-based qualitative in vitro diagnostic test system that consists of the ARIES® System or the ARIES® M1 System with their included ARIES® Software, an assay-specific cassette, and an assay-specific protocol file. The ARIES® Group A Strep Assay cassette is a disposable, single-use cassette containing nucleic acid purification reagents, internal sample process control (SPC), and an assay-specific master mix capable of performing the designated assay on one sample. The ARIES® Group A Strep Assay cassette directly detects Streptococcus pyoqenes (Group A ß-hemolytic Streptococcus) in throat swab specimens collected from the surface of human tonsils and posterior pharyngeal wall.
Throat swab specimens are collected from patients using a commercially available Liquid Amies based transport system (Nylon Flocked Swab with 1 mL modified Liquid Amies (ESwab™). The specimen is then transported to the laboratory for testing.
The specimen is lysed and nucleic acid is extracted using an ARIES® System. An extractable sample processing control (SPC) target is present in the ARIES® Group A Strep Assay cassette and is processed with the specimen. The SPC controls for recovery of extracted nucleic acid, the presence of inhibitory substances and for PCR reagent and instrument integrity. The Ct value of the SPC is designed to verify nucleic acid extraction, to identify PCR inhibition, if any, and verify proper function of the extraction system and real-time instrument. The Tm value of the SPC is used as a reference for determining the target Tm.
The extracted nucleic acid and SPC are transferred via magnetic beads through the cassette to the ARIES® Group A Strep Assay lyophilized PCR reagents in the PCR tube that contains primer pairs specific to the S. pyogenes DNaseB (sdaB) gene and the SPC sequence. Each of the primer pairs is labeled with a distinct fluorophore and detected in distinct channels of the ARIES® Systems. PCR amplification is performed and assay fluorescence is monitored. Incorporation of a quencher-labeled nucleotide results in a decrease in fluorescence for the associated primer pair. Following amplification, the reaction is slowly heated to separate the fluorescent-labeled strand from the quencher-labeled strand, a process that results in an increase in the fluorescence signal. The reaction fluorescence is measured during this process and the temperature at which the change in fluorescence is the maximum is the T ,, of the amplicon. The instrument fluorescence output is analyzed and test results are determined using the ARIES® System software and the ARIES® Group A Strep Assay protocol and run files. ARIES® Group A Strep Assay results may be reported from the ARIES® Software or from the optional SYNCT® Software.
Here's a breakdown of the acceptance criteria and the study details for the ARIES® Group A Strep Assay, based on the provided document:
1. Table of Acceptance Criteria and Reported Device Performance
| Performance Metric | Acceptance Criteria (Implicit) | Reported Device Performance (ARIES Group A Strep Assay) |
|---|---|---|
| Clinical Sensitivity | High, to ensure detection of true positives | 97.5% (156/160), 95% CI: 93.7% - 99.0% |
| Clinical Specificity | High, to minimize false positives | 97.8% (448/458), 95% CI: 96.0% - 98.8% |
| Positive Predictive Value (PPV) | High, indicating reliability of positive results | 94.0% (156/166), 95% CI: 89.3% - 96.7% |
| Negative Predictive Value (NPV) | High, indicating reliability of negative results | 99.1% (448/452), 95% CI: 97.7% - 99.7% |
| Reproducibility (Overall Moderate Positive - 3X LoD) | Consistent detection across sites | 98.9% (89/90) |
| Reproducibility (Overall Low Positive - 1X LoD) | Consistent detection across sites for low concentrations | 96.7% (87/90) |
| Reproducibility (Overall Negative) | Consistent negative results across sites | 1.1% (1/90) false positive, indicating high specificity |
| Lot-to-Lot Reproducibility (Overall Moderate Positive - 3X LoD) | Consistent detection across lots | 100% (45/45) |
| Lot-to-Lot Reproducibility (Overall Low Positive - 1X LoD) | Consistent detection across lots for low concentrations | 93.3% (42/45)* |
| Lot-to-Lot Reproducibility (Overall Negative) | Consistent negative results across lots | 0.0% (0/45) |
| Within-Laboratory Precision/Repeatability (Moderate Positive - 3X LoD) | Consistent detection within lab | 100% (30/30) |
| Within-Laboratory Precision/Repeatability (Low Positive - 1X LoD) | Consistent detection within lab for low concentrations | 93.3% (28/30)* |
| Within-Laboratory Precision/Repeatability (Negative) | Consistent negative results within lab | 0.0% (0/30) |
| Limit of Detection (LoD) - Bruno strain | Lowest concentration detected with ≥ 95% positivity | 2.58E+03 CFU/mL (95%) |
| Limit of Detection (LoD) - SF370 strain | Lowest concentration detected with ≥ 95% positivity | 4.13E+03 CFU/mL (100%) |
| Analytical Reactivity (Inclusivity) | Detection of various S. pyogenes strains | 8/9 strains detected with 100% positivity at 3x LoD, 1 strain at 5x LoD |
| Interfering Substances | No inhibition/false results for tested substances | Expected results generally obtained, except for NyQuil (false negatives) and Mucin (invalid results) |
| Cross-Reactivity/Microbial Interference | No cross-reactivity with common throat microorganisms | 34/35 tested microorganisms showed no cross-reactivity or interference with GAS positivity (1 Treponema denticola interference at high concentrations) |
| Carry-Over/Cross-Contamination | No carry-over between samples | 100% agreement with expected results for high positive and negative samples |
Note: For Lot-to-Lot Reproducibility (Low Positive) and Within-Laboratory Precision/Repeatability (Low Positive), additional replicates were tested to achieve an overall positivity >94%, indicating acceptable performance at the LoD.
2. Sample Size Used for the Test Set and Data Provenance
- Sample Size for Clinical Performance (Test Set): A total of 618 unique specimens were available for analysis in the clinical performance study. Initially, 735 specimens were collected, but 112 were excluded due to various reasons (protocol non-compliance, insufficient volume, incorrect device, etc.).
- Data Provenance: The data was prospectively collected from patients suspected of pharyngitis in four geographically distinct clinical sites within the United States.
3. Number of Experts Used to Establish Ground Truth for the Test Set and Qualifications of Those Experts
The document does not explicitly state the "number of experts" or their specific "qualifications" (e.g., "Radiologist with 10 years of experience") for establishing the ground truth of the clinical test set.
However, the ground truth for the clinical test set was established by a "reference method (bacterial culture followed by organism identification by Matrix-Assisted Laser Desorption/Ionization - Time-of-Flight Mass Spectrometry (MALDI-TOF MS))". This work was "performed at a centralized testing facility." It is implied that trained laboratory personnel and microbiologists conducted these reference tests, as is standard practice for such methods, but their specific experience levels are not detailed.
4. Adjudication Method for the Test Set
The document describes a robust "ground truth" establishment process using a reference laboratory technique (bacterial culture + MALDI-TOF MS). For specimens where the ARIES assay and the reference method disagreed, an adjudication method was employed:
- Bidirectional sequencing analysis using analytically validated primers that targeted genomic regions distinct from the ARIES Group A Strep Assay was used.
- For 2 false negative ARIES results (ARIES negative, culture positive), sequencing showed them to be GAS negative.
- For 7 false positive ARIES results (ARIES positive, culture negative), sequencing showed them to be GAS positive.
This indicates a method similar to "Truth by Consensus" with an expert-level tie-breaker (sequencing), effectively refining the ground truth against discrepancies.
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance
No, a Multi-Reader Multi-Case (MRMC) comparative effectiveness study was not done. This device is an in vitro diagnostic (IVD) assay that machine-interprets results (qualitative PCR). It does not involve human "readers" assessing images or data in a way that would be "assisted" by AI, hence the concept of a human reader improvement effect size does not apply.
6. If a Standalone (i.e. algorithm only without human-in-the-loop performance) was done
Yes, a standalone performance evaluation was done. The ARIES® Group A Strep Assay is an automated qualitative in vitro diagnostic test system. The clinical performance study directly assessed the agreement of the ARIES® Group A Strep Assay (algorithm only within the ARIES® Systems) against the established reference method, without human interpretation of the assay's raw output. The results (Positive, Negative, Invalid) are determined by the ARIES® System software.
7. The Type of Ground Truth Used (expert consensus, pathology, outcomes data, etc)
The ground truth used for the clinical performance study was primarily bacterial culture followed by organism identification by Matrix-Assisted Laser Desorption/Ionization - Time-of-Flight Mass Spectrometry (MALDI-TOF MS). In cases of discordance with the assay, bidirectional sequencing analysis was used as a confirmatory "higher truth" method. This can be considered a form of expert consensus/gold standard laboratory method.
8. The Sample Size for the Training Set
The document does not explicitly state a sample size for a training set. The ARIES® Group A Strep Assay is a PCR-based diagnostic test, where the "training" typically involves empirical optimization of assay parameters (primers, probes, reaction conditions, and cut-offs) during product development and internal verification studies, rather than machine learning on a distinct "training set" of patient data. The "Initial Assay Protocol File parameters were set during internal optimization studies" and "The final Assay Protocol File parameters were then established during internal verification studies." So, the optimization and verification process served a similar function to "training" for machine learning algorithms.
9. How the Ground Truth for the Training Set Was Established
As noted in point 8, a traditional "training set" with separate ground truth establishment isn't described in the context of a PCR assay. Instead, the "ground truth" for optimizing the assay protocol file parameters (Ct cut-off, Tm window, Tm Peak Threshold) was established through internal optimization and verification studies. These studies likely involved testing known positive and negative S. pyogenes samples (e.g., well-characterized strains, spiked samples) to define the optimal performance characteristics of the assay before clinical validation.
§ 866.2680
Streptococcus spp. nucleic acid-based assay.(a)
Identification. AStreptococcus spp. nucleic acid-based assay is a qualitative in vitro diagnostic device intended to simultaneously detect and identify variousStreptococcus spp. nucleic acids extracted directly from clinical specimens. The device detects specific nucleic acid sequences for organism identification. The identification aids in the diagnosis of diseases caused by bacteria belonging to the genusStreptococcus and provides epidemiological information on these diseases. Pathogenic streptococci are associated with infections, such as sore throat, impetigo (an infection characterized by small pustules on the skin), urinary tract infections, rheumatic fever, and kidney disease.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Premarket notification submissions must include detailed device description documentation, including the device components, ancillary reagents required but not provided, and a detailed explanation of the methodology including primer/probe sequence, design, and rationale for sequence selection.
(2) Premarket notification submissions must include detailed documentation from the following analytical and clinical performance studies: Analytical sensitivity (Limit of Detection), reactivity, inclusivity, precision, reproducibility, interference, cross reactivity, carry-over, and cross contamination.
(3) Premarket notification submissions must include detailed documentation from a clinical study. The study, performed on a study population consistent with the intended use population, must compare the device performance to results obtained from well-accepted reference methods.
(4) Premarket notification submissions must include detailed documentation for device software, including, but not limited to, software applications and hardware-based devices that incorporate software.
(5) Premarket notification submissions must include database implementation methodology, construction parameters, and quality assurance protocols, as appropriate.
(6) The device labeling must include limitations regarding the need for culture confirmation of negative specimens, as appropriate.
(7) A detailed explanation of the interpretation of results and acceptance criteria must be included in the device's 21 CFR 809.10(b)(9) compliant labeling.
(8) Premarket notification submissions must include details on an end user device training program that will be offered while marketing the device, as appropriate.