Search Results
Found 6 results
510(k) Data Aggregation
(239 days)
PZM
Ask a specific question about this device
(718 days)
PZM
The ACTOnco IVD assay is an in vitro diagnostic test that uses targeted next generation sequencing of formalin-fixed. paraffin-embedded tumor tissue from patients with solid malignant neoplasms to detect genetic alterations in a broad multi gene panel. The test is intended to provide informations, small insertions and deletions, ERBB2 gene amplification, and tumor mutational burden for use by qualified health care professionals in accordance with professional guidelines, and is not conclusive or prescriptive for labeled use of any specific therapeutic product. ACTOnco IVD is a single-site assay performed at ACT Genomics.
The ACTOnco IVD assay is an in-vitro diagnostic assay intended to provide information for use by qualified health care professionals in accordance with professional guidelines, and is not conclusive or prescriptive for labeled use of any specific therapeutic product.
The assay is a custom targeted sequencing platform, utilizing amplicon-based sequencing, to detect point mutations (single nucleotide variants, or SNVs), small insertions and deletions (Indels), ERBB2 gene amplification, and tumor mutational burden (TMB) in tumor specimens. The assay uses custom DNA primers corresponding to all exons and selected introns of oncogenes, tumor suppressor genes, drug metabolism genes, and immune-related genes. Primers are synthesized by a secondary manufacturer (Thermo Fisher Scientific). An overlapping amplicon approach is utilized in which tiled primers are designed to generate multiple overlapping amplicons of the same region to avoid allele dropout. In total, the primers target approximately 1.8Mb of the human genome. Genomic DNA is extracted from FFPE tissue samples.
Sequence libraries are prepared through a multiplex polymerase chain reaction (PCR) amplification step to enrich target sequences. Target sequences are tagged with index oligonucleotide to identify individual sample and adaptor oligonucleotide to anchor the amplicon to complimentary oligonucleotides embedded on the surface of the sequencing bead. Target sequences on the sequencing beads are amplified using emulsion PCR before sequencing. Multiple barcoded sequence libraries (from different patients) are pooled and then sequenced on a Thermo Fisher Ion GeneStudio™ S5 Prime System. Sequence reads are then aligned to the reference human genome. By comparing the identity of bases from the tumor DNA and the reference human genome, variant alterations are identified in the tumor.
The assay system includes a sequencing instrument, reagents (DNA extraction, library preparation and sequencing), software (operation of the sequencing instrument and variant calling), and standard operating procedures (SOPs) for the use of the system. ACT Genomics takes the responsibilities in monitoring the instrument; reagents and consumable materials which will be used in the assay process.
Multiple software components will be used in the assay. The NGS raw read analysis will be done using Thermo Fisher software. Variant calling for SNVs, insertions and deletions will be done using Thermo Fisher software. Mutation and variant annotation will be done using software from ACT Genomics, the Cunningham Lab and Golden Helix software. ERBB2 gene amplification will be done using software from Boeva Lab. Tumor Purity and Zygosity will be done using software from Halgamuge Lab (Kaushalya Amarasinghe). Calculations for tumor mutational burden will be done using ACT Genomics software.
I will analyze the provided text to extract information about the acceptance criteria and the study proving the device meets these criteria. I will then structure this information according to your requested format.
However, based on the provided text, it appears to be a 510(k) summary for a medical device (ACTOnco IVD, a next-generation sequencing-based tumor profiling test). The document describes performance testing (precision/reproducibility, analytical sensitivity/limit of detection, analytical specificity/interference, cross-contamination, DNA input, DNA extraction) and method comparison data.
Crucially, this document
does not contain a direct table of acceptance criteria nor explicit statements about "acceptance criteria" met by the device.
It also does not describe a "Multi Reader Multi Case (MRMC) comparative effectiveness study" or information about "human readers improve with AI vs without AI assistance" as it is a diagnostic test and not an AI-assisted diagnostic imaging device/software for human readers.
Therefore, I will provide the acceptance criteria based on the performance observed in the various analytical validation studies, as these implicitly define the performance considered acceptable for the device. I will also make clear when certain information is not present in the provided text.
Here is the information structured according to your request, extracted from the provided 510(k) summary:
Device: ACTOnco IVD - Next generation sequencing based tumor profiling test
1. Table of Acceptance Criteria (Inferred from Performance Studies) and Reported Device Performance:
Since explicit acceptance criteria are not called out as a separate table, I will infer them from the "Performance Testing" sections, assuming that the reported performance metrics were considered acceptable for the device's clearance.
Performance Metric Category & Specificity | Inferred Acceptance Criterion (Target) | Reported Device Performance (observed outcomes from studies) |
---|---|---|
Precision / Reproducibility (SNVs, Indels) | ||
Overall Call Rate (Positive) | ≥ 95% | 98.33% (35,308/35,906) |
SNV Call Rate | ≥ 95% | 98.33% (31,837/32,377) |
MNV Call Rate | ≥ 95% | 97.18% (963/991) |
Insertion (INS) Call Rate | ≥ 95% | 96.97% (512/528) |
Deletion (DEL) Call Rate | ≥ 95% | 99.30% (1,996/2,010) |
Overall Call Rate (Negative/WT) | ≥ 99% | 99.997% (723,628/723,648) |
Precision / Reproducibility (ERBB2 Amplification) | ||
Positive Call Rate | 100% | 100% (based on 144 observations from 3 amplified samples) |
Negative Call Rate | 100% | 100% (based on 816 observations from 17 non-amplified samples) |
Precision / Reproducibility (TMB) | CV% ≤ 16% for TMB scores when Tumor Purity ≥ 20% | Ranges from 4.151% to 59.144% (for tumor purity 32.5% to 72.7%). Explicitly stated acceptable across tumor purities at or above 20% with percent CV 30%. |
Analytical Sensitivity / Limit of Detection (LoD) | ||
SNVs (Hotspot, 2% cutoff) | LoD/C95 for individual variants evaluated. Target to be near or below the cutoff. | Established MAF Range: 1.5%-6.6% |
SNVs (Non-hotspot, 5% cutoff) | LoD/C95 for individual variants evaluated. Target to be near or below the cutoff. | Established MAF Range: 2.4%-15.1% |
Insertions | LoD/C95 for individual variants evaluated. Target to be near or below the cutoff. | Established MAF Range: 1.1%-45.4% |
Deletions | LoD/C95 for individual variants evaluated. Target to be near or below the cutoff. | Established MAF Range: 1.9%-22.0% |
TMB (DNA Input) | Acceptable CV across relevant tumor purity and DNA input ranges | Acceptable TMB performance across tumor purities at or above 20% with percent CV 30%. |
ERBB2 Amplification (LoD) | 100% call rate for samples > 10% tumor purity | 100% call rate for samples with tumor purity over 10%. (Data supported consistent repeatability at tumor purity 30%). |
Analytical Specificity / Interference (SNVs and Indels) | ||
Correct Positive Calls (Spike-in) | 100% (consistent with control) | 100% (3288/3288) with lower 95% CI of 99.88% |
Correct Negative Calls (Spike-in) | 99.99% (consistent with control) | 99.99% (122,656/122,664) with lower 95% CI of 99.99% |
Necrotic Tissue Tolerance | Acceptable processing and agreement up to 50% necrotic tissue | Control Group ( |
Ask a specific question about this device
(334 days)
PZM
The NYU Langone Genome PACT assay is a qualitative in vitro diagnostic test that uses targeted next generation sequencing of formalin-fixed paraffin-embedded (FFPE) tumor tissue matched with normal specimens with solid malignant neoplasms to detect tumor gene alterations in a 607-gene panel. The test is intended to provide information on somatic mutations and small insertions and deletions) for use by qualified health care professionals in accordance with professional guidelines, and is not conclusive or prescriptive for labeled use of any specific therapeutic product. NYU Langone Genome PACT is a single-site assay performed at NYU Langone Health.
Not Found
This FDA 510(k) clearance letter acknowledges the substantial equivalence of the "NYU Langone Genome PACT (Genome Profiling of Actionable Cancer Targets)" device. However, it does not contain the detailed acceptance criteria and study results typically found in the predicate device’s 510(k) summary or the manufacturer’s submission. This document is a regulatory clearance letter, not a detailed technical report.
Therefore, I cannot provide the requested information based solely on the provided text.
To answer your request, I would need access to the actual 510(k) submission summary for K202304, which would describe the device's performance characteristics and the supporting studies.
What is available from the provided text:
- Device Name: NYU Langone Genome PACT (Genome Profiling of Actionable Cancer Targets)
- Intended Use: Qualitative in vitro diagnostic test using targeted next-generation sequencing of FFPE tumor tissue matched with normal specimens with solid malignant neoplasms to detect tumor gene alterations in a 607-gene panel. Intended to provide information on somatic mutations and small insertions/deletions for use by qualified healthcare professionals.
- Regulatory Class: Class II
- Product Code: PZM
- Regulation Number: 21 CFR 866.6080
- Regulation Name: Next generation sequencing based tumor profiling test
- Type of Use: Prescription Use
The following information remains unknown from the provided text:
- A table of acceptance criteria and the reported device performance.
- Sample size used for the test set and the data provenance.
- Number of experts used to establish the ground truth for the test set and their qualifications.
- Adjudication method for the test set.
- If a multi-reader multi-case (MRMC) comparative effectiveness study was done, and if so, the effect size.
- If a standalone performance (algorithm only) was done.
- The type of ground truth used (expert consensus, pathology, outcomes data, etc.).
- The sample size for the training set.
- How the ground truth for the training set was established.
Ask a specific question about this device
(267 days)
PZM
The PGDx elio™ tissue complete assay is a qualitative in vitro diagnostic device that uses targeted next generation sequencing of DNA isolated from formalin-fixed, paraffin-embedded tumor tissue from patients with solid malignant neoplasms to detect tumor gene alterations in a broad multi-gene panel.
PGDx elio tissue complete is intended to provide tumor mutation profiling information on somatic alterations (SNVs, small insertions and deletions, one amplification and four translocations), microsatellite instability (MSI) and tumor mutation burden (TMB) for use by qualified healthcare professionals in accordance with professional guidelines in oncology for previously diagnosed cancer patients, and is not conclusive or prescriptive for labeled use of any specific therapeutic product.
PGDx elio tissue complete is an in vitro diagnostic assay that uses NGS to detect turnor gene alterations in genomic DNA isolated from formalin-fixed, paraffin-embedded (FFPE) tumor tissue from a variety of tumor types, using a targeted panel (505 genes). The assay takes less than 7 days from DNA to report and provides information on single nucleotide variants (SNVs) in a range of GC content and genomic contexts, insertion/ deletions (indels), 1 amplification as well as 4 translocations. It also identifies microsatellite instability based on select mononucleotide tracts and signatures of sequence mutations. The PGDx elio tissue complete assay utilizes a ~1.3 Mb region of interest (ROI) to calculate tumor mutation burden (TMB).
The provided text describes the analytical validation studies for the PGDx elio tissue complete assay, a qualitative in vitro diagnostic device for tumor profiling.
Here's an analysis of the acceptance criteria and the study that proves the device meets them:
1. Table of Acceptance Criteria & Reported Device Performance
The acceptance criteria are generally established as target performance metrics (e.g., call rates, concordance rates, %CV, false positive rates). The reported device performance is the outcome of the analytical studies.
Performance Metric Category | Specific Metric (Acceptance Criteria Implicitly Defined by Study Design/Results) | Reported Device Performance |
---|---|---|
Specificity | False Positive Rate (SNVs and Indels) | 99.9% |
TMB Mean Absolute Percent Error (MAPE) vs. 100 ng | 1.8% to 11.8% | |
Contamination Control | Absence of false positives in negative samples from carryover/cross-contamination | No positive variant results observed in known negative samples. |
Exogenous Interference | Concordance (PPA) with interfering substances vs. baseline | > 97.2% |
Concordance (NPA) with interfering substances vs. baseline | > 99.9% | |
TMB MAPE with interfering substances | 0% to 6.0% | |
Overall Sample Acceptance Rate | First Pass Rate | 81.8% (2352/2874) |
Overall Pass Rate (allowing single repeat) | 92.9% (2671/2874) | |
Accuracy (Concordance to Orthogonal Methods) | PPA/NPA for various variant types (SNVs, indels, amplifications, translocations) | See Table 1.10 for detailed SNV/Indel PPAs (range 80.8% - 100%) and NPAs (all 99.9% or 100%) |
ERBB2 Amplifications: PPA 75.0% (all cases), 87.0% (excluding borderline); NPA 96.7% (all cases), 95.9% (excluding borderline) | ||
ALK Translocations: PPA 92.9%; NPA 98.2% | ||
RET Translocations: PPA 55.6%; NPA 100% | ||
TMB: Spearman correlation coefficient 0.903 vs WES | ||
MSI: Overall PPA 98.8% (excluding failures), 94.0% (accounting for failures); NPA 99.3% (excluding failures), 77.6% (accounting for failures) | ||
Interlaboratory Reproducibility | First Pass Rate | 90.3% (455/504) |
Overall Pass Rate | 98.2% (495/504) | |
Overall Positive Call Rate (All Variants) | 86.2% | |
APA (all variants) | > 92% | |
ANA (all variants) | > 99% | |
TMB %CV | 3.5% | |
Lot to Lot Precision | APA (Variants with Evidence of Clinical Significance) | 96.1% to 98.7% |
ANA (Variants with Evidence of Clinical Significance) | 99.8% to 99.9% | |
TMB %CV | 40 tumor types. |
- Accuracy: 582 samples with PGDx elio tissue complete data and orthogonal data. Specifically:
- SNVs/Indels: 582 samples.
- ERBB2 Amplifications (FISH comparison): 147 cases (all); 120 cases (excluding borderline).
- ALK Translocations (FISH comparison): 71 cases. Additional in silico simulation on 10 clinical samples (410 observations).
- RET Translocations (FISH comparison): 27 cases. 3 RET translocation-positive cell lines also tested.
- TMB: 118 cases across 8 tumor types.
- MSI: 283 samples across 18 tumor types.
- Wild Type Calls: 112 specimens.
- Interlaboratory Reproducibility: 13 FFPE tissue specimens and 1 cell line (14 samples total), tested in duplicate by 2 operators on 12 sequencing runs across 3 non-consecutive days at 3 independent laboratory sites (504 total replicates).
- Lot to Lot Precision: 5 test cases in triplicate across 3 unique kit lots (45 observations).
Data Provenance: The document does not explicitly state the country of origin for the clinical samples. However, it mentions "clinical FFPE specimens," implying real-world patient data. The studies are described in a factual manner, suggesting they are retrospective analyses of collected samples for analytical validation purposes.
3. Number of Experts Used to Establish Ground Truth and Qualifications
The document does not specify the number or qualifications of experts involved in establishing the ground truth for the test set.
Instead, the ground truth for performance evaluation (accuracy) is established by:
- Orthogonal methods:
- "2 NGS targeted panels" and "PCR" (for SNVs and Indels).
- "ERBB2 FISH," "ALK FISH," "RET FISH" (for amplifications and translocations).
- "matched tumor-normal whole exome sequencing results" (for TMB).
- "MSI PCR" (for MSI).
- Validated assays/literature: For the 3 RET translocation-positive cell lines.
4. Adjudication Method for the Test Set
The document does not describe any expert adjudication process for resolving discrepancies or establishing ground truth for the test set. The ground truth is primarily based on the results from the orthogonal methods. For the in silico studies, the ground truth is derived from down-sampling "clinical samples."
5. MRMC Comparative Effectiveness Study
No multi-reader multi-case (MRMC) comparative effectiveness study was mentioned or performed. This device is a molecular diagnostic assay, not an imaging AI tool, and thus comparative effectiveness with human readers improving with AI assistance is not applicable in this context.
6. Standalone Performance
Yes, the entire document describes studies of the standalone (algorithm only, without human-in-the-loop performance) of the PGDx elio tissue complete assay. The goal is to demonstrate the analytical performance of the automated workflow, from sample preparation to data analysis and variant calling, against established ground truth methods.
7. Type of Ground Truth Used
The primary type of ground truth used is orthogonal methods, specifically:
- Other NGS targeted panels: For SNVs and indels.
- PCR: For certain SNVs/indels and MSI.
- FISH (Fluorescence In-Situ Hybridization): For gene amplifications (ERBB2) and translocations (ALK, RET).
- Whole Exome Sequencing (WES): For Tumor Mutation Burden (TMB).
- Cell line characterization/literature: For known variants in control cell lines.
This is a form of reference standard ground truth, where the device's performance is compared against established, validated measurement techniques.
8. Sample Size for the Training Set
The document details analytical validation (testing) and reproducibility studies. It does not provide information about a "training set" size. As a molecular diagnostic assay, its development likely involves internal data for algorithm development and optimization, but specific training set sizes are not mentioned in this regulatory submission, which focuses on validation data.
9. How the Ground Truth for the Training Set Was Established
Since the document does not discuss a specific "training set" or its size, it also does not describe how ground truth for such a set was established. The focus is on the performance of the final, locked-down algorithm against independent test data with ground truth established by orthogonal methods.
Ask a specific question about this device
(240 days)
PZM
The Omics Core assay is a qualitative in vitro diagnostic test that uses targeted next generation sequencing of formalin-fixed paraffinembedded tumor tissue matched with normal specimens with solid malignant neoplasms to detect tumor gene alterations in a broad multi gene panel. The test is intended to provide informations (point mutations (point mutations and deletions) and tumor mutational burden (TMB) for use by qualified health care professionals in accordance with professional guidelines, and is not conclusive or prescriptive for labeled use of any specific therapeutic product. Omics Core is a single-site assay performed at NantHealth, Inc.
The NantHealth Omics Core assay is a custom targeted whole exome sequencing platform, utilizing solution-phase exon capture and sequencing, to report somatic alterations (point mutations, small insertions and deletions) in 468 genes and sequencing of 19,396 protein-coding genes (whole exome) to determine overall tumor mutation burden in tumor specimens. Genomic DNA is extracted from both a tumor and a patient-matched normal control sample. Sequence libraries are prepared through a series of enzymatic steps including shearing of double-stranded DNA, end repair, A-base addition, ligation of barcoded sequence adaptors, and low cycle PCR amplification. Single barcoded sequence libraries are captured using the biotinylated probes. Captured DNA fragments are then pooled and sequenced on an Illumina NovaSeq 6000 as paired-end reads. Sequence reads are then aligned to the reference human genome. Somatic alterations are identified in the tumor DNA by direct comparison to the matched normal DNA.
The Omics Core assay, a targeted next-generation sequencing test, was evaluated to determine its performance specifications, including precision, analytical sensitivity (Limit of Detection), and accuracy.
Here's a breakdown of the requested information:
1. Table of Acceptance Criteria and Reported Device Performance:
The document doesn't explicitly list "acceptance criteria" as a separate column with specific numerical thresholds for each performance metric. Instead, it presents "Performance Specifications" as the expected or demonstrated performance of the device based on the studies. The reported device performance is then detailed under each characteristic.
Table: Omics Core Performance Specifications and Reported Device Performance
Characteristic | Performance Specifications (Implicit Acceptance Criteria) | Reported Device Performance |
---|---|---|
Precision – Panel-Wide Reproducibility | Demonstrated with a high overall positive call rate for all variants analyzed. | The overall positive call rate for all variants analyzed across the 12 FFPE clinical samples and one commercial cell line was 2607/2650, or 98.4% (97.8-98.8% CI). |
Precision – Per Specimen | Demonstrated with consistent repeatability and reproducibility. | Per specimen variant analysis for 12 clinical specimens and a commercial cell line demonstrated 100% concordance for 511 out of 530 unique mutations, or 96.4% (94.5%-97.8% CI). |
Precision - Well Characterized Reference Material | Demonstrated consistent repeatability and reproducibility. | Analysis of the well-characterized sample demonstrated an overall positive call rate of 99.86% (99.75%-99.93% CI). |
Precision - TMB | Demonstrated repeatable and reproducible TMB rates. | TMB precision analysis based on 12 clinical samples and 1 commercial cell line showed repeatable and reproducible TMB rates with a **%CV |
Ask a specific question about this device
(51 days)
PZM
The MSK-IMPACT assay is a qualitative in vitro diagnostic test that uses targeted next generation sequencing of formalin-fixed paraffin-embedded tumor tissue matched with normal specimens from patients with solid malignant neoplasms to detect tumor gene alterations in a broad multi gene panel. The test is intended to provide information on somatic mutations (point mutations and small insertions and deletions) and microsatellite instability for use by qualified health care professionals in accordance with professional guidelines, and is not conclusive or prescriptive for labeled use of any specific therapeutic product. MSK-IMPACT is a single-site assay performed at Memorial Sloan Kettering Cancer Center.
A description of required equipment, software, reagents, vendors, and storage conditions were provided, and are described in the product labeling (MSK-IMPACT manual). MSK assumes responsibility for the device.
Here's a breakdown of the acceptance criteria and the study that proves the device meets them, based on the provided text for the MSK-IMPACT assay.
Device Name: MSK-IMPACT (Integrated Mutation Profiling of Actionable Cancer Targets)
Type of Test: Next generation sequencing tumor profiling test
Purpose: Qualitative in vitro diagnostic test for detecting somatic mutations (point mutations, small insertions and deletions) and microsatellite instability (MSI) in formalin-fixed paraffin-embedded (FFPE) tumor tissue matched with normal specimens from patients with solid malignant neoplasms.
1. Table of Acceptance Criteria and Reported Device Performance
The acceptance criteria are generally embedded within the "Performance" section and the "Reporting" section's "Table 3. Sample Level Quality Control Metrics." The reported performance is found throughout the "Performance" section.
Acceptance Criteria (from text) | Reported Device Performance (from text) |
---|---|
Specimen Requirements: | |
Minimum Tumor Proportion: >10% of tumor cells; >20% viable tumor preferred; >25% for MSI testing. | The minimum tumor proportion required for the MSI assay was established as 25% (based on CRC specimens, assay and score reproducible to 8% tumor proportion qualitatively, but decreased trend quantitatively) (Table 1). The DNA extraction method was validated with historic data from >10,000 specimens, demonstrating invalid rates of 7.2% to 18.4%, supporting performance across FFPE tumor types (Table 5). |
Quality Control Metrics (Table 3): | |
Average target coverage: > 200X | For normal samples, mean coverage across all targeted exons was 571X (SD = 373X). Analysis of normal samples showed that with mean sample coverage of 571X, 98% of exons are sequenced with coverage greater than 306X (or normalized coverage >0.54), leading to a conservative threshold of 200X mean sample coverage. In silico downsampling to 203X coverage detected 94% of mutations with 10% VAF (Performance L.1.b and Table 3). |
Coverage Uniformity: ≥ 98% target exons above 100X coverage | 99.5% of exons were sequenced to a depth of 100X or greater, and 98.6% to 250X or greater. It’s expected that 98% of exons will be sequenced to >100X coverage when mean sample coverage is 185X. (Performance L.1.b) |
Base Quality: > 80% of bases with QS above > Q30 | Not explicitly detailed in the performance section but stated as a QC metric in Table 3. Implicitly met if overall performance is approved. |
% Cluster passing filter (Cluster PF): > 80% | Not explicitly detailed in the performance section but stated as a QC metric in Table 3. Implicitly met if overall performance is approved. |
% Reads passing filter (Reads PF): > 80% | Not explicitly detailed in the performance section but stated as a QC metric in Table 3. Implicitly met if overall performance is approved. |
Hotspot Mutation calling threshold: DP ≥ 20, AD ≥ 8, VF ≥ 2% | Filtering scheme designed to reject false positives while maintaining detection capability. Example: pre-filter SNVs (hotspot) had 1 false positive, post-filter 0 (Table 4). LoD confirmation: 5 replicates for 6 SNVs at 5% MAF showed 100% positive call rates, except one replicate failing on PTEN exon 6 due to low read depth below 5% (Performance L.2.b.ii and Table 11). |
Non-hotspot Mutation threshold: DP > 20, AD ≥ 10, VF ≥ 5% | Filtering scheme designed to reject false positives while maintaining detection capability. Example: pre-filter SNVs (non-hotspot) had 342 false positives, post-filter 0 (Table 4). LoD study showed most mutations detected at low VAFs (e.g., 2-9% in Tables 10A-J). Confirmed LoD study (Part 2) for various mutations showed 100% positive call rates for variant types except one discordant case (PTEN exon 8 deletion) at 3.6-7.9% VF (Table 11). |
Indels: Fewer than 20% of samples in an established 'standard normal' database (This seems to be a filtering criteria for indels, not a reporting metric.) | Indels had 40,793 pre-filter false positives, reduced to 8 post-filter (Rejection Rate 0.999) (Table 4). LoD confirmation: 5 replicates for 3 deletions and 4 insertions at 5% MAF showed 100% positive call rates, except one deletion (PTEN exon 6), which also failed read depth (Performance L.2.b.ii and Table 11). |
Positive Run Control: The difference between the observed and expected frequencies for the known mutations should be within 5%. | Mixed positive control sample with expected VFs: Results reviewed to confirm known mutations called and observed frequencies match expected values within 5% (Controls, b). |
Negative Run Control: The correlation between expected and observed mutation frequencies should be 0.9 or higher. | Pooled negative control: Observed mutation frequencies compared against expected for 862 common SNPs; correlation expected to be 0.9 or higher (Controls, c). Figure 2 shows correlation of 0.975 (with slope 0.971 and intercept -0.004) for observed vs. expected variant frequency, establishing consistent correlation >0.9. |
Sample-Mix up QC: Flagged if pairs of samples from the same patient with > 5% discordance and from different patients with 5% for same patient ("unexpected mismatches") or 10,000 samples mentioned in pre-analytical performance context). Table 13 presents data by DNA input amounts but not sample count for each bin. |
* **Accuracy (Method Comparison):**
* 267 unique mutations in **433 FFPE tumor specimens** for the main comparison (Table 14).
* **95 specimens** for the supplemental wildtype calls study.
* **138 colorectal cancer (CRC) and 40 endometrial carcinoma (EC) specimens** (training set) for MSI cutoff establishment.
* **135 CRC patients** (66 with both MSK-IMPACT and IHC) for MSI cutoff validation.
* **119 unique non-CRC and non-EC tumor-normal pair samples** for MSI comparison in other cancer types.
- Data Provenance:
- General: The device is performed at Memorial Sloan Kettering Cancer Center (MSK), indicating the data likely originates from their patient population.
- Retrospective/Prospective:
- The pre-analytical performance (specimen invalid rates) used historical data from >10,000 specimens, implying a retrospective chart review.
- The MSI validation study (CRC patients) was a retrospective-prospective chart review.
- The clinical performance section mentions a large-scale, prospective clinical sequencing initiative using MSK-IMPACT involving >10,000 patients, whose data are publicly accessible. This cohort likely informed the broader context and understanding of the device but was not explicitly stated as the test set for the analytical validation.
- The analytical performance studies (precision, LoD, accuracy) used clinical samples/specimens, which could be retrospective or prospectively collected for the purpose of the study. The text doesn't explicitly state for each study.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and the Qualifications of Those Experts
The text does not specify the number of experts used to establish the ground truth for the test set, nor their specific qualifications (e.g., "radiologist with 10 years of experience").
However, it does indicate:
- For the accuracy studies, results were compared to "original results obtained with the validated orthogonal methods." This implies that the ground truth was established by these validated orthogonal methods, which are presumably performed and interpreted by qualified personnel using established clinical diagnostics.
- For MSI, the MSIsensor results were compared to "a validated MSI-PCR or MMR IHC test," a "commercially available PCR assay," or a "validated IHC panel (MLH1, MSH2, MSH6 and PMS2)." Again, this suggests ground truth from established, clinical laboratory methods.
- The "Clinical Evidence Curation" section mentions that "OncoKB undergoes periodic updates through the review of new information by a panel of experts," which informs the clinical interpretation of detected mutations. This expert panel contributes to the broader clinical context of the mutations, but not directly the ground truth for the analytical test set itself.
4. Adjudication Method (e.g., 2+1, 3+1, none) for the Test Set
The text does not describe a formal adjudication method (like 2+1 or 3+1 consensus with experts) for establishing the ground truth of the test set cases. Instead, the ground truth was derived from "validated orthogonal methods."
For example, in the accuracy study, the MSK-IMPACT results were "compared to the original results obtained with the validated orthogonal methods." This indicates that the results from the comparison methods served as the reference standard, rather than requiring an additional expert adjudication process on top of those existing validated methods.
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was Done
No, an MRMC comparative effectiveness study was not described. This document pertains to the analytical validation of a genetic sequencing assay, which inherently does not involve human readers interpreting images in a multi-reader, multi-case setup. Therefore, a comparative effectiveness study measuring human reader improvement with AI assistance (which is typical for imaging AI) is not applicable here.
6. If a Standalone (Algorithm Only Without Human-in-the-Loop Performance) Was Done
Yes, the analytical performance studies (precision, analytical sensitivity, and analytical accuracy) described are all measures of the standalone performance of the MSK-IMPACT assay, which relies on its sequencing and bioinformatics pipeline without direct human-in-the-loop diagnostic interpretation to produce the raw mutation calls.
- The "Mutation calling SNVs and Indels" section and "Summary of mutation filtering scheme" (Figure 1) describe the automated pipeline for identifying mutations.
- The "Performance" section details how characteristics like precision, LoD, and accuracy were determined for the assay itself by comparing its outputs to known or established results from other validated methods. These do not involve a human interpreting the device's output to make a diagnosis within the performance evaluation but rather assess the accuracy of the device's genomic calls directly.
7. The Type of Ground Truth Used
The primary type of ground truth used was:
- Orthogonal Methods / Comparator Assays: For the accuracy studies, the MSK-IMPACT results were compared against "original results obtained with the validated orthogonal methods." This included comparison to:
- Validated orthogonal methods for SNVs and indels.
- Established MSI-PCR or MMR IHC tests for Microsatellite Instability status.
- Known Reference Material: For precision, a "well characterized reference standard (HapMap cell line NA20810)" was used, with reference genotypes obtained from the 1000 Genomes database.
- Expected Values/Dilution Series: For Limit of Detection studies, serial dilutions of patient samples with "known mutations" and "expected frequencies" were used.
Therefore, the ground truth is a combination of established methods, known reference materials, and empirically derived expected values.
8. The Sample Size for the Training Set
The document explicitly mentions training data primarily in the context of the MSI cutoff:
- MSI Cutoff Training: A "training specimen dataset consisting of 138 colorectal cancer (CRC) and 40 endometrial carcinoma (EC) specimens with matched normal and having MSI status results from a validated MSI-PCR or MMR IHC test."
For the mutation calling pipeline (SNVs and indels), the text refers to:
- Optimization of thresholds: "The threshold values for the filtering criteria were established based on paired-sample mutation analysis on replicates of normal FFPE samples, and optimized to reject all false positive SNVs and almost all false positive indel calls from the reference dataset." The size of this "reference dataset" or "replicates of normal FFPE samples" used for training/optimization of filtering thresholds is not explicitly stated as a defined "training set sample size" for the SNV/indel calling. It implies an internal dataset used during development.
9. How the Ground Truth for the Training Set Was Established
For the MSI cutoff training set:
- The ground truth was established by "validated MSI-PCR or MMR IHC test" results. These are existing, established clinical diagnostic methods for determining MSI status.
For the SNV/indel pipeline optimization/threshold establishment:
- The ground truth for optimizing filtering thresholds was based on "paired-sample mutation analysis on replicates of normal FFPE samples" and "reference dataset." This suggests that the "true" status of these calls (i.e., whether they were true positives, false positives, etc.) would have been known or definitively determined through external means (e.g., highly confident calls from a different (perhaps more laborious or deeply sequenced) method, or a known characteristic of the "normal FFPE samples"). However, the specific method for establishing this ground truth for the filtering optimization is not explicitly detailed beyond being from a "reference dataset."
Ask a specific question about this device
Page 1 of 1