(211 days)
The Cognoa ASD Diagnosis Aid is intended for use by healthcare providers as an aid in the diagnosis of Autism Spectrum Disorder (ASD) for patients ages 18 months through 72 months who are at risk for developmental delay based on concerns of a parent, caregiver, or healthcare provider.
The device is not intended for use as a stand-alone diagnostic device but as an adjunct to the diagnostic process.
The Cognoa ASD Diagnosis Aid is a software as a medical device (SaMD) that utilizes a machine-learning algorithm that receives independent information from caregivers or parents, trained analysts, and healthcare professionals (HCPs) to aid in the diagnosis of ASD. It consists of multiple software applications and hardware platforms. Input data is acquired via a Mobile App, a Video Analyst Portal, and a HCP Portal.
- . Mobile App: User interface (UI) for the caregiver or parent to upload videos of the patient via Wi-Fi connection and answer questions about key developmental behaviors. Interfaces with Application Programming Interface (API) server for transmission and management of patient data. Compatible with both iOS (versions 12 and 13) and Android platforms (versions 9 and 10).
- Video Analyst Portal: UI for trained analysts to review uploaded patient videos . remotely and answer questions about the patients' behaviors observed in the videos.
- . HCP Portal: UI for the HCP to answer questions about key developmental behaviors for the patient's age group, view device output and access the interactive dashboard to view all patient results, patient videos, answers to questionnaires administered and device performance data. Compatible with computer operating systems macOS (Catalina or Mojave) and Windows 10, and browsers Safari (versions 12 or 13) and Chrome (versions 84 or 85).
Following analysis of the input data, the Cognoa ASD Diagnosis Aid machine-learning algorithm produces a single scalar value between (1) and (6) which is then compared to preset thresholds to determine the classification. If the value is greater than the upper threshold, then the device output is 'Positive for ASD.' If the value is less than the lower threshold, then the device output is 'Negative for ASD.' If the available information does not allow the algorithm to render a reliable result, the device output is 'No Result.'
Here's a breakdown of the acceptance criteria and the study proving the Cognoa ASD Diagnosis Aid meets them, based on the provided text:
Acceptance Criteria and Device Performance
1. Table of Acceptance Criteria and Reported Device Performance
| Metric (Objective) | Acceptance Criteria (Target) | Reported Device Performance (Point Estimate) | 95% Confidence Interval |
|---|---|---|---|
| Positive Predictive Value (PPV) | Greater than 65% | 80.77% (63/78) | 70.27%, 88.82% |
| Negative Predictive Value (NPV) | Greater than 85% | 98.25% (56/57) | 90.61%, 99.96% |
| Sensitivity | (Not explicitly defined as an acceptance criteria but evaluated as a secondary objective) | 98.44% (63/64) | 91.6%, 99.96% |
| Specificity | (Not explicitly defined as an acceptance criteria but evaluated as a secondary objective) | 78.87% (56/71) | 67.56%, 87.67% |
| No Result Rate | (Not explicitly defined as a threshold, but assessed as a primary objective; implies demonstrating a reasonable rate for an aid in diagnosis) | 68.24% (290/425) | 63.58%, 72.64% |
Conclusion on Acceptance: The device successfully met both the primary effectiveness objectives criteria for PPV (80.77% > 65%) and NPV (98.25% > 85%).
2. Sample Size and Data Provenance for the Test Set
- Sample Size:
- Test Set for Analysis: 425 subjects successfully completed both the device assessment and the specialist assessment (clinical reference standard).
- Subjects with Device Output (Positive/Negative for ASD): Of the 425 completers, 135 subjects received a definitive "Positive for ASD" or "Negative for ASD" output from the device. This subset was used to calculate the performance metrics (PPV, NPV, sensitivity, specificity).
- Data Provenance:
- Country of Origin: United States.
- Retrospective or Prospective: Prospective. The study was designed and conducted specifically to evaluate the device.
3. Number of Experts and Their Qualifications for Ground Truth
- Number of Experts: Up to three specialists were involved in establishing the clinical reference standard (ground truth) for each patient. This included a site-specific specialist and one or two central specialist clinicians.
- Qualifications of Experts: The text states they were "specialists" and "specialist clinicians" using the DSM-5 criteria, implying they were qualified healthcare professionals with expertise in diagnosing ASD. While specific years of experience are not provided, their role in making a clinical diagnosis using established criteria suggests appropriate qualifications.
4. Adjudication Method for the Test Set
The adjudication method for establishing the clinical reference standard was a multi-expert consensus approach:
- Initial Diagnosis: A site-specific specialist made an initial diagnosis using DSM-5 criteria.
- First Review: A central off-site reviewing specialist clinician reviewed the case (standardized medical history, physical form, and a video of the diagnostic encounter).
- Agreement: If the first central reviewer agreed with the site diagnosing clinician, the diagnosis was considered validated.
- Disagreement/Second Review: If the first central reviewer disagreed, the case was referred to a second reviewing specialist clinician.
- Resolution: "Majority rule was used to resolve discrepancies between the two central reviewers and the site diagnosing specialist who all evaluated the same subjects." This can be characterized as a 2+1 consensus model (2 central reviewers + 1 site specialist).
5. Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study
- Was an MRMC study done? No, a traditional MRMC comparative effectiveness study was not explicitly described in terms of comparing human readers with AI vs. without AI assistance to measure an "effect size" of improvement.
- Type of Study: The clinical validation study was a prospective, double-blinded, single-arm study evaluating the device's performance against a clinical reference standard. It focused on the standalone performance characteristics of the device as an aid, not directly on the improvement of human readers when assisted by the AI. The human factors study involved HCPs interacting with the device interface and interpreting its outputs, but it wasn't designed as an MRMC to quantify diagnostic improvement with AI.
6. Standalone (Algorithm Only) Performance
- Was a standalone performance study done? Yes, the core clinical validation study effectively evaluated a form of standalone performance of the algorithm's output (Positive/Negative/No Result) against a clinical reference standard. While the algorithm receives inputs from caregivers, trained analysts, and HCPs, the evaluation of PPV, NPV, sensitivity, and specificity is a measure of the algorithm's diagnostic classification performance on the test data. The output classification is solely based on the algorithm's processing of these inputs, without further human modification of the classification itself.
7. Type of Ground Truth Used
- Type of Ground Truth: Expert Consensus (Clinical Reference Standard). This involved the determination of clinical diagnosis based on the majority assessment of up to three specialists using the DSM-5 criteria.
8. Sample Size for the Training Set
- The document does not explicitly state the sample size used for the training set.
- However, it does mention an exclusion criterion for the clinical study: "Subjects whose medical records had been included in any internal Cognoa training or validation sets." This confirms that separate training and validation sets were used, adhering to good machine learning practices, but the specific size of the training set is not provided in this regulatory summary.
9. How Ground Truth for the Training Set Was Established
- The document does not explicitly describe how the ground truth for the training set was established.
- Given the nature of the device and the rigorous establishment of ground truth for the test set (expert consensus using DSM-5), it is highly probable that a similar, robust method involving clinical experts and diagnostic criteria would have been used for the training set ground truth, but the details are not provided in this specific excerpt.
§ 882.1491 Pediatric Autism Spectrum Disorder diagnosis aid.
(a)
Identification. A pediatric Autism Spectrum Disorder diagnosis aid is a prescription device that is intended for use as an aid in the diagnosis of Autism Spectrum Disorder in pediatric patients.(b)
Classification. Class II (special controls). The special controls for this device are:(1) Clinical performance testing must demonstrate that the device performs as intended under anticipated conditions of use, including an evaluation of sensitivity, specificity, positive predictive value, and negative predictive value using a reference method of diagnosis and assessment of patient behavioral symptomology.
(2) Software verification, validation, and hazard analysis must be provided. Software documentation must include a detailed, technical description of the algorithm(s) used to generate device output(s), and a cybersecurity assessment of the impact of threats and vulnerabilities on device functionality and user(s).
(3) Usability assessment must demonstrate that the intended user(s) can safely and correctly use the device.
(4) Labeling must include:
(i) Instructions for use, including a detailed description of the device, compatibility information, and information to facilitate clinical interpretation of all device outputs; and
(ii) A summary of any clinical testing conducted to demonstrate how the device functions as an interpretation of patient behavioral symptomology associated with Autism Spectrum Disorder. The summary must include the following:
(A) A description of each device output and clinical interpretation;
(B) Any performance measures, including sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV);
(C) A description of how the cutoff values used for categorical classification of diagnoses were determined; and
(D) Any expected or observed adverse events and complications.
(iii) A statement that the device is not intended for use as a stand-alone diagnostic.