(125 days)
The Tyto Insights for Crackles Detection is an over-the-counter artificial intelligence (AI) enabled decision support software system used in the evaluation of lung sounds in adults and pediatrics (2 years and older). It automatically analyzes the acoustic signal of the lung as recorded by the FDA 510k cleared compatible Tyto Stethoscope and identifies recordings where a specific abnormal lung sound suggestive of "Crackle" is suspected. It is not intended to detect other abnormal or normal lung sounds. A licensed health care professional’s advice is required to understand the meaning of the Tyto Insights for Crackles Detection result. Healthcare providers should consider the device result in conjunction with recording and other relevant patient data.
The Tyto Insights for Crackles Detection is a web-based (AI) enabled software system designed to aid in the clinical assessment of lungs auscultation sound data by analyzing recorded lung sounds to determine whether a Crackle is detected within the recorded sound data. The Tyto Insights for Crackles Detection Software is intended to process recordings from the FDA-cleared compatible Tyto Stethoscope (Tyto Stethoscope, K181612). The acquisition of the acoustic data (recordings) is carried out by a professional user in a clinical environment or by a lay- user in a non-medical environment, in compliance with the labeling of the Tyto Stethoscope. The system is composed of the following sub-systems:
- The Tyto Insights for Crackles Detection Application Server (APS) communicates with 1. the Tyto Insights for Crackles Detection Algorithm Server (ALS) and implements an application programming interface (API) for communication with the telehealth server.
- The Tyto Insights for Crackles Detection Algorithm Server (ALS) receives an audio file 2. as input and returns an analysis result of positive or negative regarding whether a Crackles was detected as output.
- The Tyto Insights for Crackles Detection Web Server (WBS) provides a graphic 3. indication whether a Crackles is detected in the recording. It can be utilized both in patient and clinician side.
All the software subsystems (servers and storage) are hosted in the cloud and communicate through IP network.
Here's a breakdown of the acceptance criteria and the study proving the device meets them, based on the provided FDA 510(k) summary for "Tyto Insights for Crackles Detection":
Product Information:
- Trade/Device Name: Tyto Insights for Crackles Detection
- Regulation Number: 21 CFR 868.1900
- Regulation Name: Diagnostic Pulmonary-Function Interpretation Calculator
- Regulatory Class: Class II
- Product Code: PHZ
- Intended Use/Indications for Use: An over-the-counter AI-enabled decision support software system for evaluating lung sounds (adults and pediatrics 2+ years) recorded by the compatible Tyto Stethoscope. It identifies recordings where "Crackle" is suspected. It is not intended to detect other abnormal/normal lung sounds. Requires a licensed healthcare professional's advice to interpret results, which should be considered with other patient data.
1. Table of Acceptance Criteria and Reported Device Performance
| Parameter | Acceptance Criteria (from PCCP) | Reported Device Performance (Stand-Alone) | Reported Device Performance (Clinical Accuracy vs. Readers) |
|---|---|---|---|
| Co-Primary Endpoints | |||
| Sensitivity (Se) | LCI > 0.6279 (for modifications) | 0.72 (0.63-0.79) | Not directly comparable (Clinical Readers AUC is a composite measure) |
| Specificity (Sp) | LCI > 0.9668 (for modifications) | 0.99 (0.97-1.00) | Not directly comparable (Clinical Readers AUC is a composite measure) |
| AUC (Area Under the Curve) | Not explicitly defined as a direct acceptance criterion for the initial submission, but non-inferiority margin used for comparison. | Not applicable (standalone metrics are Se, Sp, PPV, NPV) | Tyto Insights for Crackles Detection AUC: 0.97 (0.95–0.98) |
| Difference in AUC | Lower bound of 95% two-sided CI for (Device AUC - Clinical Readers AUC) > -0.05 (non-inferiority margin) | Not applicable | 0.2 (0.17–0.23). Meets criterion: 0.17 is > -0.05. |
| Secondary Endpoints | |||
| Positive Predictive Value (PPV) | (for modifications) | 0.63 (0.4-0.87) | Not applicable |
| Negative Predictive Value (NPV) | (for modifications) | 0.99 (0.98-0.99) | Not applicable |
| Repeatability (Software vs. Readers) | Not explicitly defined | Software kappa: 1.0, agreement: 100% | Readers kappa: 0.42 (0.35 -0.49) |
Note: The acceptance criteria for Sensitivity and Specificity (LCI > 0.6279 and LCI > 0.9668, respectively) are specifically laid out in the "Predetermined Change Control Plan (PCCP)" section for modifications to the device. For the initial submission, the primary endpoint focused on the non-inferiority of the device's AUC compared to clinical readers. The reported standalone sensitivity and specificity are actual performance metrics from the validation study.
2. Sample Size Used for the Test Set and Data Provenance
- Test Set Sample Size: 446 recordings (120 Crackles positive, 326 Crackles negative). This corresponded to a total of 445 patients.
- Data Provenance: Retrospective validation dataset. Recordings were obtained from the real-world use of the Tyto Care FDA-cleared compatible Tyto Stethoscope (K181612). The document does not specify the country of origin for the data.
3. Number of Experts Used to Establish the Ground Truth for the Test Set and Qualifications of those Experts
- Number of Experts: Three (3) blinded experienced Pulmonologists.
- Qualifications: "Experienced Pulmonologists" are specified. Further details on years of experience or specific board certifications are not provided in the summary.
4. Adjudication Method for the Test Set
- Adjudication Method: Binary ground truth was determined by a majority vote of the three blinded Pulmonologists. This is a form of 3+ Consensus.
5. If a Multi-Reader Multi-Case (MRMC) Comparative Effectiveness Study was done, If so, what was the effect size of how much human readers improve with AI vs without AI assistance
- MRMC Study Done?: A comparative effectiveness study was done comparing the device (AI) performance to "clinical readers" (Physicians non-Pulmonologists). However, this was an AI-only vs. human-only comparison, not a "human reading with AI assistance vs. human reading without AI assistance" MRMC study.
- Effect Size of Human Improvement with AI: This study did not assess how much human readers improve with AI assistance. It directly compared the AI algorithm's performance to human clinical readers.
- Device AUC: 0.97 (0.95–0.98)
- Clinical Readers AUC: 0.77 (0.73–0.8)
- Difference in AUC (Device - Clinical Readers): 0.2 (0.17–0.23). This indicates the AI algorithm significantly outperformed the clinical readers in this specific comparison, establishing non-inferiority (and superiority) based on the defined margin.
6. If a Standalone (i.e., algorithm only without human-in-the-loop performance) was done
- Standalone Study Done?: Yes.
- Performance Metrics:
- Sensitivity: 0.72 (0.63-0.79)
- Specificity: 0.99 (0.97–1.00)
- Positive Predictive Value (PPV): 0.63 (0.4-0.87)
- Negative Predictive Value (NPV): 0.99 (0.98-0.99)
7. The Type of Ground Truth Used
- Type of Ground Truth: Expert Consensus. Specifically, a majority vote of three blinded experienced Pulmonologists on the presence or absence of "Crackle" in the lung sound recordings.
8. The Sample Size for the Training Set
- The document does not specify the exact sample size for the training set. It mentions that the device utilizes a CRNN (Convolutional Recurrent Neural Network) model and that "Each network is trained based on the target clinical class." However, it only provides details for the retrospective validation dataset.
9. How the Ground Truth for the Training Set Was Established
- The document does not explicitly describe how the ground truth for the training set was established. It primarily focuses on the ground truth establishment for the validation set used for performance evaluation. It's generally assumed that similar expert-driven annotation methods would be used for training data, but this is not detailed in the provided summary.
§ 868.1900 Diagnostic pulmonary-function interpretation calculator.
(a)
Identification. A diagnostic pulmonary-function interpretation calculator is a device that interprets pulmonary study data to determine clinical significance of pulmonary-function values.(b)
Classification. Class II (performance standards).