(147 days)
Yes
The device description explicitly states that SleepStageML is an "Artificial Intelligence/Machine Learning (Al/ML)-enabled software-only medical device" and mentions the use of "deep learning algorithm based on convolutional neural networks" and "machine learning classifier/model".
No.
The device is described as a software-only medical device intended for assisting the diagnostic evaluation of sleep quality by scoring sleep stages. It does not provide any treatment or therapy.
Yes
The device is intended to assist the diagnostic evaluation of sleep quality by analyzing physiological signals and automatically scoring sleep stages from PSG recordings, which is a diagnostic function.
Yes
The device is explicitly described as a "software-only medical device" in both the Intended Use and Device Description sections. It analyzes existing PSG recordings (data files) and outputs annotations, without requiring or including any hardware components for data acquisition or processing beyond standard computing infrastructure.
Based on the provided information, this device is not an IVD (In Vitro Diagnostic).
Here's why:
- IVD Definition: In Vitro Diagnostics are tests performed on samples taken from the human body, such as blood, urine, or tissue, to detect diseases, conditions, or infections. They are used outside of the body (in vitro) to analyze these samples.
- SleepStageML's Function: SleepStageML analyzes physiological signals (EEG) from a polysomnography (PSG) recording. While these signals are derived from the human body, the analysis is performed on the recording of the signals, not on a biological sample taken from the patient.
- Intended Use: The intended use is to assist in the diagnostic evaluation of sleep quality by analyzing PSG recordings. This is a different type of diagnostic process than analyzing biological samples.
Therefore, SleepStageML falls under the category of a medical device that analyzes physiological data, but it does not meet the definition of an In Vitro Diagnostic.
Yes
The letter explicitly states, "SleepStageML includes an authorized Predetermined Change Control Plan (PCCP) that allows for planned updates..." indicating that the FDA has reviewed and authorized the PCCP for this specific device.
Intended Use / Indications for Use
SleepStageML is intended for assisting the diagnostic evaluation by a qualified clinician to assess sleep quality from level 1 polysomnography (PSG) recordings in a clinical environment in patients aged 18 and older.
SleepStageML is a software-only medical device to be used to analyze physiological signals and automatically score sleep stages. All outputs are subject to review by a qualified clinician.
Product codes
OLZ
Device Description
SleepStageML is an Artificial Intelligence/Machine Learning (AI/ML)-enabled software-only medical device that analyzes polysomnography (PSG) recordings and automatically scores sleep stages. It is intended for assisting the diagnostic evaluation by a qualified clinician to assess sleep quality in patients aged 18 and older.
Qualified clinicians (also referred to as clinical users) such as sleep physicians, sleep technicians, or registered PSG technologists (RPSGTs) who are qualified to review PSG studies, provide PSG recordings in European Data Format (EDF) file format through a secure file transfer system to Beacon Biosignals. SleepStageML automatically analyzes the provided PSG recording and return an EDF file containing the original PSG recording with software-generated sleep stage annotations (i.e., Wake (W), non-REM 1 (N1), non-REM 2 (N2), non-REM 3 (N3), and REM (R)) back to the clinical user. The EDF files containing PSG signals as well as sleep stage annotations are referred to as EDF+. The returned EDF+ files can then be reviewed by the qualified clinicians via the users' PSG viewing software. The recordings processed by SleepStageML are level-1 PSG recordings obtained in an attended setting in accordance with American Association of Sleep Medicine (AASM) recommendations with respect to minimum sampling rate, electroencephalography (EEG) channels, and EEG locations. SleepStageML only uses the EEG signals in provided PSGs and does not consider electromyography (EMG) or electrooculography (EOG) signals when performing sleep staging. The sleep stage outputs of SleepStageML are intended to be comparable to sleep stages as defined by AASM guidelines. SleepStageML software outputs are subject to qualified clinician's review.
The intended patients for PSG studies include those who are suspected of having, or have been diagnosed with, a sleep or sleep-related disorder. This could include sleep disordered breathing, insomnia, periodic limb movement disorder, narcolepsy, hypersomnia, or other parasomnias.
SleepStageML uses a deep learning algorithm based on convolutional neural networks, which was trained on a large and diverse set of PSG recordings with sleep staging labels.
Mentions image processing
Not Found
Mentions AI, DNN, or ML
SleepStageML is an Artificial Intelligence/Machine Learning (AI/ML)-enabled software-only medical device
SleepStageML uses a deep learning algorithm based on convolutional neural networks
SleepStageML utilizes machine learning and deep learning in a similar fashion as predicate and reference devices.
SleepStageML includes an authorized Predetermined Change Control Plan (PCCP) that allows for planned updates of the machine learning software device function (ML-DSF) and non-ML algorithmic components to improve sleep staging performance within the existing intended use and indications for use.
Modifications 1 and 2 above would trigger re-training of the machine learning model, while modifications 3 and 4 would not trigger re-training of the machine learning model.
Input Imaging Modality
Polysomnography (PSG) recordings, specifically electroencephalography (EEG) signals.
Anatomical Site
Not Found
Indicated Patient Age Range
18 and older.
Intended User / Care Setting
Qualified clinicians (sleep physicians, sleep technicians, or registered PSG technologists (RPSGTs) who are qualified to review PSG studies) in a clinical environment (Professional Healthcare Facility).
Description of the training set, sample size, data source, and annotation protocol
SleepStageML uses a deep learning algorithm based on convolutional neural networks, which was trained on a large and diverse set of PSG recordings with sleep staging labels.
The recordings used for development were collected from a variety of sources containing a wide range of patient demographics and clinical sites.
Description of the test set, sample size, data source, and annotation protocol
The efficacy of SleepStageML was established through a retrospective pivotal validation study using previously collected clinical polysomnography (PSG) recordings from a representative set of 100 patients. Recordings for inclusion in the validation data set were randomly selected from three level 1 clinical PSG data sources and were required to (1) have all EEG channels recommended by the American Academy of Sleep Medicine (AASM), (2) have EEG sampling rates between 128 and 512Hz, (3) have a recording duration between 4 and 24 hours, and (4) be from patients of least 18 years of age, The included patients spanned a variety of ages (between 19 and 83), were balanced across sex (46 female and 54 male) and were stratified across clinical apnea categories (25 patients in each category, normal (AHI 30)).
Each PSG recording was manually and independently sleep staged by three (3) registered PSG technologists (RPSGTs) each with at least 5 years of experience in clinical scoring of sleep studies. To ensure that the sleep staging outputs obtained from the experts were of sufficient quality, several data-quality and consistency checks were performed. SleepStageML software performance was evaluated against the expert consensus sleep stages that were constructed using 2/3 majority scoring (i.e., the stage per epoch where at least 2 of the 3 experts agree).
Summary of Performance Studies
Study type: Retrospective pivotal validation study.
Sample size: 100 patients.
Key results:
Objective performance targets and acceptance criteria of positive agreement (PA), negative agreement (NA), and overall agreement (OA) were predefined to demonstrate non-inferiority to the identified predicate device, Sleep Profiler (K153412). These metrics were computed for all 30-second epochs that had an expert consensus sleep stage (excluding epochs where all 3 RPSGTs disagreed) after pooling across all recordings in the study.
Comparing the 95% bootstrapped confidence intervals and mean values to the reference values indicates that SleepStageML is non-inferior to the predicate device. Therefore, the results confirm that the positive, negative, and overall agreement performances of SleepStageML are substantially equivalent to those of Sleep Profiler (K153412), respectively.
Key Metrics
| Sleep Staging Comparisons | N=100 subjects | Percent Agreement (%) with 95% percentile bootstrap confidence interval (N=5000 resamples) | N=43 subjects | Point-estimate of Percent Agreement (%) | |||||
|---|---|---|---|---|---|---|---|---|---|
| Total Epochs | Overall Agreement (OA) | Positive Agreement (PA) | Negative Agreement (NA) | Total Epochs | Overall Agreement (OA) | Positive Agreement (PA) | Negative Agreement (NA) | ||
| Overall epochs using 2/3 consensus scoring | W | 21,668 | 96.1% (95.4%, 96.8%) | 88.9% (86.5%, 91.2%) | 98.5% (98.2%, 98.8%) | 7,424 | 89% | 73% | 94% |
| N1 | 3,877 | 94.5% (93.7%, 95.2%) | 58.4% (54.2%, 62.4%) | 96.2% (95.4%, 96.9%) | 1,752 | 89% | 25% | 93% | |
| N2 | 42,587 | 87.1% (85.9%, 88.3%) | 79.8% (77.7%, 81.8%) | 94.2% (93.2%, 95.0%) | 12,582 | 81% | 77% | 84% | |
| N3 | 6,210 | 92.9% (91.8%, 93.8%) | 93.0% (89.8%, 95.7%) | 92.9% (91.7%, 93.9%) | 4,704 | 91% | 76% | 94% | |
| R | 12,641 | 97.3% (96.7%, 97.9%) | 93.1% (91.5%, 94.5%) | 98.0% (97.3%, 98.6%) | 3,749 | 95% | 74% | 97% | |
| Total | 86,983 | Multi-stage Agreement: 84.02% | -- | -- | 31,361 | -- | -- | -- | |
| No Consensus | 2,289 | -- | -- | -- | 1,150 | -- | -- | -- |
Predicate Device(s)
Sleep Profiler (K153412, Advanced Brain Monitoring, Inc.)
Reference Device(s)
SomnoMetry (K221179, Neumetry Medical Inc.)
Predetermined Change Control Plan (PCCP) - All Relevant Information
SleepStageML includes an authorized Predetermined Change Control Plan (PCCP) that allows for planned updates of the machine learning software device function (ML-DSF) and non-ML algorithmic components to improve sleep staging performance within the existing intended use and indications for use. This PCCP allows for the modification of the algorithmic components of SleepStageML including the signal preprocessing, machine learning model, postprocessing, or signal quality check to achieve increased sleep staging performance.
The four modifications are:
- Update of Machine Learning Model: SleepStageML's sleep staging neural network may be modified for the purposes of improving sleep staging performance within the intended use population by: Retraining with an updated training/tuning dataset; Retraining with updated hyper-parameters, loss function, optimizer; Retraining with updated model selection criteria; Retraining with an updated neural network architecture with limitations on model size and type.
- Update of Signal Preprocessing Steps: SleepStageML's EEG signal preprocessing may be modified for the purposes of improving sleep staging performance within the intended use population by: Updating the parameters of the digital signal processing steps (e.g., filtering) applied to the EEG signals before being input to the machine learning model.
- Update of Probability Postprocessing: SleepStageML's probability postprocessing may be modified for the purposes of improving sleep staging performance within the intended use population by: Updating the methods by which sleep stages are generated from the model output sleep stage probabilities.
- Update of Signal Quality Check: SleepStageML's signal quality check component may be modified for the purposes of improving sleep staging performance within the intended use population by: Updating the criteria/thresholds used to check that the input EEG signals are analyzable.
Modifications 1 and 2 above would trigger re-training of the machine learning model, while modifications 3 and 4 would not trigger re-training of the machine learning model.
The testing of any modification to SleepStageML within the scope of the PCCP will include comprehensive software verification and validation testing, including repeating all unit, integration, and system level tests performed in the development for the original SleepStageML software. All software verification tests linked to requirements and specifications must pass for a modification to be considered valid. In addition, clinical performance validation will also be repeated and will require that the performance of any modification to SleepStageML to be non-inferior in per-stage performance metrics to both the original SleepStageML device and its predicate device. In addition, the performance of any modification to SleepStageML must also be non-inferior to the best performance among released versions of SleepStageML with respect to the multi-stage agreement.
The acceptance criteria for the clinical performance validation are:
- The per-stage overall, positive, and negative agreements of the modified SleepStageML, as compared to human expert consensus, is non-inferior to both the Original SleepStageML and the predicate device for full overnight recordings. (Applies to Modifications 1, 2, 3, 4)
- The percentage of human scorable recordings that are unanalyzable by the updated SleepStageML is less than or equal to a predefined threshold. (Applies to Modification 4)
- The per-stage overall, positive, and negative agreements of the modified SleepStageML, as compared to human expert consensus, is non-inferior to both the Original SleepStageML and the predicate device for 2-hour recording segments. (Applies to Modifications 1, 2, 3, 4)
- The per-stage overall, positive, and negative agreements of the modified SleepStageML, as compared to human expert consensus, is non-inferior to both the Original SleepStageML and the predicate device for recordings with the minimum number of AASM recommended EEG channels. (Applies to Modifications 1, 2, 3, 4)
- The overall multi-stage agreement (across all 5 sleep stages) of the modified SleepStageML is non-inferior to the highest overall multi-stage agreement across all released versions of SleepStageML. (Applies to Modifications 1, 2, 3, 4)
Upon a release of an updated version of SleepStageML based on this PCCP, communication will be sent to all external clinical users of SleepStageML, informing them that a new version of SleepStageML is available, with a description of the release and its updated performance.
§ 882.1400 Electroencephalograph.
(a)
Identification. An electroencephalograph is a device used to measure and record the electrical activity of the patient's brain obtained by placing two or more electrodes on the head.(b)
Classification. Class II (performance standards).
0
March 8, 2024
Image /page/0/Picture/1 description: The image shows the logo of the U.S. Food and Drug Administration (FDA). On the left is the Department of Health & Human Services logo. To the right of that is the FDA logo, which is a blue square with the letters "FDA" in white. To the right of the blue square is the text "U.S. FOOD & DRUG ADMINISTRATION" in blue.
Beacon Biosignals, Inc. Delphine Lemoine Senior Regulatory Affairs Specialist 22 Boston Wharf Road 7th Floor, Unit 41 Boston, Massachusetts 02210
Re: K233438
Trade/Device Name: SleepStageML Regulation Number: 21 CFR 882.1400 Regulation Name: Electroencephalograph Regulatory Class: Class II Product Code: OLZ Dated: February 9, 2024 Received: February 9, 2024
Dear Delphine Lemoine:
We have reviewed your section 510(k) premarket notification of intent to market the device referenced above and have determined the device is substantially equivalent (for the indications for use stated in the enclosure) to legally marketed predicate devices marketed in interstate commerce prior to May 28, 1976, the enactment date of the Medical Device Amendments, or to devices that have been reclassified in accordance with the provisions of the Federal Food, Drug, and Cosmetic Act (the Act) that do not require approval of a premarket approval application (PMA). You may, therefore, market the device, subject to the general controls provisions of the Act. Although this letter refers to your product as a device, please be aware that some cleared products may instead be combination products. The 510(k) Premarket Notification Database available at https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm identifies combination product submissions. The general controls provisions of the Act include requirements for annual registration, listing of devices, good manufacturing practice, labeling, and prohibitions against misbranding and adulteration. Please note: CDRH does not evaluate information related to contract liability warranties. We remind you, however, that device labeling must be truthful and not misleading.
If your device is classified (see above) into either class II (Special Controls) or class III (PMA), it may be subject to additional controls. Existing major regulations affecting your device can be found in the Code of Federal Regulations, Title 21, Parts 800 to 898. In addition, FDA may publish further announcements concerning your device in the Federal Register.
FDA's substantial equivalence determination also included the review and clearance of your Predetermined Change Control Plan (PCCP). Under section 515C(b)(1) of the Act, a new premarket notification is not required for a change to a device cleared under section 510(k) of the Act, if such change is consistent with an
1
established PCCP granted pursuant to section 515C(b)(2) of the Act. Under 21 CFR 807.81(a)(3), a new premarket notification is required if there is a major change or modification in the intended use of a device, or if there is a change or modification in a device that could significantly affect the safety or effectiveness of the device, e.g., a significant change or modification in design, material, chemical composition, energy source, or manufacturing process. Accordingly, if deviations from the established PCCP result in a major change or modification in the intended use of the device, or result in a change or modification in the device that could significantly affect the safety or effectiveness of the a new premarket notification would be required consistent with section 515C(b)(1) of the Act and 21 CFR 807.81(a)(3). Failure to submit such a premarket submission would constitute adulteration and misbranding under sections 501(f)(1)(B) and 502(o) of the Act, respectively.
Additional information about changes that may require a new premarket notification are provided in the FDA guidance documents entitled "Deciding When to Submit a 510(k) for a Change to an Existing Device" (https://www.fda.gov/media/99812/download) and "Deciding When to Submit a 510(k) for a Software Change to an Existing Device" (https://www.fda.gov/media/99785/download).
Your device is also subject to, among other requirements, the Quality System (OS) regulation (21 CFR Part 820), which includes, but is not limited to, 21 CFR 820.30, Design controls; 21 CFR 820.90, Nonconforming product; and 21 CFR 820.100, Corrective and preventive action. Please note that regardless of whether a change requires premarket review, the QS regulation requires device manufacturers to review and approve changes to device design and production (21 CFR 820.30 and 21 CFR 820.70) and document changes and approvals in the device master record (21 CFR 820.181).
Please be advised that FDA's issuance of a substantial equivalence determination does not mean that FDA has made a determination that your device complies with other requirements of the Act or any Federal statutes and regulations administered by other Federal agencies. You must comply with all the Act's requirements, including, but not limited to: registration and listing (21 CFR Part 807); labeling (21 CFR Part 801); medical device reporting of medical device-related adverse events) (21 CFR Part 803) for devices or postmarketing safety reporting (21 CFR Part 4, Subpart B) for combination products (see https://www.fda.gov/combination-products/guidance-regulatory-information/postmarketing-safety-reportingcombination-products); good manufacturing practice requirements as set forth in the quality systems (QS) regulation (21 CFR Part 820) for devices or current good manufacturing practices (21 CFR Part 4, Subpart A) for combination products; and, if applicable, the electronic product radiation control provisions (Sections 531-542 of the Act); 21 CFR Parts 1000-1050.
Also, please note the regulation entitled, "Misbranding by reference to premarket notification" (21 CFR 807.97). For questions regarding the reporting of adverse events under the MDR regulation (21 CFR Part 803), please go to https://www.fda.gov/medical-device-safety/medical-device-reportingmdr-how-report-medical-device-problems.
For comprehensive regulatory information about medical devices and radiation-emitting products, including information about labeling regulations, please see Device Advice (https://www.fda.gov/medicaldevices/device-advice-comprehensive-regulatory-assistance) and CDRH Learn (https://www.fda.gov/training-and-continuing-education/cdrh-learn). Additionally, you may contact the Division of Industry and Consumer Education (DICE) to ask a question about a specific regulatory topic. See the DICE website (https://www.fda.gov/medical-device-advice-comprehensive-regulatory
2
assistance/contact-us-division-industry-and-consumer-education-dice) for more information or contact DICE by email (DICE@fda.hhs.gov) or phone (1-800-638-2041 or 301-796-7100).
Sincerely,
Image /page/2/Picture/3 description: The image contains the text "Patrick Antkowiak -S" in black font. To the left of the text is a faded, light blue graphic that appears to be the letters "FDA". The text is arranged in two lines, with "Patrick" on the first line and "Antkowiak -S" on the second line.
for Jay Gupta Assistant Director DHT5A: Division of Neurosurgical, Neurointerventional and Neurodiagnostic Devices OHT5: Office of Neurological and Physical Medicine Devices Office of Product Evaluation and Quality Center for Devices and Radiological Health
Enclosure
3
Indications for Use
Submission Number (if known)
Device Name
SleepStageML
Indications for Use (Describe)
SleepStageML is intended for assisting the diagnostic evaluation by a qualified clinician to assess sleep quality from level 1 polysomnography (PSG) recordings in a clinical environment in patients aged 18 and older.
SleepStageML is a software-only medical device to be used to analyze physiological signals and automatically score sleep stages. All outputs are subject to review by a qualified clinician.
Type of Use (Select one or both, as applicable)
Prescription Use (Part 21 CFR 801 Subpart D)
Over-The-Counter Use (21 CFR 801 Subpart C)
CONTINUE ON A SEPARATE PAGE IF NEEDED.
This section applies only to requirements of the Paperwork Reduction Act of 1995.
DO NOT SEND YOUR COMPLETED FORM TO THE PRA STAFF EMAIL ADDRESS BELOW.
The burden time for this collection of information is estimated to average 79 hours per response, including the time to review instructions, search existing data sources, gather and maintain the data needed and complete and review the collection of information. Send comments regarding this burden estimate or any other aspect of this information collection, including suggestions for reducing this burden, to:
Department of Health and Human Services Food and Drug Administration Office of Chief Information Officer Paperwork Reduction Act (PRA) Staff PRAStaff(@fda.hhs.gov
"An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB number."
4
Image /page/4/Picture/0 description: The image shows the logo for Beacon Biosignals. The logo has two parts: the text on the left and a graphic on the right. The text reads "BEACON" in large, bold, dark gray letters above "BIOSIGNALS" in smaller, lighter gray letters. To the right of the text is a graphic of a brain made of a continuous, squiggly line in a light green color.
Submitter
Name: Beacon Biosignals, Inc.
Contact Person: Delphine Lemoine
Address: 22 Boston Wharf Road, 7th floor, Unit 41, Boston MA, 02210
Telephone: +33 159068311
Date Submitted: October 13, 2023
Subject Device
Trade Name: SleepStageML Common Name: SleepStageML Product Code: OLZ Regulatory Class: II (21 C.F.R. 882.1400) Review Panel: Neurology Predicate Device: Sleep Profiler (K153412, Advanced Brain Monitoring, Inc.) Reference Device: SomnoMetry (K221179, Neumetry Medical Inc.)
Device description
SleepStageML is an Artificial Intelligence/Machine Learning (Al/ML)-enabled software-only medical device that analyzes polysomnography (PSG) recordings and automatically scores sleep stages. It is intended for assisting the diagnostic evaluation by a qualified clinician to assess sleep quality in patients aged 18 and older.
Qualified clinicians (also referred to as clinical users) such as sleep physicians, sleep technicians, or registered PSG technologists (RPSGTs) who are qualified to review PSG studies, provide PSG recordings in European Data Format (EDF) file format through a secure file transfer system to Beacon Biosignals. SleepStageML automatically analyzes the provided PSG recording and return an EDF file containing the original PSG recording with software-generated sleep stage annotations (i.e., Wake (W), non-REM 1 (N1), non-REM 2 (N2), non-REM 3 (N3), and REM (R)) back to the clinical user. The EDF files containing PSG signals as well as sleep stage annotations are referred to as EDF+. The returned EDF+ files can then be reviewed by the qualified clinicians via the users' PSG viewing software. The recordings processed by SleepStageML are level-1 PSG recordings obtained in an attended setting in accordance with American Association of Sleep Medicine (AASM) recommendations with respect to minimum sampling rate, electroencephalography (EEG) channels, and EEG locations. SleepStageML only uses the EEG signals in provided PSGs and does not consider electromyography (EMG) or electrooculography (EOG) signals when performing sleep staging. The sleep stage outputs of SleepStageML are intended to be comparable to sleep stages as defined by AASM guidelines. SleepStageML software outputs are subject to qualified clinician's review.
The intended patients for PSG studies include those who are suspected of having, or have been diagnosed with, a sleep or sleep-related disorder. This could include sleep disordered breathing, insomnia, periodic limb movement disorder, narcolepsy, hypersomnia, or other parasomnias.
SleepStageML uses a deep learning algorithm based on convolutional neural networks, which was trained on a large and diverse set of PSG recordings with sleep staging labels.
5
Image /page/5/Picture/0 description: The image shows the logo for Beacon Biosignals. The logo consists of the word "BEACON" in bold, dark gray letters stacked on top of the word "BIOSIGNALS" in a smaller font and the same color. To the right of the text is a stylized graphic of a brain, depicted as a series of connected, rounded lines in a light teal color. The lines form a continuous, wavy pattern that resembles brainwaves.
The recordings used for development were collected from a variety of sources containing a wide range of patient demographics and clinical sites for robust performance.
Indication for Use
SleepStageML is intended for assisting the diagnostic evaluation by a qualified clinician to assess sleep quality from level 1 polysomnography (PSG) recordings in a clinical environment in patients aged 18 and older.
SleepStageML is a software-only medical device to be used to analyze physiological signals and automatically score sleep stages. All outputs are subject to review by a qualified clinician.
Indication for Use Comparison and Technological Characteristics Comparison
SleepStageML uses an automated algorithm that performs signal preprocessing and categorizes each epoch as one of the defined sleep stages using a machine learning classifier. It has similar intended use and indications for use as the predicate device Sleep Profiler (K153412, Advanced Brain Monitoring, Inc.). The predicate device does not explicitly mention the use of AI/ML, therefore, SomnoMetry (K221179, Neumetry Medical Inc.), which does use a similar Al/ML-based algorithm for staging sleep, was added as the reference device. Verification and validation methods utilized to evaluate safety and efficacy of SleepStageML were consistent with methods used for evaluating the predicate and reference devices. Software documentation was provided as recommended in FDA "Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices" dated June 14th, 2023, and the pivotal clinical validation study confirmed that the SleepStageML AI/ML software performance was substantially equivalent to that of the predicate device.
| Element | Subject device
SleepStageML | Predicate Device
Sleep Profiler
(K153412) | Reference Device
SomnoMetry
(K221179) | Comparison of
Technological
Characteristics |
|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Classification | Class II, OLZ,
Automated Event
Detection Software for
Polysomnograph with
Electroencephalograph | Class II, OLZ,
Automated Event
Detection Software for
Polysomnograph with
Electroencephalograph | Class II, OLZ,
Automated Event
Detection Software for
Polysomnograph with
Electroencephalograph | Same. |
| Indication for use | SleepStageML is
intended for assisting
the diagnostic
evaluation by a
qualified clinician to
assess sleep quality
from level 1
polysomnography
(PSG) recordings in a
clinical environment in
patients aged 18 and
older. SleepStageML is
a software-only
medical device to be
used to analyze
physiological signals
and automatically
score sleep stages. All
outputs are subject to
review by a qualified
clinician. | Sleep Profiler is
intended for use for
the diagnostic
evaluation by a
physician to assess
sleep quality and score
sleep disordered
breathing events in
adults only. The Sleep
Profiler is a software-
only device to be used
under the supervision
of a clinician to analyze
physiological signals
and automatically
score sleep study
results; including the
staging of sleep,
detection of arousals,
snoring and sleep
disordered breathing | SomnoMetry is
intended for use for
the diagnostic
evaluation by a
physician to assess
sleep quality and as an
aid for the diagnosis of
sleep and respiratory-
related sleep disorders
in adults only.
SomnoMetry is a
software-only medical
device to be used to
analyze physiological
signals and
automatically score
sleep study results,
including the staging
of sleep, AHI, and
detection of sleep-
disordered breathing | Similar.
SleepStageML only
assists a qualified
clinician in assessing
sleep staging and does
not include outputs
related to sleep
disordered breathing. |
| | | | | |
| | | events (obstructive
apneas, hypopneas
and respiratory event
related
arousals). Central and
mixed apneas can be
manually marked
within the records | events including
obstructive apneas. It
is intended to be used
under the supervision
of a clinician in a
clinical environment.
All automatically
scored events are
subject to verification
by a qualified clinician. | |
| | Adults only (≥ 18 years
old) | Adults only | Adults only | Similar.
SleepStageML is
cleared for individuals
18 years and older,
while the predicate and
reference devices are
cleared for individuals
adults 22 years and
older. |
| Patient
population | | | | |
| Sleep staging
guidelines | American Academy of
Sleep Medicine scoring
manual and guidelines. | American Academy of
Sleep Medicine scoring
manual and guidelines. | American Academy of
Sleep Medicine scoring
manual and guidelines. | Same. |
| Environment of
use | Professional Healthcare
Facility | Professional Healthcare
Facility | Professional Healthcare
Facility | Same. |
| Score sleep
staging | Yes | Yes | Yes | Same. |
| Algorithm
description | Automated algorithm
that performs signal
preprocessing and
categorizes each
epoch as one of the
defined sleep stages
using a machine
learning classifier. | Automated algorithm
which spectrally
decomposes the EEG
signal, computes
descriptors of sleep
macro- and
microstructure, and
categorizes each
epoch as one of the
defined sleep stages | A broad array of signal
processing, data
indexing, conventional
machine learning and
deep learning
algorithms/approaches
are applied to the raw
signals to derive
actionable clinical
insights. | Same.
SleepStageML utilizes
machine learning and
deep learning in a
similar fashion as
predicate and
reference devices. |
| Physical
Characteristics | User requires a
computer with access
to the internet to
provide original PSG
recordings in EDF files
to and receive returned
EDF+ files with added
software-generated
sleep stage
annotations from
Beacon Biosignals via
the provided secure
file transfer system. | Sleep Profiler will
operate on any
personal computer
with Windows XP
operating system and
at least 2 GB of RAM.
The speed by which a
study will process is
dependent on the
computer processor
and the amount of
RAM | Web-based software
operates in the cloud
with Windows, Mac
OS, or Linux | The subject device
requires that the user
has a computer to
send/receive EDFs
to/from Beacon
Biosignals. The
predicate and
reference devices are
software that are
directly operated by
the user. |
| Software operates
within infrastructure
that requires user
authentication,
encryption of data at-
rest and in-transit,
checksum verifications,
and access controls. | | The person installing
the Java applet must
have administrator
privileges.
User must have rights
to the account and
group to access data
via the portal
The firewall is not
configured to prevent | User authentication
with strong password,
authorization, end to
end SSL encryption,
access controls,
checksum, network
and database controls,
intrusion prevention
system, and
anonymization. | Similar.
SleepStageML
operates within
infrastructure that
implements
cybersecurity
measures similar to the
measures implemented
for the predicate and
reference devices. |
| Cybersecurity | | | | |
6
Image /page/6/Picture/0 description: The image shows the logo for Beacon Biosignals. The logo has the word "BEACON" stacked on top of the word "BIOSIGNALS" on the left side of the image. To the right of the words is a graphic that looks like a brainwave in a light green color.
7
Image /page/7/Picture/0 description: The image shows the logo for Beacon Biosignals. The logo consists of two parts: the text "BEACON BIOSIGNALS" on the left and a stylized brainwave graphic on the right. The text is in a sans-serif font, with "BEACON" in a larger, bolder font than "BIOSIGNALS." The brainwave graphic is a series of connected, rounded lines in a light green color, resembling a sine wave or brainwave pattern.
| | | communication with
access to the server via
ports 22 and 30-29 | | |
|------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------|---------|--------------------------------------------------------------|
| Predetermined
Change Control
Plan (PCCP) | Includes PCCP that
allows for update of
the signal
preprocessing,
machine learning
model, probability
post-processing, and
signal quality check. | No PCCP | No PCCP | Predicate and
reference devices do
not include a PCCP. |
Summary of Tests
Beacon Biosignals conducted the necessary non-clinical testing and clinical validation testing of SleepStageML with results supporting the determination of substantial equivalence. Activities that were conducted included the following:
- . Software verification testing which included software code reviews, automated testing, acceptance testing, labeling review, cybersecurity and data protection, which confirmed that all software requirements are developed as expected.
- . Design validation testing which simulated the intended use to confirm that the endto-end functionality of SleepStageML meets the design requirements.
- . Study that utilized clinical data to demonstrate automatic scoring sleep study performance.
- . Design traceability confirming that all requirement tracing is complete from design inputs to verification/validation and that all risk controls are implemented and effective.
Non-Clinical Tests
Support for the substantial equivalence of SleepStageML was provided by requirements verification, risk management and software testing, at the unit, integration, and system levels. Unit and integration tests verify the functionality for individual software parts; and system-level integration tests cover each specified requirement. System level tests were defined with detailed protocols and objective pass/fail criteria and were clearly documented with test execution results for review. SleepStageML passed all unit, integration, and system level testing demonstrating that all requirements were verified and validated.
Clinical Validation Test
The efficacy of SleepStageML was established through a retrospective pivotal validation study using previously collected clinical polysomnography (PSG) recordings from a representative set of 100 patients. Recordings for inclusion in the validation data set were randomly selected from three level 1 clinical PSG data sources and were required to (1) have all EEG channels recommended by the American Academy of Sleep Medicine (AASM), (2) have EEG sampling rates between 128 and 512Hz, (3) have a recording duration between 4 and 24 hours, and (4) be from patients of least 18 years of age, The included patients spanned a variety of ages (between 19 and 83), were balanced across sex (46 female and 54 male) and were stratified across clinical apnea categories (25 patients in each category, normal (AHI 30)).
Each PSG recording was manually and independently sleep staged by three (3) reqistered PSG technologists (RPSGTs) each with at least 5 years of experience in clinical scoring of
8
Image /page/8/Picture/0 description: The image shows the text "510(K) SUMMARY" in bold, black font. The text is centered and appears to be the title or heading of a document. The letters are capitalized and evenly spaced.
Image /page/8/Picture/2 description: The image shows the logo for Beacon Biosignals. The logo has the word "BEACON" in bold, dark gray letters above the word "BIOSIGNALS" in smaller, lighter gray letters. To the right of the words is a stylized image of a brain in a light teal color, represented by a series of connected curved lines.
sleep studies. To ensure that the sleep staging outputs obtained from the experts were of sufficient quality, several data-quality and consistency checks were performed. SleepStageML software performance was evaluated against the expert consensus sleep stages that were constructed using 2/3 majority scoring (i.e., the stage per epoch where at least 2 of the 3 experts agree).
Objective performance targets and acceptance criteria of positive agreement (PA), negative agreement (NA), and overall agreement (OA) were predefined to demonstrate non-inferiority to the identified predicate device, Sleep Profiler (K153412). These metrics were computed for all 30-second epochs that had an expert consensus sleep stage (excluding epochs where all 3 RPSGTs disagreed) after pooling across all recordings in the study.
Table 1 shows the SleepStageML clinical performance results as compared to the predicate reported results. Comparing the 95% bootstrapped confidence intervals and mean values to the reference values indicates that SleepStageML is non-inferior to the predicate device. Therefore, the results confirm that the positive, negative, and overall agreement performances of SleepStageML are substantially equivalent to those of Sleep Profiler (K153412), respectively.
| Sleep Staging
Comparisons | | N=100
subjects | Percent Agreement (%) with 95%
percentile bootstrap confidence
interval (N=5000 resamples) | | | N=43
subjects | Point-estimate of Percent
Agreement (%) | | |
|--------------------------------------------|-----------------|-------------------|--------------------------------------------------------------------------------------------------|-------------------------------|-------------------------------|------------------|--------------------------------------------|-------------------------------|-------------------------------|
| | | Total
Epochs | Overall
Agreement
(OA) | Positive
Agreement
(PA) | Negative
Agreement
(NA) | Total
Epochs | Overall
Agreement
(OA) | Positive
Agreement
(PA) | Negative
Agreement
(NA) |
| Overall epochs using 2/3 consensus scoring | W | 21,668 | 96.1%
(95.4%,
96.8%) | 88.9%
(86.5%,
91.2%) | 98.5%
(98.2%,
98.8%) | 7,424 | 89% | 73% | 94% |
| | N1 | 3,877 | 94.5%
(93.7%,
95.2%) | 58.4%
(54.2%,
62.4%) | 96.2%
(95.4%,
96.9%) | 1,752 | 89% | 25% | 93% |
| | N2 | 42,587 | 87.1%
(85.9%,
88.3%) | 79.8%
(77.7%,
81.8%) | 94.2%
(93.2%,
95.0%) | 12,582 | 81% | 77% | 84% |
| | N3 | 6,210 | 92.9%
(91.8%,
93.8%) | 93.0%
(89.8%,
95.7%) | 92.9%
(91.7%,
93.9%) | 4,704 | 91% | 76% | 94% |
| | R | 12,641 | 97.3%
(96.7%,
97.9%) | 93.1%
(91.5%,
94.5%) | 98.0%
(97.3%,
98.6%) | 3,749 | 95% | 74% | 97% |
| | Total | 86,983 | Multi-stage
Agreement:
84.02% | -- | -- | 31,361 | -- | -- | -- |
| | No
Consensus | 2,289 | -- | -- | -- | 1,150 | -- | -- | -- |
Table 1: SleepStageML Clinical Performance Results
9
Image /page/9/Picture/2 description: The image shows the logo for Beacon Biosignals. The logo consists of the word "BEACON" in bold, dark gray letters stacked on top of the word "BIOSIGNALS" in a smaller, lighter gray font. To the right of the text is a stylized graphic of a brain, represented by a series of connected, rounded lines in a light teal color. The brain graphic is positioned so that it appears to be emanating from the text.
Conclusion
Non-Clinical verification, validation, and clinical validation testing were conducted in accordance with FDA guidance recommendations to confirm the device design meets all specifications and user needs. SleepStageML has passed all software verification and validation tests and its clinical validation testing results demonstrated effective performance and non-inferiority to predicate device's performance. It is therefore concluded that SleepStageML is substantially equivalent to the predicate device.
Predetermined Change Control Plan
SleepStageML includes an authorized Predetermined Change Control Plan (PCCP) that allows for planned updates of the machine learning software device function (ML-DSF) and non-ML algorithmic components to improve sleep staging performance within the existing intended use and indications for use. This PCCP allows for the modification of the algorithmic components of SleepStageML including the signal preprocessing, machine learning model, postprocessing, or signal quality check to achieve increased sleep staging performance. The four modifications are summarized in Table 2 below.
| # | Modification | Description |
|---|---|---|
| 1 | Update of Machine | |
| Learning Model | SleepStageML's sleep staging neural network may be modified for the | |
| purposes of improving sleep staging performance within the intended | ||
| use population by: | ||
| Retraining with an updated training/tuning dataset Retraining with updated hyper-parameters, loss function, | ||
| optimizer Retraining with updated model selection criteria Retraining with an updated neural network architecture with | ||
| limitations on model size and type | ||
| 2 | Update of Signal | |
| Preprocessing | ||
| Steps | SleepStageML's EEG signal preprocessing may be modified for the | |
| purposes of improving sleep staging performance within the intended | ||
| use population by: | ||
| Updating the parameters of the digital signal processing steps | ||
| (e.g., filtering) applied to the EEG signals before being input to | ||
| the machine learning model | ||
| 3 | Update of | |
| Probability | ||
| Postprocessing | SleepStageML's probability postprocessing may be modified for the | |
| purposes of improving sleep staging performance within the intended | ||
| use population by: | ||
| Updating the methods by which sleep stages are generated | ||
| from the model output sleep stage probabilities | ||
| 4 | Update of Signal | |
| Quality Check | SleepStageML's signal quality check component may be modified for | |
| the purposes of improving sleep staging performance within the | ||
| intended use population by: | ||
| Updating the criteria/thresholds used to check that the input | ||
| EEG signals are analyzable |
Table 2: Summary of PCCP Modifications
Modifications 1 and 2 above would trigger re-training of the machine learning model, while modifications 3 and 4 would not trigger re-training of the machine learning model.
The testing of any modification to SleepStageML within the scope of the PCCP will include comprehensive software verification and validation testing, including repeating all unit, integration, and system level tests performed in the development for the original
10
Image /page/10/Picture/2 description: The image shows the logo for Beacon Biosignals. The logo has the word "BEACON" in bold, dark gray letters on top of the word "BIOSIGNALS" in smaller, light gray letters. To the right of the words is a light green image that looks like a brainwave.
SleepStageML software. All software verification tests linked to requirements and specifications must pass for a modification to be considered valid. In addition, clinical performance validation will also be repeated and will require that the performance of any modification to SleepStageML to be non-inferior in per-stage performance metrics to both the original SleepStageML device and its predicate device. In addition, the performance of any modification to SleepStageML must also be non-inferior to the best performance among released versions of SleepStageML with respect to the multi-stage agreement. The acceptance criteria for the clinical performance validation are summarized in the Table 3 below, as well as the modifications for which each criterion is required to pass.
| Acceptance Criteria | Modifications that
require passing
this criterion |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|
| The per-stage overall, positive, and negative agreements of the modified
SleepStageML, as compared to human expert consensus, is non-inferior to
both the Original SleepStageML and the predicate device for full overnight
recordings. | 1, 2, 3, 4 |
| The percentage of human scorable recordings that are unanalyzable by the
updated SleepStageML is less than or equal to a predefined threshold. | 4 |
| The per-stage overall, positive, and negative agreements of the modified
SleepStageML, as compared to human expert consensus, is non-inferior to
both the Original SleepStageML and the predicate device for 2-hour
recording segments. | 1, 2, 3, 4 |
| The per-stage overall, positive, and negative agreements of the modified
SleepStageML, as compared to human expert consensus, is non-inferior to
both the Original SleepStageML and the predicate device for recordings with
the minimum number of AASM recommended EEG channels. | 1, 2, 3, 4 |
| The overall multi-stage agreement (across all 5 sleep stages) of the modified
SleepStageML is non-inferior to the highest overall multi-stage agreement
across all released versions of SleepStageML. | 1, 2, 3, 4 |
TABLE 3: SUMMARY OF PCCP CLINICAL VALIDATION ACCEPTANCE CRITERIA
Upon a release of an updated version of SleepStageML based on this PCCP, communication will be sent to all external clinical users of SleepStageML, informing them that a new version of SleepStageML is available, with a description of the release and its updated performance.