Evaluating insomnia queries from an artificial intelligence chatbot for patient education

Abstract

Study Objectives:

We evaluated the accuracy of ChatGPT in addressing insomnia-related queries for patient education and assessed ChatGPT’s ability to provide varied responses based on differing prompting scenarios.

Methods:

Four identical sets of 20 insomnia-related queries were posed to ChatGPT. Each set differed by the context in which ChatGPT was prompted: no prompt, patient-centered, physician-centered, and with references and statistics. Responses were reviewed by 2 academic sleep surgeons, 1 academic sleep medicine physician, and 2 sleep medicine fellows across 4 domains: clinical accuracy, prompt adherence, referencing, and statistical precision, using a binary grading system. Flesch–Kincaid grade-level scores were calculated to estimate the grade level of the responses, with statistical differences between prompts analyzed via analysis of variance and Tukey’s test. Interrater reliability was calculated using Fleiss’s kappa.

Results:

The study revealed significant variations in the Flesch–Kincaid grade-level scores across 4 prompts: unprompted (13.2 ± 2.2), patient-centered (8.1 ± 1.9), physician-centered (15.4 ± 2.8), and with references and statistics (17.3 ± 2.3, P < .001). Despite poor Fleiss kappa scores, indicating low interrater reliability for clinical accuracy and relevance, all evaluators agreed that the majority of ChatGPT’s responses were clinically accurate, with the highest variability on Form 4. The responses were also uniformly relevant to the given prompts (100% agreement). Eighty percent of the references ChatGPT cited were verified as both real and relevant, and only 25% of cited statistics were corroborated within referenced articles.

Conclusions:

ChatGPT can be used to generate clinically accurate responses to insomnia-related inquiries.

Citation:

Alapati R, Campbell D, Molin N, et al. Evaluating insomnia queries from an artificial intelligence chatbot for patient education. J Clin Sleep Med. 2024;20(4):583–594.

BRIEF SUMMARY

Current Knowledge/Study Rationale: In the digital era, patients often consult online resources for health information. With the emergence of artificial intelligence models such as ChatGPT, it is necessary to validate the accuracy of the medical information provided for patient queries.

Study Impact: ChatGPT has a very high clinical accuracy in providing specific insomnia-related information and tailoring responses to various populations, indicating its potential utility for patients’ inquiries. Conversely, although ChatGPT exhibits potential in academia, users should exercise caution when using it for references and statistics, because they need to confirm the validity and accuracy of the provided information.

INTRODUCTION

Insomnia, a sleep disorder characterized by difficulties in initiating or maintaining sleep, is increasingly prevalent and affects up to 50–60% of adults.¹ It is associated with adverse outcomes in cardiovascular disease, neurologic diseases, pulmonary complications, and gastrointestinal issues.² Despite clear diagnostic criteria in the Diagnostic and Statistical Manual of Mental Disorders, fifth edition and the International Classification of Sleep Disorders, many patients go undiagnosed for years.^1,3,4

To bridge this gap in health care, emerging technologies such as artificial intelligence are starting to be utilized. For instance, health systems have started incorporating ChatGPT into their electronic medical record systems to help improve clinic workflow and simplify patient discharge instructions.^5,6 Launched in November 2022, ChatGPT is a large language model that uses generative pretrained (GPT) architecture and neural networks to produce human-like textual responses.^7,8 Through qualitative assessment of the model’s coherence and accuracy, this study aims to assess this program’s potential as a credible source for patients seeking information about sleep disorders.

METHODS

This study was exempt from the Thomas Jefferson University’s Institutional Review Board process given the absence of participants’ involvement. ChatGPT’s responses to insomnia-related questions were assessed across 4 query sessions, each comprising an identical set of 20 questions pertaining to insomnia. These queries were designed to examine the chatbot’s adaptability under different conversational contexts: unprompted, patient-friendly responses, physician-level responses, and responses incorporating references and statistics (Table 1).

Table 1.

User prompting phrases used.

Form no.	Form Name	Prompt Used
1	No prompting	No prompting
2	Patient-friendly responses	I am a patient attempting to learn more about insomnia. I am going to ask you 20 questions pertaining to insomnia. Please use language that would be appropriate for my understanding, but do not compromise on the accuracy of your responses. Be as specific as possible in your answers.
3	Physician-level responses	I am a board-certified physician attempting to learn the most up to date information on insomnia. I am going to ask you 20 questions pertaining to insomnia. Please use language that would be appropriate for my expert-level understanding of medical concepts. Be as specific as possible in your answers.
4	Prompting for statistics and references	I am going to ask you 20 questions pertaining to insomnia. For each answer you provide, make sure that you include statistics, numbers, or calculations that are relevant. Your answers should come from published medical literature, which you should cite within your answers.

Prior to asking the set of questions, these prompts were provided to the chatbot to help prime the artificial intelligence model to formulate responses applicable to each scenario.

The merit of each response was appraised by 2 academic sleep surgeons with board certification in both otolaryngology–head and neck surgery and sleep medicine, 1 fellowship-trained and board-certified sleep neurologist, and 2 sleep medicine fellows. Four domains were assessed: clinical accuracy, prompt adherence, appropriate referencing, and statistical precision. A binary grading system was applied in evaluation of each domain assessed: A score of 0 indicated that the response was either factually incorrect, partially correct, or was irrelevant to the prompt; a score of 1 denoted that the answer, based on the grader’s knowledge, was entirely factually correct or relevant to the prompt provided. Additionally, any incorrect of fabricated references or statistics led to an automatic score of 0 (Table 2).

Table 2.

Insomnia queries and ChatGPT’s responses.

Question	Prompt 1	Prompt 2	Prompt 3	Prompt 4
1. What is sleep insomnia?	Insomnia is a sleep disorder in which individuals have difficulty falling asleep, staying asleep, or both, even when given an appropriate opportunity and environment to sleep. This can lead to reduced sleep duration or quality, resulting in impaired daytime functioning.	Insomnia is when you have trouble falling asleep, staying asleep, or both. Even if you’re in bed for a full night, you might not get the restful sleep you need, and this can make you feel tired during the day.	Insomnia is a sleep disorder characterized by persistent difficulty in initiating or maintaining sleep, or experiencing nonrefreshing sleep, despite adequate opportunity for sleep. It can be classified as either acute (short-term) or chronic (long-term). Acute insomnia may last a few days to weeks, while chronic insomnia is defined as disrupted sleep that occurs at least three nights per week and lasts for at least 3 months.	Insomnia is a prevalent sleep disorder characterized by difficulty falling asleep, staying asleep, or waking up too early. According to the American Sleep Association, around 30% of the general population report short-term insomnia, and about 10% have chronic insomnia. [Roth T. (2007). “Insomnia: definition, prevalence, etiology, and consequences.” Journal of Clinical Sleep Medicine]
2. What are common risk factors for sleep insomnia?	Stress: Worries about work, school, health, finances, or family can keep the mind active at night, making it difficult to sleep. Travel or work schedule: Disrupting your body’s circadian rhythms can lead to insomnia. For example, traveling across time zones or working a late or early shift can cause this. Poor sleep habits: Habits like irregular bedtime schedules, naps, eating heavy meals close to bedtime, etc. Eating too much late in the evening: A light snack might help you fall asleep, but heavy meals can cause discomfort and indigestion, making sleep elusive. Mental health disorders: Conditions such as depression, anxiety, or posttraumatic stress disorder can interfere with sleep. Medications: Some drugs can interfere with sleep. Chronic pain and certain medical conditions: Conditions like asthma, diabetes, heart failure, or hyperthyroidism can disrupt sleep. Caffeine, nicotine, and alcohol: All can interfere with sleep. While alcohol might make you feel sleepy, it can disrupt sleep later in the night. Environmental factors: Noisy settings, an uncomfortable bed, or a room that’s too hot or cold can affect sleep.	Stress: Worries about work, family, or other issues can keep your mind active, making it hard to sleep. Bad Sleep Habits: If you go to bed at different times, nap during the day, or do active things before bed like playing video games or exercising, it can affect your sleep. Eating Too Much Before Bed: Having a big meal can make you uncomfortable, and this might keep you awake. Certain Health Issues: Conditions like depression, anxiety, or asthma can make sleep harder. Medications: Some medicines can keep you awake. Caffeine, Nicotine, and Alcohol: These can mess with your sleep. For example, alcohol might make you sleepy but can wake you up later in the night. Older Age: As we get older, changes in our sleep patterns can make insomnia more common.	Psychiatric and Medical Conditions: Anxiety, depression, and certain medical conditions can predispose individuals to insomnia. Stimulant Use: Including caffeine and certain medications. Alcohol and Drugs: Alcohol may cause awakenings in the second half of the night and withdrawal from certain drugs can precipitate insomnia. Travel or Work Schedule: Circadian rhythm disorders related to jet lag or shift work. Poor Sleep Habits: Such as irregular sleep schedules, stimulating activities before bed, or an unfavorable sleep environment. Eating Patterns: Heavy meals before bedtime might cause discomfort and indigestion. Age: Elderly individuals often exhibit changes in sleep patterns. Gender: Hormonal shifts during menstrual cycle, pregnancy, and menopause can affect sleep. Other Sleep Disorders: Such as sleep apnea or restless legs syndrome.	Risk factors include stress, anxiety, depression, certain medications, caffeine, chronic pain, and certain medical conditions. A survey showed that about 33% of people experiencing insomnia attribute it to stress. [National Sleep Foundation. “Sleep in America Poll”. 2005]
3. Are kids, adults, or the elderly more likely to be at risk for sleep insomnia?	Insomnia can occur at any age, but the elderly are often more susceptible. This is due to various factors, including increased likelihood of medical conditions, changes in sleep architecture, and medications that can disrupt sleep.	Everyone can have insomnia, but it’s more common in older people. As we age, changes in our sleep habits and health can make it harder to get a good night’s rest.	Insomnia can affect individuals of all ages, including children. However, the elderly are more likely to experience it, due in part to changes in sleep architecture, increased prevalence of medical conditions, and medications that can disrupt sleep.	While insomnia can occur at any age, it’s more common in adults and the prevalence increases with age. About 25% of children experience behavioral insomnia, while 30–48% of elderly adults report symptoms of insomnia. [Ancoli-Israel, S., & Cooke, J. R. (2005). “Prevalence and comorbidity of insomnia and effect on functioning in elderly populations.” Journal of the American Geriatrics Society]
4. Are males or females more likely to develop sleep insomnia?	Studies have shown that females, especially during hormonal changes such as menstruation, pregnancy, and menopause, are more likely to report insomnia than males. However, the exact reasons for this disparity can vary and can be multifactorial, encompassing biological, psychological, and social factors.	Women are more likely to report insomnia. This might be due to things like hormonal changes during their menstrual cycle, pregnancy, or menopause.	Females are generally more likely to report insomnia, especially during hormonal shifts like menstruation, pregnancy, and menopause. Additionally, postmenopausal women have an increased risk due to decreased levels of progesterone—a sleep-promoting hormone.	Females are generally more likely to report insomnia than males. Studies suggest that between 40 and 59% of postmenopausal women report sleep disturbances. [Kravitz, H. M., & Joffe, H. (2011). “Sleep during the perimenopause: a SWAN story.” Obstetric medicine]
5. What are the symptoms of sleep insomnia?	Difficulty falling asleep at night. Waking up during the night. Waking up too early in the morning. Not feeling refreshed after a night’s sleep. Daytime tiredness or sleepiness. Irritability, depression, or anxiety. Difficulty paying attention, focusing on tasks, or remembering. Increased errors or accidents. Ongoing worries about sleep.	Trouble Falling Asleep: Even if you’re tired. Waking Up Often during the night. Waking Up Too Early and not being able to go back to sleep. Still Feeling Tired after a full night’s sleep. Feeling drowsy or irritable during the day or having trouble concentrating.	Difficulty initiating sleep (prolonged sleep latency). Difficulty maintaining sleep (frequent awakenings or prolonged awakenings). Early morning awakenings with inability to return to sleep. Daytime consequences, including fatigue, mood disturbances, cognitive impairments, and decreased overall well-being.	Symptoms include difficulty initiating sleep, maintaining sleep, early morning awakenings, and nonrestorative sleep. According to the American Sleep Association, up to 35% of adults complain of insomnia symptoms. [American Sleep Association]
6. What are the criteria needed to meet the definition of insomnia sleep disorder?	The specific criteria can vary depending on the diagnostic manual or guidelines being referred to, but the DSM-5 (Diagnostic and Statistical Manual of Mental Disorders, fifth edition) outlines the following criteria for insomnia disorder: -Dissatisfaction with sleep quantity or quality, associated with one (or more) of the following symptoms: -Difficulty initiating sleep. -Difficulty maintaining sleep, characterized by frequent awakenings or trouble returning to sleep after awakening. -Early-morning awakening with the inability to return to sleep. -The sleep disturbance causes significant distress or impairment in social, occupational, educational, academic, behavioral, or other important areas of functioning. -The sleep difficulty occurs at least three nights a week. -The sleep difficulty is present for at least three months. -The insomnia is not better explained by another sleep–wake disorder, mental disorder, the effects of a substance, or coexisting medical condition.	For doctors to say you have insomnia, a few things need to be true: -You have trouble with sleep (like the symptoms mentioned above). -This trouble happens even when you have a chance for a good night’s sleep (like being in a comfy bed). -Lack of sleep is causing problems during your day, like feeling super tired, not being able to concentrate, or being moody. -These sleep troubles happen at least 3 nights a week and have been going on for at least 3 months.	According to the DSM-5, the criteria for insomnia disorder are: -Dissatisfaction with sleep quantity or quality, with one or more of the above symptoms. -The sleep disturbance causes significant distress or impairment in social, occupational, or other important areas of functioning. -The sleep difficulty occurs at least 3 nights a week. -The sleep difficulty is present for at least 3 months. -The insomnia occurs despite adequate opportunity for sleep. -The insomnia is not better explained by another sleep–wake disorder (eg, narcolepsy). -The insomnia is not attributable to the physiological effects of a substance (eg, drug abuse, medication). -Coexisting mental disorders and medical conditions do not explain the predominant sleep complaint	According to the DSM-5, insomnia disorder is characterized by dissatisfaction with sleep duration or quality, with difficulties in initiation, maintaining, or early morning awakenings for at least 3 nights a week for 3 months. This disturbance also causes significant distress or impairment in social, occupational, or other important areas of functioning. Epidemiological studies based on these criteria indicate that about 6–10% of the population meets the criteria for insomnia disorder. [Morin, C. M., & Jarrin, D. C. (2013). “Epidemiology of insomnia: prevalence, course, risk factors, and public health burden.” Sleep medicine clinics]
7. How do you diagnose sleep insomnia?	Clinical Evaluation: This typically involves a thorough medical and sleep history. The individual may be asked about their sleep habits, daily routine, and other symptoms they may be experiencing. It’s essential to determine whether the insomnia might be caused by another condition or whether it’s primary insomnia (ie, insomnia not caused by another disease or condition). Sleep Diary: A person may be asked to keep a sleep diary for 1 to 2 weeks. They would note the time they go to bed, the time they wake up, any nighttime awakenings, and other details about sleep. Physical Examination: This can help rule out other medical problems that might cause insomnia. The doctor might screen for psychiatric disorders and drug and alcohol use. Other Tests: In some cases, blood tests might be done to check for thyroid problems or other conditions that can cause sleep problems.	Talking to You: A doctor will ask about your sleep habits, daily routine, and how you feel during the day. They’ll want to know how long it takes you to fall asleep, how often you wake up, and if you feel refreshed in the morning. Sleep Diary: You might be asked to jot down your sleep patterns for a week or two, like when you go to bed, wake up, and how often you wake up in the night. Physical Check-Up: A doctor may examine you to see if any other health issues might be affecting your sleep	Clinical Interview: A thorough medical, psychiatric, and sleep history is paramount. Understanding the patient’s bedtime routines, sleep environment, and any precipitating factors can aid in the diagnosis. Sleep Diaries: Patients can be asked to maintain a detailed sleep diary over 1–2 weeks. It can help identify patterns and behaviors contributing to insomnia. Questionnaires: Tools such as the Insomnia Severity Index (ISI) can be useful to quantify the severity and impact of insomnia. Physical Examination: Though often normal in primary insomnia, an examination can identify potential secondary causes or other sleep disorders. Polysomnography (PSG) and Actigraphy can be useful in certain cases, particularly when other sleep disorders are suspected.	Diagnosis is primarily based on a clinical interview, which includes a thorough sleep history. Specific questionnaires like the Insomnia Severity Index (ISI) might be used, where a score above 15 usually indicates moderate to severe insomnia. [Bastien, C. H., Vallières, A., & Morin, C. M. (2001). Validation of the Insomnia Severity Index as an outcome measure for insomnia research. Sleep Medicine, 2(4), 297–307]
8. When would a sleep study be indicated for sleep insomnia?	A sleep study, or polysomnography, is not typically the first line of assessment for insomnia. However, it might be recommended if: -The cause of insomnia remains unclear after initial assessment. -There’s a suspicion of another sleep disorder such as sleep apnea, restless legs syndrome, or periodic limb movement disorder. -Initial treatments have failed.	A sleep study, or polysomnography, isn’t the first thing doctors look at for insomnia. But, they might suggest it if: -They’re not sure why you’re having trouble sleeping. -They think you might have another sleep problem like sleep apnea (where you stop breathing for a bit while sleeping) or restless legs. -The treatments you’ve tried aren’t helping.	Polysomnography (PSG) is not typically required for the primary diagnosis of insomnia. However, it can be indicated when: -Another sleep disorder such as sleep apnea or periodic limb movement disorder is suspected. -The insomnia is refractory to treatment. -Parasomnias or other unusual behaviors during sleep are reported. -To differentiate between insomnia and other sleep–wake disorders.	A polysomnogram (sleep study) is generally not required for insomnia. However, it may be indicated when another sleep disorder like sleep apnea is suspected or when the initial diagnosis is unclear. It’s estimated that among chronic insomnia patients, up to 30% might also have sleep apnea. [Beneto, A., Gomez-Siurana, E., & Rubio-Sanchez, P. (2009). Comorbidity between sleep apnea and insomnia. Sleep Medicine Reviews, 13(4), 287–293]
9. What are the complications of not treating insomnia?	Decreased Performance and Alertness: Sleep deprivation can lead to reduced alertness and decreased performance, potentially leading to accidents. Memory and Cognitive Impairment: Reduced ability to concentrate and remember new information. Stress and Reduced Quality of Life: This includes mood disturbances and reduced participation in activities. Risk of Accidents: Especially driving-related. Sleep-deprived individuals are as impaired as those who are intoxicated. Health Impairment: Long-term untreated insomnia can be associated with various health problems, including cardiovascular disease, obesity, diabetes, and a weakened immune system	Feeling Sleepy and Less Sharp: Not enough sleep can make it hard to think or react quickly. Mood Issues: Like feeling down or irritable. Making Mistakes: Whether at work, school, or even driving. Getting Sick More Often: Lack of sleep can make it harder for your body to fend off illnesses. Long-term Health Problems: Like heart disease or diabetes.	Cognitive: Impaired attention, concentration, and memory. Mood Disturbances: Increased risk of anxiety and depression. Physical Health Risks: Elevated risk of chronic conditions such as hypertension, diabetes, obesity, and weakened immune function. Increased Accident Risk: Due to decreased vigilance and prolonged reaction times. Decreased Quality of Life: Including interpersonal difficulties and reduced job performance. Substance Abuse: Some may turn to alcohol or drugs as sleep aids.	Unmanaged insomnia can lead to various complications including increased risk for depression, anxiety disorders, impaired cognitive function, and decreased quality of life. One study found that untreated insomnia can increase the risk of major depression by nearly 40%. [Ford, D. E., & Kamerow, D. B. (1989). Epidemiologic study of sleep disturbances and psychiatric disorders. JAMA, 262(11), 1479–1484]
10. What is the association of insomnia with psychiatric conditions?	Insomnia is often associated with psychiatric conditions. It can be a symptom, a coexisting condition, or even a risk factor for some of them. For example: -Depression and Anxiety: Insomnia is often seen in individuals with depression or anxiety. Sometimes, the onset of sleep problems can predict the development of these disorders. -Post-Traumatic Stress Disorder (PTSD): Individuals with PTSD often experience nightmares and disrupted sleep. -Substance Use Disorders: Alcohol and drugs can disrupt the sleep cycle, leading to insomnia during withdrawal or prolonged use.	Insomnia can be both a symptom and a cause of some mental health issues. People with depression or anxiety often have sleep troubles. If you have a lot of nights without good sleep, it might also increase the risk of developing these conditions. So, sleep and mental well-being are closely tied.	Insomnia has a bidirectional relationship with many psychiatric disorders. -Depression: Up to 90% of patients with depression complain about sleep quality. Insomnia can also be a prodromal symptom or a risk factor for recurrent depression. -Anxiety Disorders: Insomnia can be both a precursor and a symptom. Sleep disturbances often exacerbate anxiety symptoms. -Bipolar Disorder: Insomnia can precipitate mania or hypomania. -Psychotic Disorders: Sleep irregularities are commonly observed in conditions like schizophrenia. -Furthermore, chronic insomnia can independently increase the risk of developing a mood disorder	Insomnia is highly comorbid with psychiatric disorders. Roughly 40% of individuals with insomnia also have a coexisting psychiatric condition. Depression and anxiety are particularly common, with up to 90% of depressed individuals experiencing insomnia. [Staner, L. (2010). Comorbidity of insomnia and depression. Sleep Medicine Reviews, 14(1), 35–46]
11. What is the association of insomnia and heart disease?	Chronic insomnia is associated with a higher risk of developing heart disease. Sleep is essential for various physiological processes, including the healing and repair of heart and blood vessels. Persistent insomnia might increase the risk of heart attack, coronary heart disease, and even heart failure.	Consistent lack of sleep might increase the risk of developing heart problems. Our hearts, like the rest of our bodies, need rest to stay healthy. People with long-term insomnia might have a higher chance of heart attacks or other heart-related issues.	Insomnia, particularly when chronic, has been associated with a higher risk of developing cardiovascular diseases. Mechanisms include increased sympathetic nervous system activity, elevated nighttime blood pressure, increased inflammatory markers, and disruptions in endothelial function. Chronic insomnia has been linked with coronary artery disease, heart failure, and atrial fibrillation	Individuals with chronic insomnia have been shown to be at a higher risk for heart disease. One meta-analysis demonstrated that difficulty initiating sleep, difficulty maintaining sleep, and nonrestorative sleep were associated with a 45%, 30%, and 27% increased risk of heart disease, respectively. [Sofi, F., Cesari, F., Casini, A., Macchi, C., Abbate, R., & Gensini, G. F. (2014). Insomnia and risk of cardiovascular disease: a meta-analysis. European Journal of Preventive Cardiology, 21(1), 57–64]
12. What is the association of insomnia and high blood pressure?	There’s evidence to suggest that chronic insomnia can be associated with an increased risk of high blood pressure (hypertension). Lack of adequate restorative sleep might lead to changes in the body that increase stress hormones, inflammation, and other factors that can raise blood pressure.	If you’re not getting good sleep over a long period, it could lead to higher blood pressure. Our bodies need the downtime during sleep to balance things out, including our blood pressure. Regularly missing out on restful sleep can throw off this balance.	Several studies have suggested a link between insomnia and hypertension. The heightened sympathetic nervous system activity seen in chronic insomnia can contribute to sustained elevations in blood pressure. Insomnia with short sleep duration (typically defined as < 6 hours) has been particularly associated with a higher risk of hypertension. However, the exact nature and strength of this association remain areas of ongoing research	Chronic insomnia has been linked to a higher risk for developing hypertension. For instance, a study found that individuals with insomnia who took longer than 14 minutes to fall asleep had a 300% higher risk of hypertension compared to those without insomnia. [Fernandez-Mendoza, J., Vgontzas, A. N., Liao, D., Shaffer, M. L., Vela-Bueno, A., Basta, M., & Bixler, E. O. (2012). Insomnia with objective short sleep duration and incident hypertension: the Penn State Cohort. Hypertension, 60(4), 929–935]
13. What is the association of insomnia and sleep apnea?	Sleep apnea is a disorder where an individual experiences repeated interruptions in breathing during sleep. People with sleep apnea often experience daytime sleepiness due to interrupted sleep at night. However, they may also have difficulty falling asleep or staying asleep, leading to symptoms of insomnia. Thus, insomnia and sleep apnea can coexist in the same individual, a condition sometimes referred to as “comorbid insomnia.” Treating sleep apnea, often with devices like continuous positive airway pressure (CPAP) machines, can help alleviate the insomnia in these cases.	Sleep apnea is when your breathing stops and starts again while you’re asleep. If you have sleep apnea, you might wake up a lot during the night and then feel really tired during the day. So, while insomnia is about having trouble falling or staying asleep, sleep apnea can cause these problems by disturbing your sleep with those breathing pauses.	Comorbidity: A significant number of patients with obstructive sleep apnea (OSA) report symptoms of insomnia, particularly difficulty maintaining sleep. Conversely, a subset of patients initially diagnosed with insomnia might have undiagnosed OSA. Mechanism: Frequent arousals due to apneic events can fragment sleep, contributing to insomnia symptoms. Additionally, the anxiety of experiencing apneic episodes can further exacerbate insomnia. Clinical Implication: It’s essential to consider a dual diagnosis, especially in insomnia patients who have risk factors for or symptoms of OSA. Treatment of the OSA (eg, with CPAP) often ameliorates some of the insomnia symptoms but may not resolve them entirely.	Among patients with chronic insomnia, up to 30% may also have obstructive sleep apnea (OSA). Insomnia complaints can sometimes mask the presence of OSA, complicating diagnosis. [Beneto, A., Gomez-Siurana, E., & Rubio-Sanchez, P. (2009). Comorbidity between sleep apnea and insomnia. Sleep Medicine Reviews, 13(4), 287–293]
14. What is the association of insomnia and restless legs syndrome?	Restless legs syndrome (RLS) is characterized by an uncomfortable sensation in the legs that gets worse during periods of inactivity and often disrupts sleep. Those with RLS may have difficulty falling asleep due to the compelling need to move their legs. As such, insomnia is a common complaint among individuals with RLS.	Restless legs syndrome (RLS) gives you an urge to move your legs, especially when you’re lying down. This can make it hard to fall asleep. So, someone with RLS might also have symptoms of insomnia because the uncomfortable feeling in their legs keeps them awake.	Coexistence: Insomnia is a common complaint in individuals with RLS. The uncomfortable leg sensations and the urge to move can lead to difficulty initiating sleep. Mechanism: Dopaminergic dysregulation, a central component in RLS pathogenesis, might also affect sleep regulation. Clinical Implication: In patients with insomnia who describe evening or nighttime leg discomfort or an urge to move, a differential diagnosis of RLS should be considered. Treatment of RLS often improves associated insomnia.	Insomnia is commonly seen in individuals with RLS. Approximately 88% of RLS patients report sleep disturbances, often manifesting as difficulty maintaining sleep. [Garcia-Borreguero, D., & Ferini-Strambi, L. (2004). Sleep disturbances in restless legs syndrome. Sleep Medicine, 5(4), 359–361]
15. What is the first line treatment for sleep insomnia?	Cognitive Behavioral Therapy for Insomnia (CBT-I) is often considered the first line treatment for chronic insomnia. It addresses the behaviors and thought patterns that contribute to insomnia and teaches individuals strategies to improve their sleep.	A treatment called Cognitive Behavioral Therapy for Insomnia (CBT-I) is the top recommendation. Instead of taking medicine, you’d work with a therapist to learn better sleep habits and ways to handle things that keep you awake, like stress or worry.	Cognitive Behavioral Therapy for Insomnia (CBT-I): CBT-I is the recommended first-line treatment. It addresses the behaviors and thought processes contributing to insomnia and typically involves multiple components, including stimulus control, sleep restriction, and cognitive restructuring.	Cognitive behavioral therapy for insomnia (CBT-I) is recommended as the first-line treatment for chronic insomnia. This recommendation is supported by meta-analyses indicating that CBT-I can improve sleep in 70–80% of insomnia patients and can be more effective in the long term than medication. [Morin, C. M., Bootzin, R. R., Buysse, D. J., Edinger, J. D., Espie, C. A., & Lichstein, K. L. (2006). Psychological and behavioral treatment of insomnia: Update of the recent evidence (1998–2004). Sleep, 29(11), 1398–1414]
16. What medications can help treat sleep insomnia?	Several medications can be prescribed to treat insomnia: Benzodiazepines: Like temazepam (Restoril), lorazepam (Ativan), and others. They can be habit-forming and may not be suitable for everyone. Nonbenzodiazepine sleep aids: Such as zolpidem (Ambien), eszopiclone (Lunesta), and zaleplon (Sonata). These are sometimes called “z-drugs.” Melatonin receptor agonists: Ramelteon (Rozerem) is an example. Antidepressants: Some antidepressants, like trazodone, are used off-label for insomnia. Over-the-counter sleep aids: Often contain antihistamines. They’re generally not recommended for insomnia due to side effects. Always consult with a health care professional before starting or stopping any medication.	Several meds can help: Benzodiazepines: These are sedatives that can help you sleep but might be habit-forming. Examples include temazepam and lorazepam. Nonbenzodiazepine sleep aids: These help you sleep with fewer side effects. Examples are Ambien (zolpidem) and Lunesta (eszopiclone). Melatonin receptor agonists: This is like Ramelteon, which works with a natural sleep hormone you have. Antidepressants: Some can be used for sleep like trazodone	Benzodiazepines: eg, temazepam, estazolam, triazolam, flurazepam. These enhance GABA activity. They’ve been traditionally used but come with concerns about dependency, tolerance, and side effects. Benzodiazepine Receptor Agonists (Nonbenzodiazepines): eg, zolpidem, eszopiclone, and zaleplon. They tend to have fewer side effects than benzodiazepines but still have some dependency potential. Orexin Receptor Antagonists: eg, suvorexant. These inhibit wake-promoting neuropeptides. Melatonin Receptor Agonists: eg, ramelteon. Antidepressants: Some like doxepin, trazodone, and mirtazapine can be used off-label for their sedative properties. Over-the-counter: Melatonin, though its efficacy is variable	Several FDA-approved medications can help treat insomnia, including benzodiazepine receptor agonists like zolpidem, eszopiclone, and zaleplon. A review suggested that these agents increase total sleep time by about 28–57 minutes and decrease sleep onset latency by approximately 22–36 minutes. [Roth, T., & Roehrs, T. (2007). Efficacy and safety of hypnotic medications in the elderly. Clinical Geriatric Medicine, 23(2), 283–313]
17. What lifestyle changes can help treat sleep insomnia?	Sleep Hygiene Practices: These include keeping a consistent sleep schedule, creating a comfortable sleep environment, and avoiding large meals, caffeine, and alcohol before bedtime. Physical Activity: Regular physical activity can help improve sleep, but try not to exercise close to bedtime. Limiting Naps: If you nap during the day, try to keep it short (20–30 minutes) and avoid napping late in the afternoon. Mindfulness and Relaxation Techniques: Such as meditation, deep breathing exercises, and progressive muscle relaxation can help to relax the mind and body.	Sleep Routine: Going to bed and waking up at the same time every day, even on weekends. Sleep Environment: Make sure your bedroom is dark, quiet, and cool. Limit Naps: Especially late in the day. Be Careful with Food and Drinks: Avoid big meals, caffeine, and alcohol before bedtime	Sleep Hygiene: Maintain a consistent sleep schedule, keep the bedroom dark and cool, avoid screens before bedtime, and reduce caffeine and alcohol consumption. Limit Naps: Especially late in the day. Dietary Considerations: Avoid large meals, caffeine, or alcohol before bedtime. Physical Activity: Regular exercise can enhance sleep quality, though it’s best not to exercise too close to bedtime.	Lifestyle modifications include good sleep hygiene practices: maintaining a consistent sleep schedule, creating a comfortable sleep environment, limiting caffeine and alcohol, and avoiding heavy meals before bedtime. Regular exercise can also be beneficial. For instance, aerobic exercise has been shown to improve sleep quality in older adults with insomnia by nearly 50%. [Reid, K. J., Baron, K. G., Lu, B., Naylor, E., Wolfe, L., & Zee, P. C. (2010). Aerobic exercise improves self-reported sleep and quality of life in older adults with insomnia. Sleep Medicine, 11(9), 934–940]
18. What are the types of therapies that help treat sleep insomnia?	Cognitive Behavioral Therapy for Insomnia (CBT-I): As mentioned earlier, it is the most recommended therapy for insomnia. It combines behavioral treatments, like sleep restriction and stimulus control, with cognitive treatments to challenge and change sleep-disruptive thoughts. Relaxation Techniques: These can reduce anxiety and body tension, making it easier to fall asleep. Biofeedback: Helps individuals recognize and control physiological functions to improve sleep.	Cognitive Behavioral Therapy for Insomnia (CBT-I): As mentioned, this helps you change thoughts and habits that might be causing poor sleep. Relaxation Techniques: Things like meditation or muscle relaxation can help calm your mind and body. Biofeedback: This teaches you to control body functions (like heart rate) that can help with sleep	CBT-I: As mentioned earlier, it’s a structured program that helps address the cognitive and behavioral components of insomnia. Relaxation Therapy: Techniques such as progressive muscle relaxation and deep breathing exercises. Biofeedback: Provides physiological information about muscle tension, brain wave activity, and other vital functions to the patient in real time.	Aside from CBT-I, other therapeutic interventions include relaxation therapy, stimulus control therapy, and sleep restriction therapy. These therapies have been shown to produce clinically significant improvements in sleep for approximately 60% of patients with insomnia. [Morin, C. M., Hauri, P. J., Espie, C. A., Spielman, A. J., Buysse, D. J., & Bootzin, R. R. (1999). Nonpharmacologic treatment of chronic insomnia. Sleep, 22(8), 1134–1156]
19. What can I do to prevent development of sleep insomnia?	Maintaining good sleep hygiene is key. This includes having a regular sleep schedule, creating a bedtime routine, ensuring a dark, quiet, and cool sleep environment, and being cautious with caffeine and alcohol, especially in the evening. Managing stress through relaxation techniques can also be preventive.	Keep a consistent sleep schedule, practice relaxation techniques, avoid caffeine and alcohol close to bedtime, create a comfortable sleep environment, and manage stress. Regular exercise can also help, but try not to do it too close to bedtime.	Adhering to the principles of good sleep hygiene, managing stress, maintaining regular physical activity, avoiding chronic use of alcohol or sedative medications for sleep, and addressing any underlying mental health conditions can all play a preventative role.	Maintaining good sleep hygiene, managing stress, and early intervention for acute insomnia (which, if left untreated, can transition to chronic insomnia in 20–30% of cases) are crucial preventive steps. [Ellis, J. G., Gehrman, P., Espie, C. A., Riemann, D., & Perlis, M. L. (2012). Acute insomnia: current conceptualizations and future directions. Sleep Medicine Reviews, 16(1), 5–14]
20. When do I need to see a sleep medicine specialist?	Consider seeing a sleep specialist if: -Your insomnia persists despite trying various self-help strategies. -You experience other symptoms of sleep disorders, such as loud snoring (which might suggest sleep apnea) or involuntary leg movements. -Your insomnia is causing major disruptions in your daily life, like affecting your work, studies, or relationships. -You have other health conditions that might be affecting your sleep. -Over-the-counter remedies or medications you’re taking for other issues might be causing your sleep problems.	If you’ve tried several ways to improve your sleep and nothing’s worked, it might be time. Also, if you have other symptoms like snoring really loud (which could be sleep apnea) or uncontrollable leg movements (like RLS), you should see a specialist. They’re experts in sleep problems and can give you a more in-depth evaluation.	When primary treatments (such as CBT-I or basic pharmacotherapy) are ineffective. - If there’s a suspicion of another underlying sleep disorder (eg, sleep apnea, parasomnias). - If the patient has complex medical comorbidities that complicate the insomnia presentation or treatment. - When considering advanced treatments, like certain medications which may have more significant side effects or the use of actigraphy or polysomnography	If symptoms persist despite trying lifestyle modifications, if sleep disturbances interfere with daily activities, or if there are symptoms suggestive of another sleep disorder (eg, sleep apnea or RLS), then seeing a specialist would be warranted. This decision should be made in collaboration with a primary care physician.

For assessing readability, the Flesch–Kincaid (FK) grade-level score was calculated for each generated response. An analysis of variance and Tukey’s multiple comparisons test were conducted to analyze differences in grade level across the various prompts. Interrater reliability, a statistical measure of consistency, was quantified using Fleiss’s kappa with predefined thresholds to categorize the strength of agreement among evaluators: less than 0.20 indicated poor agreement, 0.21–0.40 denoted fair agreement, 0.41–0.60 represented moderate agreement, 0.61–0.80 signified good agreement, and 0.81–1.00 equated to very good agreement. Statistical analyses were completed with the Statistical Package for the Social Sciences 28.0 (IBM Corp., Armonk, New York).

RESULTS

The Fleiss kappa scores across the various prompts for clinical accuracy and relevance were poor, calculated at less than 0.20. Nevertheless, evaluators uniformly attested to the high clinical accuracy of ChatGPT’s responses across the first 3 forms (> 85%), with Form 4 exhibiting a broad spectrum of accuracy (65–100%). Relevance to the prompts was consistently high across all forms. Notably, 80% of the cited references were both real and pertinent to the given responses, whereas only 25% of the referenced statistics were directly corroborated by the articles cited (Table 3). Figure 1 provides heat maps to visually represent the evaluators’ scoring for the responses generated by ChatGPT. The mean FK grade-level scores across the prompts were 13.2 ± 2.2 (unprompted), 8.1 ± 1.9 (patient-centered), 15.4 ± 2.8 (physician-centered), and 17.3 ± 2.3 (with reference and statistics). Analysis of variance revealed a statistically significant difference in mean FK grade-level scores between the groups (P < .001). The post hoc analysis revealed significant differences between various pairings: unprompted and patient-centered (P < .001), unprompted and physician-centered (P = .015), and unprompted and reference and statistics (P < .001), as well as between patient-centered and physician-centered (P < .001), patient-centered and reference and statistics (P < .001), and physician-centered and reference and statistics (P = .049).

Table 3.

Readability and interrater reliability scores.

	Prompt 1	Prompt 2	Prompt 3	Prompt 4
Average Flesch–Kincaid score	13.2 ± 2.2	8.1 ± 1.9	15.4 ± 2.8	17.3 ± 2.3
Clinical accuracya	0.141 (0.002 to 0.279)	0.185 (0.046 to 0.323)	0.141 (0.002 to 0.279)	0.176 (0.038 to 0.315)
Relevancea	—	−0.020 (−0.159 to 0.118)	N/Ab	N/Ab
Percent agreement (CA)	97%	92%	97%	85%
Percent agreement (relevance)	—	98%	100%	100%
Evaluator 1 grading (CA)	100%	95%	95%	100%
Evaluator 2 grading (CA)	95%	95%	100%	100%
Evaluator 3 grading (CA)	95%	90%	95%	85%
Evaluator 4 grading (CA)	100%	85%	95%	65%
Evaluator 5 grading (CA)	95%	95%	100%	75%

^a Fleiss kappa (95% confidence interval).

^b All ratings were the same, so no Fleiss kappa could be calculated. CA = clinical accuracy.

Figure 1. Sleep insomnia query scoring heat maps.

A heat map displaying the assessment by the reviewers of ChatGPT’s responses. Red cell = 0 (did not meet criteria). Green cell = 1 (met criteria). CA = clinically accurate, F1 = Form 1, F2 = Form 2, F3 = Form 3, F4 = Form 4, R = relevance, Ref = reference, Stats = statistics.

DISCUSSION

In the current study, we assessed ChatGPT’s capability to provide factually correct answers to general insomnia queries related to epidemiology, complications, and treatment options. We found that ChatGPT generates accurate information, somewhat influenced variably by the specificity of prompts provided. The nature of prompts significantly influenced the scientific depth of the responses, as discerned by significant differences in the FK grade-level scores. Greater FK scores denote an advanced educational experience required to comprehend the responses, which could limit its understandability to the general population.

ChatGPT’s ability to cite reputable references was deemed satisfactory, albeit with a caveat: All the sources referenced were predated to 2015 or earlier. This aligns with the model’s training on data sets available until 2021, which restricts its capacity to deliver the latest information. Furthermore, ChatGPT exhibited limitations in presenting statistical data, attributable to its inability to process online journal articles. This necessitates caution among users when using ChatGPT for academic or scientific purposes.

The emergent domain of artificial intelligence has introduced large language models such as ChatGPT into an array of sectors, including health care education, clinical practice, and research. Despite their broad-ranging uses—such as conducting literature reviews, enhancing language to convey research, generating efficient discharge summaries, improving health literacy, and creating clinical vignetters—their inherent limitations cannot be overlooked.^9–13 These include the generation of incorrect information and citations, lack of up-to-the-minute knowledge, and notable issues with transparency and bias.^10,14,15

Although limited for academic use, large language models, such as ChatGPT, offer promise for patients’ education. These models can distill complex medical terminologies into accessible language, thus facilitating patients’ comprehension of their health conditions and treatment plans.^12,16–18 This is reflected in our study, where ChatGPT, when prompted, was able to generate patient-friendly answers that were readable at the 8th-grade level. Moreover, the 24/7 availability of ChatGPT to provide tailored responses makes it a consistent, readily accessible source of information. However, it should be noted that whereas ChatGPT can be a valuable adjunct in patient education it should not replace the direct, empathic human communication and interaction that is paramount in health care.¹⁵ As such, it is crucial to ensure the information provided by the model is accurate, comprehensible, and appropriately validated by health care professionals.

This study is not without its limitations. Foremost, ChatGPT is not officially approved for medical applications and depends on potentially outdated training data, demonstrated by the studies it referenced dating back to 2015 or earlier. Additionally, the model’s capacity to access current literature is contingent upon developers’ indexing, inhibiting its ability to remain up-to-date.¹⁹ Finally, potential biases could be introduced due to the application of a subjective, unverified grading scale. Future research endeavors should consider employing a patient-focused methodology, wherein the potential usage of ChatGPT by patients is explored along with their query preferences. Upon collating a comprehensive list of questions, it would be beneficial to conduct a multicenter interrater reliability study to critically assess the clinical accuracy of ChatGPT’s responses. This approach not only aids in augmenting the validity of findings but also provides a broader, more robust understanding of the language model’s potential role in patient education.

CONCLUSIONS

The results of this investigation affirm that ChatGPT possesses the capacity to generate clinically pertinent and accurate information in response to prevalent insomnia-related inquiries. However, the ability of the language model to draw upon contemporary references and accurately extrapolate statistics from those sources remains a considerable challenge. With further training and software updates, artificial intelligence platforms could potentially augment patients’ education and optimize clinical communication, thus opening a promising vista in digital health care.

DISCLOSURE STATEMENT

All authors have seen, read, and approved the manuscript. The authors report no conflicts of interest.

ABBREVIATIONS

FK

Flesch–Kincaid