Experience sampling methods require more than numbers

Abstract

The experience sampling method (ESM) collects people’s real-time reports about their feelings, actions, and surroundings. While this method originally included both numerical and open-ended responses, most studies today focus only on the numbers. We argue that ESM researchers should collect and analyze open-ended responses again, as they are crucial for understanding what the numbers really mean, and for capturing parts of experience that numbers alone cannot, such as context, the “why” behind responses, and the temporal order of events. Open-ended responses can improve ESM data by grounding it in real-world experiences and phenomena as they are experienced in everyday life. Therefore, handbooks and guidelines on ESM should again include dedicated sections on collecting and analyzing open-ended text items. We highlight future work that is needed to achieve the systematic integration of open-ended items and their analysis into ESM research.

The experience sampling method (ESM) collects real-time reports of people’s feelings, actions, and surroundings, and originally included both numerical and open-ended responses. Whereas most studies today focus only on the numbers, we argue that open-ended responses should be collected and analyzed again, because they are essential for understanding what the numbers mean and for capturing context and the temporal order of experiences.

Introduction

The experience sampling method (ESM)¹, also known as ecological momentary assessment (EMA)^2,3 or ambulatory assessment (AA)⁴, is a research approach designed to capture individuals’ experiences, behaviors, and contexts repeatedly in their everyday environment. Participants typically complete short questionnaires daily or several times per day (usually on smartphones) over an extended period of time (often weeks or months). The use of ESM is particularly prevalent in disciplines such as psychology, health sciences, organizational behavior, and human-computer interaction, where understanding individuals’ context-specific experiences in everyday life is crucial. This methodology has become an influential and rapidly growing area of study⁵. The type of data gathered typically includes individuals’ self-reported experiences of emotions, cognitions, behaviors, physiological sensations, and features of their environmental context (e.g., where they occurred and with whom).

From its inception, ESM was designed to collect both quantitative data and open-ended responses, allowing participants to share their experiences, behaviors, social contexts, and thoughts in their own words¹. For example: “What caused you the most anger today? Briefly describe”⁶. Or: “What have you been doing for the past 15 min?”⁷. Responses to these kinds of open-ended items (see Box 1 for a definition and examples, and Table 1 for an overview of open-ended ESM items) result in what we refer to as qualitative data. Over time, however, this kind of qualitative data has gradually been set aside in the field, which has increasingly focused on collecting responses to closed-ended items that are numeric or easily translated into numbers, including dichotomous scales, Likert scales, visual analog scales, and predefined categories^8,9 — what we refer to as quantitative data¹⁰. Initially, this was partly due to technical reasons, as it was difficult to fill in open-ended items with the personal digital assistants (PDAs) that were used for data collection in the 1990s and 2000s¹¹. This shift to quantitative data is also clearly reflected in ESM handbooks and overview papers. While earlier handbooks and review articles (e.g., Hektner et al.¹¹ or Bolger et al.¹²) extensively cover the collection and coding of open-ended responses, more recent overviews and handbooks (e.g., Mehl & Conner⁸, Bolger & Laurenceau ¹³, or Trull & Ebner-Priemer^4,14,15) provide minimal substantive discussion, reflecting the narrowing focus on quantitative information.

Table 1.

Overview of current practices using open-ended items in ESM

Study element	Decision point	Representative options	Description	Example publication(s)
Data collection	What information to capture?	General event descriptions	Participants recall the most important or emotionally salient event since the last assessment	17,120
		Specific types of events	Participants only report on particular kinds of experiences	20,23
		Immediate context and experience	Participants describe what they are doing, where they are, who they are with, current thoughts and feelings	21
		Response processes	Participants explain how or why they responded as they did	25
	What format to use?	Fully open	Participants are prompted for freely-generated, running text	17
		Fully open (list)	Participants are prompted to provide short phrases or lists	22
		Partially open	Participants are provided with multiple-choice items, including an editable ‘other’ option	121
	How to integrate with other items?	Standalone	Responses can be interpreted independently of other items	21
		Follow-up	Responses expand upon responses to other, closed-ended items	25
	What type of data to collect?	Typed text	Participants write their responses using a computer or phone keyboard	17
		Voice recordings	Participants speak their responses into a phone or recording device	24
		Non-linguistic data	Participants upload additional media (e.g., images)	28
Data analysis	How will data be processed?	Deductive manual coding	Researchers classify participant responses based on established frameworks or prior work	17
		Inductive manual coding	Researchers derive response categories from the data	25
		Combination of deductive and inductive	Researchers add new response categories, based on the data, to a predefined list	6
		Automated coding	Responses are classified or scored using Natural Language Processing techniques, such as word counting programs and Large Language Models	89,120
	How will data be analyzed?	Descriptive statistics	Raw or coded responses are quantitatively characterized	27,122
		Inferential statistics	Coded responses are used to predict outcomes of interest or compare across conditions/groups	6,112
		Qualitative interpretation	Researcher observations are verbally described, guided by example quotes	123
	How will data be presented?	Illustrative quotes	Raw responses are selected to highlight key observations	123
		Visualizations	Distributions of coded responses are illustrated using pie charts, word clouds, Venn diagrams, etc.	22

Despite this apparent decline in methodological attention, open-ended responses continue to be collected in many ESM studies. Our targeted review of published ESM studies and a scan of the ESM item repository (https://esmitemrepositoryinfo.com/; see the Supplementary Note for further details: https://osf.io/jemrp/files/tv94p) indicate that studies have included a broad variety of open-ended items, ranging from more general questions about momentary feelings or recent events to more targeted questions about why participants missed an assessment (i.e., beep or ping¹⁶; see Table 1 for examples). Many studies prompt participants to recall the most important or emotionally salient event since the last beep¹⁷, or to anticipate an upcoming event^18,19. Some studies have focused on specific kinds of events, such as identity-related experiences²⁰ or daily emotional triggers⁶. Open-ended items are also used to capture immediate context, such as what participants are doing, where they are, and who they are with, as well as concurrent thoughts²¹ and feelings²². In some cases, prompts are used to query particular phenomena, such as self-harm²³ or travel experiences²⁴. Relatedly, open-ended items are sometimes used to probe response processes and other measurement-related matters, such as asking participants to explain their ratings^25,26.

The most typical format for open-ended items is free-text boxes¹⁷, although some invite short phrases or lists²². Open-ended items can be stand-alone²¹, or follow up on participants’ responses to closed-ended items²⁵. Responses to open-ended items are most often typed, but several studies have collected voice recordings^24,25,27 or combined open-text data with images, such as photos of meals^28,29 (Box 1). Some studies have also used adaptive, self-learning items that participants can personalize by adding their own response categories^30,31. For example, in response to an item about activities, a participant might add options such as “hanging out with my best friend”, “soccer practice”, or “having patient consultations”, which they can then simply select in future prompts when applicable.

However, open-ended items are seldom analyzed. In our targeted review, we found over 150 ESM publications that included open-ended items, but only 40 that also analyzed and reported them, with 18 of these published after 2020 (Supplementary Note). While this may reflect a renewed interest in open-ended items, the widespread practice of collecting but not analyzing qualitative data warrants reflection. Such practice implies suboptimal resource use³² and even raises ethical questions. Open-ended items increase participant burden, which may affect compliance and data quality³³ (though see refs. ³⁴ and ³⁵). Such a burden may be justified if the data contribute meaningfully to scientific understanding, but it is difficult to justify when open-ended responses are left unanalyzed. At the same time, open-ended responses can lead to valuable insights, and failing to examine them risks losing information that could enrich our understanding of daily life experiences.

We adopt a history and philosophy of science perspective to examine the perks and pitfalls of quantitative methods and to consider the potential of qualitative data to provide detailed insight into experiences in daily life. As there are very few resources explicitly addressing the benefits of open-ended items, we present a case for the use of open-ended responses in ESM. We propose that the recent reemergence of interest in using open-ended ESM items should herald a revival of the method, and discuss what is needed to achieve the systematic integration of open-ended items and their analysis into ESM research.

Box 1 Open-ended ESM items.

Definition	ESM items that provide open space for freely typed text, or for recording audio that is later transcribed into text.
Examples	Text entry	Audio entry
fully open standalone	“What have you done since the previous notification?”89	“Describe where you are, what you are doing, and who you are with”27
fully open follow-up	“What was going on, just before you thought about hurting yourself?”23	“What is the best or most enjoyable part of this experience?”27
Open-ended in disguise	ESM items that are open-ended in form, but are actually used to gather quantitative data, such as the number of times a type of behavior was performed on that day. Such items are not the focus of this article.
Items that involve video recordings, photographs, or drawings could also be seen as open-ended items. However, such approaches remain uncommon and are not the focus of this article.

The enduring influence of quantification

The current literature about ESM has a clear quantitative focus. Review papers, handbooks, and overview articles focus on quantitative data, often collected through Likert or visual analog scales, and on statistical models used to analyze these data^8,9,13,36. For example, valence can be repeatedly measured over time (e.g., with 0–100 visual analog scales) in order to get insight into affect dynamics, which are then typically modeled using multilevel approaches and (multilevel) vector autoregressive techniques^13,37–40. These models are then used to examine how features of these dynamics (e.g., autocorrelation or inertia) relate to outcomes such as depression^41,42. Even suggestions on how to improve ESM measures focus on visual analog or Likert scales and statistics^5,16,43–45. As a result, both reliability and validity are treated primarily as statistical problems: reliability is assessed through increasingly sophisticated statistical models^46–49, and validity is typically examined through statistical associations with other quantitative measures^50,51.

The quantitative focus in ESM research aligns with the general tradition in psychology, where, already in the late 19th century, leading figures such as Gustav Fechner and Wilhelm Wundt were adamant that psychological phenomena could and should be quantified⁵². This belief was influenced by the long history of successes that quantification had brought to the physical sciences in previous centuries. In turn, this led to the idea that quantitative science is real science, making it even more important for psychologists to quantify their subject matter in order to be recognized as a science^52–54. This tradition was particularly strong in early psychometrics, where the focus was on developing standardized tests for education, industry, and the military to order and differentiate between individuals⁵². Accordingly, psychometric tools for evaluating response scales have strongly emphasized calculating reliability, while validity has received much less attention, and generally, the tools for assessing validity have also been quantitative⁵⁵ (see, e.g., ref. ⁵⁶). Quantification was further reinforced by the historical intertwining of psychology and statistics; for example, statistics also developed through psychological research topics such as general intelligence, in which correlational and factor analysis played a key role^57,58.

As is clear from this history, quantification and statistical analysis are often seen as essential for scientific rigor. In psychological science in particular, the idea of objectivity is seen as essential for research quality and is closely linked to the use of statistical methods⁵⁴. A common belief is that quantification and corresponding statistical techniques inherently reduce personal biases and prevent subjective influences from affecting results^59,60. This perspective thus assumes that collecting quantitative data, together with the application of statistical procedures, ensures neutrality by minimizing individual intuition or subjective judgment in research. Furthermore, quantification is often thought to increase precision and exactness^60,61, allowing comparisons across individuals and within individuals across time. It is not only possible to conclude that an individual is happy; rather, we can also compare their happiness to that of others, showing who is the least or most happy, or what the average level of happiness is⁶². Practically speaking, quantification makes it easier to summarize data, identify patterns, and generalize findings, especially with datasets containing a large number of data points (per person), as is typical for ESM studies; this is something far more time-consuming to achieve with qualitative data^54,55,60. Finally, as discussed above, psychometric models used to evaluate construct validity and reliability are generally designed for quantitative data (see also ref. ⁶⁰). Taken together, these factors incentivize the collection, analysis, and validation of (closed-ended) quantitative data instead of (open-ended) qualitative data⁵⁵.

The essential role of qualitative research

The status of qualitative methods in psychology

Despite the dominance and recognized value of quantitative research in psychology, qualitative methods have never been entirely absent⁶³. Although pioneers of psychology such as Wilhelm Wundt, Sigmund Freud, and William James affirmed the scientific value of qualitative approaches, qualitative methods remained largely absent from psychology’s educational curricula throughout much of the 20th century⁶³. As psychology sought to establish itself as an objective science, it was natural to adopt closed-ended scales, such as Likert scales, for self-report data collection (see p. 17 in ref. ³⁶). At the same time, prominent individual researchers such as Gordon Allport⁶⁴, Daniel Brower⁶⁵, Sigmund Koch⁶⁶, and Solomon Asch⁶⁷ continued to explicitly advocate for qualitative methods, even as such methods were sometimes labeled as pseudoscientific (see p. 19 in ref. ³⁶).

A more general push for qualitative methods occurred in the 1980s, as interest in narrative inquiry, in which stories or narratives are used to describe experiences, increased⁶⁸. For instance, Jerome Bruner differentiated two modes of thought: the logical-scientific or paradigmatic and the narrative⁶⁹. He argued that these two ways of knowing are distinct and essential in how humans make sense of the world, yet the narrative has been largely neglected in psychological science. In contrast to abstract logical-scientific knowledge (paradigmatic), narrative knowledge captures the story-based logic humans use to understand their experiences. Scholars like Polkinghorne⁷⁰ continued this line of work, with lasting influences on narrative traditions in psychology and qualitative research^71–73. This narrative turn also led to increased interest in daily diary methods, in which participants record and describe their experiences in logs or diaries³⁶. In line with these developments, Hurlburt^74,75 introduced the “descriptive experience sampling” method, which was an explicitly qualitative form of ESM, aiming to describe experiences instead of quantifying them.

In recent decades, the status of qualitative methods in psychology has gradually moved from being merely recommended to becoming institutionalized and formally recognized (see also Stecyk et al.⁷⁶). For example, the American Psychological Association has a section (APA Division 5) and a journal (Qualitative Psychology) for qualitative research methods⁶³. Notably, current standards for evaluating the validity of psychological questionnaires acknowledge that gathering validity evidence often requires both quantitative and qualitative approaches: “Both qualitative and quantitative sources of evidence are important in evaluating whether items are psychometrically sound…” (American Educational Research Association [AERA]⁷⁷, p. 64; see also Wolf⁵⁵). Despite this, validity evidence remains predominantly quantitative, and forms of validity that require qualitative methods, such as face validity and response process validation (e.g., with cognitive interviewing⁷⁸, or event-centered interviewing⁷⁹), are largely absent in meta-analyses summarizing validity evidence in self-report questionnaires. Even in ESM studies that collect and analyze open-ended items, very few use these to check or improve validity, for example, by asking about the response processes (for exceptions, see, e.g., Schorrlepp et al.²⁶; see also the Supplementary Note). Additionally, there is a lack of clear standards for how qualitative validity evidence should be collected and reported⁵⁵.

Limits of quantitative data in psychology

Close-ended (Likert) scales and quantitative methods are dominating the field of psychology, and are widely believed to produce more rigorous and objective results than qualitative methods (see “The enduring influence of quantification” section). However, this belief may not hold up to scrutiny. Consider, for example, Kahneman’s definition of ‘objective happiness’ as ‘the average of utility over a period of time’, which reflects a strong belief in objective self-report measurement^59,80. Yet whether self-report data are collected in qualitative or quantitative form, they originate from the same source: the participant⁸⁰. In both cases, the participant expresses an internal state, either by choosing a number or by using words and sentences. The primary distinction, therefore, is not one of objectivity versus subjectivity⁵⁹, but one of abstraction. Saying ‘sunlight was streaming through the café window, and I couldn’t stop smiling while talking’ is less abstract than selecting a 5 on a scale measuring how happy you feel right now. Numbers thus compress and reduce experience, making it more straightforward to compare across individuals, but leaving out much of the complexity and nuance that can be expressed in language⁸¹.

Moreover, quantitative data can give the misleading impression that there are no unanswered questions about the psychological realities they represent⁵⁴. Expressing something in numerical form often implies completeness, that the numbers are sufficient and no further exploration is necessary⁵⁴. Yet, psychological phenomena are rarely so neatly bound. Researchers cannot anticipate every relevant factor in advance, and unexpected experiences or motives remain invisible to predefined numbers or categories. For example, Truijens et al.⁸² discuss a case in which a participant started entering random data out of frustration over the fact that the data was not shared with their therapist. Such unexpected occurrences and the reasons behind them cannot be seen in quantitative data alone. Quantification, therefore, restricts researchers to anticipated forms of information and is inherently limited in its ability to detect and explain surprising or novel insights⁸³.

Furthermore, numbers can give a sense of false precision or accuracy, a phenomenon sometimes referred to as spurious precision⁶¹. A participant might select a score of 6 for sad mood, but not because their experience maps exactly onto that number. Rather, it may be because the participant feels that their mood is somewhere between 5 and 8, but not a 4 or a 9, and this nuance is masked because the participant has to select a single number. All of these issues and examples show how numbers may hide underlying problems, reasons, and assumptions, which cannot be detected with quantitative data alone^26,83.

Reasons for including open-ended items in ESM

Due to the limitations of quantitative data, qualitative data are essential to determine the meaning of quantitative responses to self-reports^55,65. That is, we must understand how participants interpret the items and how they arrive at a specific number or category on a response scale⁷⁸. Currently in ESM studies, researchers often simply assume that participants interpret and respond to a questionnaire as intended, without actually verifying it. In practice, participants can interpret items and response scales quite differently from each other, especially when they come from different cultural or social backgrounds (see, e.g., Scollon et al.⁸⁴). For example, in one study, the item “physical activity” meant different things to different participants. Those who had just arrived in the Netherlands described biking as a physical activity, whereas participants who had lived there longer considered it merely a form of transportation²⁶. Similarly, how does a researcher know whether a participant selecting the ‘doing nothing’ option means lying on the couch while listening to music, or literally doing nothing at all^26,31? In the same study²⁶, one participant used 4 as a neutral score, while another used 8 on a scale of 1–10 to express neutrality. Interpretation may also shift over time, a phenomenon known as response shift^26,85. As participants repeatedly complete the same ESM questions, their understanding of emotions or scale endpoints may change. For instance, a day spent alone, not leaving the house, and eating nothing may have originally felt like a 1 (on a happiness scale from 1 to 10), but after the participant experiences a personal loss, this endpoint may seem completely inadequate, as the participant would like to express an even lower score^26,81,85.

Other distortions may result from disengagement, fatigue, or sadness, leading to missing responses or shallow engagement with the items (see, e.g., the example of a participant entering random data mentioned in the previous section). These issues can often be detected only by asking participants, for example, by including open-text boxes in ESM protocols to let participants explain their answers and their context, which can shed light on their interpretation, reasons for skipping entries, or disengagement^16,26,83,86.

A further advantage of using open-ended items is that they provide access to the complexity of lived experience by allowing participants to express themselves in their own words^83,86,87. For example, Hurlburt, Happé, and Frith used Descriptive Experience Sampling with autistic individuals, and the results indicated that their reported inner experience often consisted only of visual imagery, with little or no inner speech or emotional awareness⁸⁸. This finding had potentially significant implications for autism research, suggesting that the difficulty in perspective-taking experienced by some autistic individuals might stem from the absence of internal verbal or emotional representations⁸⁸. Such findings illustrate what is lost when we rely solely on quantitative data: not just nuance, but often also the core phenomena we aim to understand⁸⁶. In the case of autism, researchers might not have studied internal cognitive styles at all, despite their potential to offer essential insights into autistic experience and development.

As the cases discussed above show, open-ended responses are also helpful when it comes to understanding causality, in particular, the why behind certain outcomes, patterns, or anomalies in data, such as missing values or unexpected responses⁸³. Consider, for example, predefined categories for social context and activities. Participants may report, on average, being happier when they are with someone (e.g., “with a friend”) and engaged in “self-care” (another predefined category). However, you may find that a participant who selected these categories still reports being unhappy. An open-text box can help clarify the reasons for this apparent discrepancy: “Friend was doing my hair but burned my skin with the hair curler and I was in tremendous pain. Furthermore, I just do not have a buffer anymore since the financial problems started last month and they are still stressing me” (adjusted example from Stadel et al.⁸⁹). Understanding the unique momentary context and how the person came to such a low score on happiness would have been impossible to derive from predefined categories and the resulting quantitative data alone. Quantitative data are suited for studying increases or decreases in, for example, emotions or behaviors, whereas qualitative data can complement these analyses by offering insights into complex and context-dependent causal processes^83,90.

Furthermore, processes that unfold across different time scales, such as financial problems that occurred last month but are still having an impact on a day-to-day basis, are difficult to capture through quantitative measures alone. Due to their less constrained format, open-ended items in ESM can allow researchers to identify start- and endpoints of activities or processes operating on longer time scales (e.g., a participant describing traveling to another country by plane and then a few days later mentioning the return flight⁸⁹). Additionally, open-ended descriptions can contain information on the temporal order of events (e.g., “after volunteering at [a café] i came back home and took a nap because i was too tired. then after waking up i worked on my thesis, called my dad and now i’m about to make dinner”⁸⁹) or refer to earlier events to provide more context (see the example about financial problems in the previous paragraph).

To conclude, quantitative or categorical ESM items alone cannot fully capture the details and context of daily life. Quantitative data only skim the surface of what people actually experience, and it would not be feasible to list every possible category in advance, as the range of experiences people have is vast and varied⁹¹. As a result, it is unlikely that quantitative ESM data alone can explain why people felt or acted in a particular way⁸³. Instead of trying to capture the vast complexity of different time scales or attempting to account for every possible scenario with numbers and predefined categories, it is more effective to collect qualitative data and build understanding from there^74,90 (see Box 2 for a summary of the reasons for including open-ended items).

Box 2 Reasons for including open-ended items in ESM.

1. Understanding response processes

   1. Item and response scale interpretation
   2. Detecting cultural and contextual differences
   3. Response shifts

2. Detecting and understanding anomalies in data, such as missing values or unexpected responses

3. Capturing the complexity and uniqueness of lived experience

4. Understanding complex and context-dependent causal processes (the why behind outcomes or patterns)

5. Identifying processes at different time scales and their temporal ordering

The future of open-ended items in ESM

ESM research initially included both closed- and open-ended items, but the field has increasingly focused on the former. This dominance of quantitative data is understandable: numbers are easily comparable, patterns can be quickly summarized, and there are well-established methods to analyze the data and evaluate psychometric quality. In line with this, open-ended responses are rarely analyzed or reported, even though many studies still collect them. In general, there is little explicit discussion of the rationale for including or excluding open-ended items, or, when included, why they are not further analyzed or used. One possible contributing factor is the widespread tendency to follow established ESM designs, often rooted in early work such as Delespaul⁹², which may lead researchers to include open-ended items more out of convention than active intention to analyze them. However, there has been a notable shift in recent years in the literature, as the proportion of ESM papers that actually analyze open-ended items has been increasing. Although this is still a very small proportion of all ESM studies, it nevertheless suggests that interest in open-ended items is growing. This brings us back to a central question: should open-ended items be set aside due to the effort they require, or is it time to, at a large scale, collect, code, and analyze them as an integral part of ESM research?

We think the answer to the latter question is a clear “yes”: It is exactly because of the intrinsic limitations of quantitative data that qualitative or open-ended items are essential in ESM. The abstraction inherent in quantitative data should not be mistaken for scientific rigor or objectivity. In the end, both quantitative and qualitative data come from the same source, humans reporting on their experiences. Quantitative data do not provide a more scientific representation of reality; they are simply representations where the richness of experience has been filtered into numbers and predefined categories. In this way, the upside of quantitative data becomes its biggest downside: once data are abstracted, we no longer know how they came to be. Moving too quickly to quantification, or failing to allow space for revision or new information, risks producing “false abstractions”^36,64 and leads away from life as it is lived, from human experiences in all their complexity^36,93. Therefore, careful qualitative description should serve as the foundation on which quantification is built^36,74,93, as already suggested by early proponents of qualitative methods such as Allport⁶⁴, Brower⁶⁵, Koch⁶⁶, and Asch⁶⁷.

When researchers also have access to the qualitative data in open-ended responses, they can furthermore perform the abstraction themselves, making the process more transparent and standardized. Researchers can then document and share how they arrived at particular categories or interpretations, enabling others to assess, question, or reproduce their reasoning (for a discussion, see Braun & Clarke⁹⁴). This approach opens up the possibility of evaluating both the participant’s abstraction (e.g., the number they gave or the category they chose) and the researcher’s, making the interpretive process more transparent and subject to scrutiny, something that is rarely possible with purely quantitative data. In this way, incorporating open-ended qualitative data actually enhances the scientific rigor of ESM studies (see also Douglas⁹⁵ and Kidder & Fine⁹⁶). Moreover, retaining access to different levels of abstraction allows researchers to revisit the data with new research questions in mind.

Qualitative data are also indispensable for addressing aspects of experiences that are hard to quantify: the “why” behind responses, unexpected patterns, and the multiple time scales at which experiences and events unfold^83,90,97. Numerical abstractions are not sufficient for capturing these dimensions. For example, what looks like a stable pattern in the numbers may turn out to mask essential differences. Two participants might both select “3” on a happiness scale from 1 to 5, but in the open-text items, one might describe their mood as “peaceful and content after a walk in nature”, while the other felt “emotionally numb but not actively distressed”. These are very different experiences, which would not be visible from the quantitative data alone. Some form of open-ended items might therefore always be necessary to improve the validity of the quantitative data, to capture the unexpected, and to study patterns, reasons, and phenomena that resist quantification. Thus, we strongly advocate for mixed methods in ESM, collecting qualitative data alongside quantitative data and using them together to enrich the insights drawn from ESM studies^91,96–100.

Suggestions for future research

While the added value of open-ended items is clear, it is less clear how to get the most out of such data, both in terms of collection and analysis. One important opportunity that remains underexplored in the ESM field is how to formulate questions and guide participants toward providing the right level of detail in their responses. As noted by Hektner et al.¹¹, individuals vary in how they describe their daily activities or experiences: some offer detailed accounts, while others remain abstract or vague. Importantly, item wordings should encourage descriptions that are of appropriate length for further qualitative or quantitative analyses, such as natural language processing techniques (e.g., Boyd¹⁰¹). Hektner et al.¹¹ recommend that researchers define their research questions and coding categories in advance whenever possible, so that participants can be appropriately instructed from the start. For example, participants could be encouraged, depending on the study aims, to avoid abstract statements (e.g., “I feel stressed”), to explain the reasons for their feelings, to provide contextual information, or simply to give more details in their answers (see also Fritz et al.¹⁰²).

In addition, including open-ended items alongside predefined activity categories can also be beneficial. This not only allows researchers to see how participants would categorize their own activities, but also helps capture more routine behaviors, such as showering, using the bathroom, or brushing teeth, which participants may overlook or deem too mundane to mention in open-text responses⁸⁹. A combination of predefined categories and open-ended items, therefore, ensures a richer contextual picture without forcing researchers to rely solely on participants’ self-categorization. One potential challenge with this is that participants might also adopt the abstract, predefined categories to describe their experiences in the open-ended items, thereby limiting the richness and spontaneity of their responses. However, preliminary results⁸⁹ suggest that participants do not tend to systematically reduce their responses to overly abstract categories, indicating that this challenge is likely to be manageable in practice.

It is also important to keep in mind that completing open-ended items to obtain qualitative data takes more time than simply responding to Likert-type or visual analog scale items. Fortunately, this burden is partly offset by the fact that many participants value the opportunity to express themselves more freely. Although more systematic research is needed, in many studies where participants received feedback on their qualitative data (e.g., word clouds), compliance was high, and participants continued to provide open-ended responses over an extended time period^{89,97,103–105}. Also, new data collection techniques offer further possibilities for alleviating participant burden, although it should be taken into account that they will result in different kinds of information. For example, voice recordings that are subsequently transcribed into text are perceived by many participants as more convenient than open-ended text items, and tend to result in longer responses⁹⁷. Similarly, participants can create short videos or other multimedia content, which can then be transcribed and coded⁸⁶. ESM measures can also be combined with more passive sensing approaches, such as the Electronically Activated Recorder (EAR), which samples brief snippets of daily life¹⁰⁶, or by tracking what participants are typing on their smartphones in different apps throughout the day¹⁰⁷. The latter especially will result in qualitative data that can complement quantitative ESM data, but will contain different information compared to prompted open-ended items.

Regarding data analysis, given that ESM datasets can be very large, including many participants and up to hundreds of time points, it is also important to look into the new possibilities offered by automated text-analytic tools, such as large language models (LLMs) and other artificial intelligence tools (AI). Such tools can help to lift the burden in analyzing large datasets, for example, by anonymizing¹⁰⁸ and coding^22,109 qualitative responses into predefined categories¹¹⁰. However, it is crucial to approach the use of AI with care (e.g., not simply sharing data with online LLMs owned by private corporations), as well as to keep in mind that the choice of analytic tools must be driven by the research goals and study-specific considerations.

Another important challenge is that ESM researchers often lack the tools and knowledge to effectively analyze open-ended data. Currently, the analysis of open-ended ESM data typically involves coding and counting participants’ categorizations (i.e., applying content analysis, see Hsieh & Shannon¹¹¹), and then applying inferential statistics on them (see Table 1 and Supplementary Note). For example, Delespaul et al.¹¹² collected open-ended responses on the context and activities of schizophrenic vs. non-schizophrenic individuals, and then coded this data into categories to use it in a (multilevel) regression model. While this practice aligns with the overall quantitative orientation of ESM, more advanced qualitative analysis methods for open-ended responses should be considered in future ESM studies, including researchers with experience in these methods as collaborators. This includes qualitative methods such as reflexive thematic analysis¹¹³ or interpretative phenomenological analysis¹¹⁴. For example, the work of Sullivan et al.¹¹⁵ demonstrates how rich insights can be gained from qualitatively analyzing open-ended ESM responses: They collected open-ended descriptions of oppression-based stress experiences in daily life among sexual and gender minority adolescents (SGMA). An inductive content analysis (i.e., not starting from predefined categories) yielded a map of stressors clarifying how both threats and absent safety signals shape SGMA adolescents’ daily lives. As another example, De Smet et al.⁹⁹ used thematic analyses to analyze open-ended momentary and end-of-day responses together with semi-structured interviews to contextualize quantitative ESM data in depressed youth.

A further recommendation and aim for future research is to make open-ended and qualitative data more front and center in endeavors to improve and check quantitative ESM data. After all, how can we refine our measurement if we do not engage with the entire iterative research cycle: conceptualizing the phenomenon, collecting and analyzing data, and evaluating whether our quantitative measures capture the phenomenon adequately^99,116,117? Moreover, qualitative data are not only crucial for validity, but also for improving reliability, especially in ESM, where we must consider whether participants interpret and respond to questionnaires consistently over many time points¹⁶. Currently, open-ended items are only rarely used to check or improve the validity or reliability of the data. Thus, to enhance the quality of ESM data, these kinds of items and other qualitative input should be integrated into ESM studies, so that we can go back and forth between the qualitative data and the numbers, enabling iterative improvement of our measurement^26,99,116 (see also Eisele et al.¹¹⁸, Eronen¹¹⁷, Grunfeld et al.⁸⁶, and Truijens et al.¹¹⁹). For example, open-ended follow-up questions can help clarify which time frame respondents actually use when rating “momentary” emotions or thoughts (e.g., whether they base this on the immediate present, events from the recent past, anticipated future states, or a combination). This kind of information can then be used to adjust item wordings to better capture the intended construct.

Finally, to turn these ideas (see Box 3 for a summary) into reality rather than leaving them as an aspiration, papers like this are not enough. They need to be complemented with overviews and concrete examples of how qualitative data can be used in ESM, such as in Schorrlepp et al.²⁶ and De Smet et al.⁹⁹. There also remains a lack of resources on more advanced topics such as developing and analyzing open-ended ESM items for qualitative research purposes, for instance phenomenological approaches or applying AI tools (e.g., LLMs) to open-ended ESM data. What is further needed are workshops and tutorials, and ideally, formal education on these topics within psychology curricula, starting already at the bachelor’s level, where qualitative methods are underrepresented¹¹⁶. In addition, ESM handbooks and methodological guidelines should include dedicated sections on collecting and analyzing open-ended items in ESM research. Taken together, these steps can provide the foundation to systematically integrate open-ended items and qualitative methods into ESM research.

Box 3 Topics for future research on open-ended items in ESM.

1. Studying what constitutes a high-quality open-ended item and how to formulate open-ended items in ways that elicit the appropriate level of detail and context from participants

2. Determining when and how to combine predefined categories with open-text responses to capture both routine patterns and richer contextual information

3. Studying how participant burden can be reduced while still allowing for rich qualitative data collection, for example, using voice recordings

4. Increasing the efficiency and feasibility of transcribing and (quantitatively) analyzing open-ended responses, for example, using automated text-analytic tools or large language models (LLMs)

5. Implementing qualitative approaches within ESM, such as reflexive thematic analysis or interpretative phenomenological analysis

6. Integrating qualitative data as a standard component of the iterative ESM research cycle, from conceptualizing constructs to evaluating numerical measures

7. Developing concrete tutorials, workshops, and including dedicated sections in ESM handbooks on designing and (qualitatively) analyzing open-ended ESM items

Conclusion

We strongly advocate for the collection and analysis of open-ended items in ESM research, and we hope that handbooks and guidelines on ESM will, in the future, no longer give insufficient attention to them but have dedicated sections on collecting and analyzing open-ended data. Thus, open-ended items should not be included just out of tradition or habit, but should actually be used to improve ESM research and ground our research in the richness of real-world experience, bringing us closer to the phenomena as they are experienced in everyday life.

Author contributions

L.F.B.: Conceptualization, Investigation, Writing – original draft, Writing – review & editing, Project administration. G.R.G.: Conceptualization, Investigation, Writing – original draft, Writing – review & editing. L.S.: Conceptualization, Investigation, Writing – original draft, Writing – review & editing. L.V.D.E.V.: Conceptualization, Investigation, Writing – review & editing. M.M.D.S.: Conceptualization, Investigation, Writing – original draft, Writing – review & editing. C.V.: Conceptualization, Investigation, Writing – original draft, Writing – review & editing. E.R.: Conceptualization, Writing – original draft, Writing – review & editing. J.F.: Conceptualization, Investigation, Writing – review & editing. J.X.C.: Investigation, Writing – review & editing. O.K.: Conceptualization, Investigation, Writing – review & editing. M.I.E.: Conceptualization, Writing – original draft, Writing – review & editing. J.M.C.: Conceptualization, Investigation, Writing – original draft, Writing – review & editing. M.S.: Conceptualization, Investigation, Visualization, Writing – original draft, Writing – review & editing. K.H.: Conceptualization, Investigation, Writing – original draft, Writing – review & editing.

Peer review

Peer review information

Communications Psychology thanks Michael Krämer, Deanna Kaplan, Laura König for their contribution to the peer review of this work. Primary Handling Editor: Jennifer Bellingtier. A peer review file is available.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s44271-026-00416-9.