The first step is the hardest: pitfalls of representing and tokenizing temporal data for large language models

Abstract

Objectives

Large language models (LLMs) have demonstrated remarkable generalization and across diverse tasks, leading individuals to increasingly use them as personal assistants due to their emerging reasoning capabilities. Nevertheless, a notable obstacle emerges when including numerical/temporal data into these prompts, such as data sourced from wearables or electronic health records. LLMs employ tokenizers in their input that break down text into smaller units. However, tokenizers are not designed to represent numerical values and might struggle to understand repetitive patterns and context, treating consecutive values as separate tokens and disregarding their temporal relationships. This article discusses the challenges of representing and tokenizing temporal data. It argues that naively passing timeseries to LLMs can be ineffective due to the modality gap between numbers and text.

Materials and methods

We conduct a case study by tokenizing a sample mobile sensing dataset using the OpenAI tokenizer. We also review recent works that feed timeseries data into LLMs for human-centric tasks, outlining common experimental setups like zero-shot prompting and few-shot learning.

Results

The case study shows that popular LLMs split timestamps and sensor values into multiple nonmeaningful tokens, indicating they struggle with temporal data. We find that preliminary works rely heavily on prompt engineering and timeseries aggregation to “ground” LLMs, hinting that the “modality gap” hampers progress. The literature was critically analyzed through the lens of models optimizing for expressiveness versus parameter efficiency. On one end of the spectrum, training large domain-specific models from scratch is expressive but not parameter-efficient. On the other end, zero-shot prompting of LLMs is parameter-efficient but lacks expressiveness for temporal data.

Discussion

We argue tokenizers are not optimized for numerical data, while the scarcity of timeseries examples in training corpora exacerbates difficulties. We advocate balancing model expressiveness and computational efficiency when integrating temporal data. Prompt tuning, model grafting, and improved tokenizers are highlighted as promising directions.

Conclusion

We underscore that despite promising capabilities, LLMs cannot meaningfully process temporal data unless the input representation is addressed. We argue that this paradigm shift in how we leverage pretrained models will particularly affect the area of biomedical signals, given the lack of modality-specific foundation models.

Introduction

In recent years, large language models (LLMs)—also known as foundation models¹—have garnered attention for their ability to generalize across a wide array of tasks. From natural language understanding to generating creative text, these models have demonstrated their prowess, often mimicking human-like performance. Unlike previous AI models, prompting enables example-based learning without any additional training (a concept known as in-context learning²). As their capabilities have grown, so too the idea of employing LLMs as general-purpose assistants has emerged, promising enhanced user experiences through personal devices.

However, amidst these advancements, a significant challenge arises when attempting to use existing LLMs with numerical and temporal data, particularly those originating from mobile sensors or medical timeseries. Perhaps overestimating LLMs’ reasoning abilities, users are writing increasingly complex prompts combining, eg, text and data from lab tests or other measurements, expecting that models will be able to process them correctly. These data sources, often representing fine-grained behavioral and physiological states, introduce a complex layer of information that cannot be easily “translated” into text. LLMs, designed primarily for processing natural language, exhibit remarkable efficiency in representing textual input through specialized tokenizers, effectively dividing text into manageable units. Yet, as we start to integrate LLMs with temporal data, the complexities of this union become evident.

The tokenizers employed by LLMs appear to stumble when grappling with numerical inputs.³ As we show below, repetitive patterns, an inherent characteristic of time-series data, can confound tokenizers, leading to the unintentional fragmentation of continuous sequences into disjointed tokens. Consequently, the temporal relationships that underpin such data may be lost in translation, potentially undermining the very essence of the information being processed (Arguably, language also presents inherent short- and long-term dependencies, however, given that LLMs are trained on large predominantly textual datasets, we expect that they learn more meaningful patterns between such tokens compared to numbers.).

We argue that naively passing time series to LLMs can be ineffective because their input tokenizer layers are not optimized for numerical data. Therefore, we first show that such “zero-shot” approaches fail short due to the modality gap⁴ (We define “modality gap” as the challenge of mapping different modalities like text and signals in the same latent space. Existing works have shown that even in models whose modalities have been trained jointly, they reside in completely separate regions of the embedding space.) and advocate for different strategies. In this context, this article delves into the nuances and obstacles that emerge when LLMs are confronted with the task of representing temporal data. We focus on the interplay between numerical and textual information, uncovering the potential pitfalls that can hamper the effective utilization of LLMs in scenarios where temporal context is important. We highlight that the key challenge for correctly processing such modalities into LLMs is the lack of available timeseries training data within their textual corpora. Last, we discuss potential solutions from the rapidly growing area of parameter-efficient transfer learning and multimodal adapters that could enable better integration of nontextual data into LLMs.

Tokenization in language models

Tokenization is a fundamental process underpinning the operation of LLMs. It involves the division of input and output texts into smaller, manageable units known as tokens. These tokens serve as the building blocks of language comprehension, enabling LLMs to process and generate text effectively. Tokens can encompass a variety of segments, such as characters, words, subwords, or symbols, depending on the chosen tokenization scheme.

The purpose of tokenization extends beyond mere text segmentation. It enables LLMs to handle diverse languages, vocabularies, and formats while mitigating computational and memory demands. The quality and diversity of generated text are influenced by tokenization, as it shapes the meaning and context of individual tokens. Tokenization strategies vary and can be rule-based, statistical, or learnable in nature, adapting to the complexity and variability of the input texts.

One of the prominent subword tokenization methods used in Transformers, the architecture underlying many LLMs, is byte-pair encoding (BPE).⁵ OpenAI, for instance, employs BPE in its GPT-based models, which include around 50 000 tokens. BPE operates by iteratively merging the most frequently occurring pairs of characters or bytes into a single token until a predefined number of tokens or vocabulary size is reached. This approach is especially useful for accommodating rare or previously unseen words, resulting in more compact and consistent representations of text.

BPE variants include WordPiece⁶ and SentencePiece.⁷ WordPiece tokenization segments text into words or subwords, often leading to smaller vocabularies and efficient processing. SentencePiece, on the other hand, extends tokenization to the sentence level, empowering models to handle morphologically rich languages with complex sentence structures, such as Chinese and Japanese. Recent state-of-the-art models like Llama 2 employ BPE and SentencePiece by splitting all numbers into individual digits and use bytes to decompose unknown UTF-8 characters.⁸

As an example, let’s consider the word “playing” to illustrate how BPE works. First, the algorithm identifies the most frequent pair of characters, that according to our hypothetical corpus is “ay”. It then merges “ay” to create a new token: “pl”+“ay”=“play”. Our new updated vocabulary now is “play” and “ing”. In this example, by breaking down words into subword units and merging frequent pairs, BPE captures meaningful components of words, even those that might not have appeared in the original vocabulary.

Where do tokenizers struggle?

Despite its importance, tokenization introduces various challenges and open problems that impact the quality, interpretation, and usability of LLMs’ outputs. While tokenization schemes like BPE have been successful in segmenting text into meaningful units, several issues persist, highlighting areas where further research is needed.

Case sensitivity

Tokens are assigned numerical identifiers within a tokenizer’s vocabulary. However, cases of words are treated as separate tokens, leading to inconsistencies in representation. For instance, “good” and “Good” are encoded as distinct tokens, hampering the LLM’s ability to understand text, particularly in scenarios where capitalization matters.

Trailing whitespace

Tokens with whitespace are treated as separate entities; for instance, tokens such as “the first step is” and “the first step is” exhibit different representations, impacting the probabilities of subsequent tokens. This behavior introduces subtle biases that affect the model’s language generation process and response characteristics.

Digit chunking

The tokenization of numerical values poses challenges, particularly when digits are inconsistently chunked. Numbers like “480” might be tokenized as a single unit, while “481” and “482” are split into 2 tokens. This inconsistency not only hampers the model’s mathematical capabilities but also introduces complexities when dealing with consecutive values and temporal dependencies.

Input         → Token IDs

480, 481, 482 → 22148, 11, 4764, 16, 11, 4764, 17

Integers

The particular case of integer tokenization was investigated in a recent work.⁹ In a series of simulations with popular LLMs, it was found that the tokenization of integers lacks a coherent decimal representation, leading to a fragmented approach where even basic mathematical operations require memorization rather than algorithmic processing. On the other hand, nonunique integers are tokenized through arbitrary and inconsistent chunking, complicating multi-digit arithmetic operations. The irregular division of numbers into tokens varies even for adjacent integers, demanding models to memorize arbitrary tokenization patterns. Notably, tokenization is influenced by the temporal distribution of data. Common dates, mostly in the 20th century, are assigned unique tokens, revealing the problematic interplay between tokenization and the characteristics of the training dataset.

Floating-point numbers

These numbers, often representing decimal values, can be challenging to tokenize consistently. The representation of floating-point numbers can vary based on factors such as precision, scientific notation, and rounding errors. Tokenization of floating-point numbers involves decisions on whether to treat them as a single token or break them down into constituent parts. For instance, consider the number “3.14159”, which might be tokenized as 4 different chunks by popular LLMs: “3” + “.” + “14” + “159”.

Input → Token IDs

3.14159 → 18, 13, 1415, 19707

Arithmetic reasoning

LLMs exhibiting inconsistent tokenization of numbers struggle with addition tasks that involve 2-digit numbers and completely falter in adding larger numbers.¹⁰ Specifically, their accuracy drops to zero for digits totaling 5 or more. This deficiency is attributed to the absence of a systematic approach to tokenizing individual digits. For instance, “1234” might be segmented into “12”, “34”, while “5678” could be split into “5” and “678”. Consequently, the model must grasp that a token’s embedding could signify either a single digit or multiple digits, introducing complexity when mapping embeddings to varying-digit numbers. This irregular mapping challenge hinders the model’s ability to accurately establish associations between embeddings and numbers of varying digit lengths.

Model-specific behavior

Tokenization is not universally consistent across different language models. Even when employing the same underlying method, different models can have distinct token representations. This model-specific nature of tokenization poses challenges for interoperability, preprocessing, and multi-modal modeling. It necessitates careful consideration when assessing various LLMs for real-world applications.

A case study on tokenizing temporal data

To showcase some of these challenges in the real world, we encode the first 20 samples of a popular mobile sensing dataset (WISDM¹¹) via the OpenAI Tokenizer that powers ChatGPT (The public OpenAI tokenizer was used in all examples in sections “Where do tokenizers struggle?” and “A case study on tokenizing temporal data”: https://platform.openai.com/tokenizer. The following file from user’s 1600 smartwatch in WSDM dataset was used: data_1600_accel_watch.txt.). Each row contains the Participant ID, the activity label code, a UNIX timestamp, and the 3 accelerometer axes (x, y, z). As is common, models learn the temporal dependencies across the 3 axes in order to predict whether users perform activities such as running or walking. Activity recognition is a fundamental task in personal devices that, for example, can detect falls and notify one’s family or emergency services. For illustration purposes, we choose not to filter or preprocess any information of the dataset.

In Figure 1, we show the raw input data along with the tokenized output and the token IDs. Affirming the concerns outlined in the previous section, we note the model’s tendency to divide timestamps into multiple tokens, potentially rendering temporal dependencies across different rows irrelevant. With regard to the actual sensor data, each accelerometer value, which is naturally represented as a floating point number, is also split into multiple tokens. Adding to the complexity, the negative sign in front of the sensor values is grouped with the comma separator, thereby disregarding the direction of the sensor reading.

Figure 1.

Example of incorrect tokenization of a mobile sensing dataset. Top panel: each row includes the Participant ID, activity label, UNIX timestamp, and x, y, z acceleration. The first few samples (A) of WISDM activity recognition dataset as it is processed into tokens (B). The timestamps, acceleration values, and negative signs are split into subtokens (C) that are not meaningful to models. Bottom panel: the acceleration values of each axis (x, y, z) as processed individually by the tokenizer (D). The different colors denote different token splits and might be repeated whereas the corresponding tokens are unique.

We argue that this representation cannot allow for any meaningful inference from LLMs. As we discuss in the following section, all existing preliminary works that attempt such inferences, have resorted to extensive filtering, such as downsampling, aggregating the signal, as well as rounding. As a result, raw data are rarely provided without context to models, with a lot of effort put into constructing well-crafted textual templates (prompts), in order to overcome this modality mismatch.

Large language models and timeseries

Researchers have started experimenting with treating LLMs as universal pretrained models,¹² despite any modality gap. In this section, we will discuss the motivation behind the need to incorporate temporal data to LLMs, common scenarios, and recent works on timeseries tasks, with a focus on high-frequency temporal data such as those found in biomedical signals. We also note that while there are plenty of works that borrow the Transformer/GPT architecture and retrain from scratch,^13,14  here we focus on the paradigm of directly feeding temporal data to pretrained LLMs.  Figure 2 provides a conceptual comparison between the key methods we discuss below, focusing on the tradeoff between computational efficiency (adaptation) and advanced reasoning capability (expressiveness).

Figure 2.

Conceptual comparison of proposed methods for timeseries-aware LLMs seen as a function of expressiveness and adaptation/computational requirements. The main paradigms are listed in purple alongside some key works in the area in green. We notice 4 distinct areas: pretraining from scratch using domain data, new tokenizers, and prompting methods which require minimal fine-tuning, light-weight adaptation like prompt-tuning that try to balance both dimensions, and multimodal adapters that provide more reasoning capabilities at the cost of aligning every modality. There is a noticeable gap in the lower right quadrant showcasing the tradeoff between parameter efficiency and advanced reasoning. We note that the exact position of each method is based on a qualitative assessment of the overall contribution and should not be used for individual comparisons (similar works are “jittered”); we recommend interpreting this figure on the quadrant level.

Why would we pass time series into LLMs?

LLMs are predominantly trained on large text datasets crawled from the public web. Datasets such as The Pile and Common Crawl¹⁵ contain hundreds of gigabytes of text sourced from Wikipedia, Google Patents, Reddit, and other places. While the exact percentage of text versus other types of data such as numbers is not explicitly mentioned in the available sources, we expect that the available numerical information is limited. As a result, the most frequent tokens dominate the rest. Additionally, due to the inherent nature of language being a discrete medium, compared to numbers being continuous, there are considerably more variations in the latter. On the other hand, training foundation models on timeseries data only might seem like a promising direction, especially in cases we are interested in accuracy over reasoning capabilities. Attempts like TimeCLR¹⁶ or Apple’s AHMS¹⁷ combine multiple timeseries datasets from appropriate repositories like the UCR Archive or health studies, which are, however, magnitudes smaller than text datasets. The above also motivates the adaptation of LLMs over modality-specific models.

Despite the popularity of LLMs in language tasks, the established modeling paradigm of using neural networks for timeseries tasks still involves a combination of convolutional, recurrent, and attention layers.¹⁸ For example, in activity recognition or arrhythmia detection, a common workflow involves a sliding window which splits the input signal into fixed-sized sequences of channels (accelerometer axes or ECG leads, respectively). Due to inherent difficulties in collecting large labeled datasets, self-supervised learning has proved to be particularly effective in such tasks.^19–22 However, even large unlabeled data of that kind is hard to encounter because it is not publicly available on the web, unlike images or text. Another crucial limiting factor is the number of potential modalities; for example, aligned and paired data of movement, heart rate, audio, and sleep patterns might be impossible to collect and share at scale due to privacy concerns. As a result, available pretrained models in areas such as physical activity or heart health are limited in size and generalization capabilities, compared to popular foundation models.^17,23,24 This further motivates the shift from scarce domain-specific models to repurposing general foundation models and LLMs.

Feeding timeseries to LLMs

Following the established experimental setup from the NLP community, the following evaluation protocols are usually followed when feeding timeseries data to LLMs:

• Zero-shot inference ²⁵ : The input is either the raw timeseries or its combination with associated text that might describe the data. The LLM makes predictions directly on the test data without any additional tuning or training. This tests the out-of-the-box capabilities of the model but most usually performance is poor and hence it is used as a “lower bound” baseline. Especially in cases where the numerical data dominate the input, the model cannot “ground” the data to any language patterns.

• Prompt engineering ²⁶ : Building on the textual component of the previous scenario, prompt engineering refers to methods that enrich the text accompanying the signals. Enrichment methods include adding context such as additional metadata, health context, temporal context, and combinations thereof. This guides the model to the potential “solution space” because it helps to contextualize the raw signals. As in zero-shot mode, it operates on test-time and does not update any model parameters. The 2 methods can also be combined.

• Few-shot/in-context learning ²⁷ : While zero-shot inference involves single inputs without any associated labels or answers, few-shot or in-context learning uses a limited selection of demonstrations within the prompts and in contrast to the previous approaches, this requires pairs of <data, label>, however, the model parameters still stay fixed. Few-shot learning can be used together with prompt engineering.

• Fine-tuning ²⁸ : This approach involves a partial retraining of the model parameters and as such it is the most computationally expensive of the four. A trainable embedding is added to the input of the LLM (prompt-tuning) or the intermediate layers (prefix-tuning). While fine-tuning helps to specialize the model to each respective task, it might result in forgetting more general knowledge. We delve into the tradeoffs of fine-tuning in section “How to better integrate temporal data into LLMs?”.

Applications to human-centric signals

Recent works have applied the above scenarios to physiological signals, human gestures, and mobility data. These initial works have started investigating time series and LLMs for health, but they have limitations: they mostly rely on data curation or aggregation, handcrafted prompt-engineering, and proprietary models whose results cannot be reproduced independently. For instance, a preprint by Liu et al.²⁹ (referred to as “GoogleFewShot” in Figure 2) looked into heart rhythm classification, activity recognition, calorie estimation, and predicting stress levels and mental health scores from Fitbit data. The article also explores the effects of providing textual context versus solely numerical data as input to the LLM. Across almost all tasks, including more contextual information like “Classify the given Interbeat Interval sequence in ms” led to improved performance over just feeding in the raw numbers while fine-tuning achieved the best performance. Regarding the base model, the authors employ a proprietary LLM (PaLM³⁰) which uses the SentencePad tokenizer with a 256-K-sized token vocabulary. While acknowledging the promise of these results, there is limited information on any preprocessing or tokenization. We assume that once in textual prompt format, the time-series data are directly fed into the base model without any further vectorization or embedding. In section “How to better integrate temporal data into LLMs?”, we expand on the importance of prompt tuning and compare it to other approaches that could alleviate modality mismatches between numbers and language.

Similar to the above study, another preprint (Health-LLM) evaluated various physiological signals for the prediction of stress, physical activities, calorie expenditure, sleep quality, and cardiovascular health.³¹ This work focused on prompt enhancement by proposing multiple strategies to incorporate context into the prompts. These strategies included adding user metadata such as age or sex, health context such as definitions of the outcome at hand, and temporal context that summarized the timeseries in sentence. The combination of such methods yielded up to 23.8% improvement in performance in a fine-tuned model. In a medical context, the prompting enrichment approaches discussed in this work could provide better interpretability when audited by medical professionals, compared to adapter-based methods that map data to vectors.

Other preliminary works followed similar experimental setups. In a zero-shot scenario using a small-scale dataset, the popular GPT4 model was fed in hand gesture data through light and vibration sensors.³² The authors report high performance using prompt engineering templates such as: “I use an RGB Light and Proximity Sensor to detect the gestures Single air tap, Double air Tap, and Triple air Tap. [.] Given below are sample data collected at three instances for each gesture: [Numbers]”.

PromptCast also followed a similar scenario by framing timeseries forecasting as a language generation problem and introduced a new prompt template dataset covering weather, energy, and human mobility forecasting.³³ Experiments compared state-of-the-art numerical forecasting methods like Informer, Autoformer, and FEDformer against language models like T5, BART, and BigBird. The results showed that language models are competitive to numerical methods when fine-tuned on the datasets, with Bigbird, BART, and RoBERTa being the top performers. The language models also showed stronger generalization under zero-shot transfer. We attribute the success of such a setup to careful dataset creation and the short sequence length (15 timesteps) used in the prompts. We expect that by incorporating prompt-tuning or other adapter layers, these models would further improve over purely numerical models.

How to better integrate temporal data into LLMs?

Despite the encouraging results, we see that these preliminary works have to “ground” the LLMs in numerical data using verbose hand-engineered prompts and extensive aggregation of timeseries. In other words, they rely predominantly on the power of text to contextualize numerical data. This article argues that we will only unlock the real value of temporal data of that kind, once we map it to a more meaningful representation. While modality mappings between images and text are well-studied,^4,34 the link of timeseries to text remains virtually unexplored. We point the reader to Jin et al.³⁵ for a comprehensive view of LLMs for timeseries. In this section, we highlight the following 4 key research directions towards better integrating time series into language models: prompt-tuning, model grafting, improved tokenizers, and hybrid approaches.

Prompt tuning

Parameter-efficient fine-tuning (PEFT) has emerged as a potential solution to this problem. Traditional neural network fine-tuning would involve a pretrained network that is fixed (or frozen) and new trainable layers that are added to the end of the network in order to tailor it to a specific task. Instead, PEFT involves prompting or lightweight layers that are added usually to the input (prompt-tuning) or selected layers throughout the model (prefix-tuning).

In general, prompt tuning refers to techniques that change the LLM prompt to achieve better results. Directly changing the input content is known as hard prompt. More interestingly, soft prompting involves prepending a trainable tensor to the input which can then be optimized to a downstream task via backpropagation.³⁶ As seen in the previous section, this approach is particularly effective because the prompt layer encodes information that assists the model in comprehending timeseries data that is absent in the original frozen LLM.²⁹ This is significantly more parameter-efficient than full fine-tuning which becomes prohibitive as the model size and the number of tasks grow.

On a similar note, modules known as Adapters insert small trainable blocks to each layer of the pretrained network with only these blocks being fine-tuned. For example, a popular method, LoRa, injects trainable low-rank matrices into transformer layers to approximate the weight updates.³⁷ While these approaches have similar goals, here we are interested in prompt-tuning because it operates on the input space and can effectively steer the model to process timeseries data (A unified view across all tuning methods is provided in Ref.³⁸).

Model grafting

By viewing the mismatch between numbers and text as a multimodal understanding problem, the notion of model grafting has emerged as another solution.³⁹ For example, in a recently proposed model called HeLM, nontext modalities are mapped via encoders (trained over input examples) into the same token embedding space as text, where a separate encoder is learned for each modality.⁴⁰ In particular, HeLM trained 2 individual encoders to map spirogram sequences (lung capacity measurements) and demographic data into a limited-size token vocabulary that were inserted into the text tokens (being order-agnostic). Modality-specific encoders are also common in multimodal foundation models such as ImageBind,⁴¹ IMU2CLIP,⁴² AnyMAL,⁴³ and Meta-Transformer,⁴⁴ with the latter proposing a data-agnostic approach to merge token embeddings into a shared manifold space. Most notably, while approaches like Imagebind align every modality with vision, AnyMAL does so through language by leveraging more powerful and larger LLMs, hinting that language can be used as the universal proxy to other modalities. In the same vein, ELIXR,⁴⁵ LLaVA,⁴⁶ and BLIP2⁴⁷ introduced adapters (Term disambiguation: while adapters like LoRa insert trainable modules within layers, adapters like ELIXR correspond to independent models that plug into other pretrained LLMs.) that mapped the activations of an image encoder into an LLM-understandable form.

Model grafting has both advantages and disadvantages. On the one hand, it is computationally efficient when training the adapter layers, while allowing the LLM to connect to other high-performing models from different domains (eg, a well-trained encoder of ECG data). Breaking down the problem into encoder, adapter, and LLM components can also facilitate faster testing and iteration cycles. On the other hand, this modularization introduces complexity by dealing with a system that is not trained end-to-end. Last, compared to natural language prompts, the communication between the specialist encoder (eg, image or timeseries) and the LLM is no longer meaningful to humans since the mapping is represented as high-dimensional vectors.⁴⁸

Improved tokenizers

Besides training lightweight adapters, we can rethink the design of tokenizers for mixed textual and numerical data. LLaMa was recently shown to outperform GPT-4 in arithmetic tasks due to splitting each digit into an individual token, thereby ensuring consistent tokenization of numbers.⁴⁹ Specialized models trained on the scientific literature and notation also used similar digit-level splitting, pointing to improved understanding of symbolic and numerical data.⁵⁰ LLMTime⁵¹ introduced a new tokenization method that separates the digits with spaces, drops decimal points, scales down values, and converts tokens into continuous values, enabling frozen LLMs to outperform domain-specific models. xVAL⁵² focused on processing the numbers independently by storing each number in a separate vector, using a single token. Then, the preprocessed text is encoded into a finite series of word tokens, while the embeddings of the number tokens are multiplied with their corresponding numerical values.

Hybrid approaches

While all these approaches can be seen in isolation, some works have started incorporating them into single models. For instance, Time-LLM⁵³ uses a reprogramming⁵⁴ method to align timeseries and text clusters (prototypes) via patching, which acts as tokenization. Additional prompt prefixes such as trends and lags and other statistics are concatenated to the input as embeddings to enrich reasoning capabilities. Other works like the LLM4TS⁵⁵ used both alignment and fine-tuning but addressed the challenge of limited context window length by adopting channel-independence along with patching for time-series tokenization, by converting multivariate timeseries into multiple univariate timeseries.

Discussion

This article discussed the emerging idea of incorporating temporal data into pretrained LLMs and proposed solutions for better encoding of numerical inputs into LLM-understandable information. We argue that this paradigm shift on how we leverage pretrained models will particularly affect the area of biomedical signals, given the lack of modality-specific foundation models. To bridge the “modality gap”, practitioners should focus on methods that optimize for both expressiveness and parameter-efficiency such as model grafting with multimodal adapters, along with careful reassessment of existing tokenizers and embedders. Beyond tokenization, the next generation of LLMs should address other architectural constraints such as longer context windows, towards better handling of real-world temporal datasets. Last, while our scope is limited to temporal data, these approaches could generalize to other nonlanguage tasks.⁵⁶

Author contributions

DS wrote the first draft of the manuscript and produced the figures. All authors (DS, FK) critically reviewed, contributed to the preparation of the manuscript, and approved the final version. All authors vouch for the data, analyses, and interpretations.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Conflicts of interest

None declared.

Data availability

All datasets used in our analysis are available in previous works.¹¹