By Hemanth Ram

TSFM Meets LLM: Context as Covariates

Accepted to the 40th Conference on Neural Information Processing Systems (NeurIPS 2026).

Authors: Hemanth Ram^*, Sai Shankar Narasimhan^*, Shubhankar Agarwal, and Sandeep Chinchali.

^*Equal contribution.

Paper | Code

TL;DR: Large Language models (LLMs) understand news and reports, but are not numerically precise. Time series foundation models (TSFMs) are trained for numerical precision but can’t read. CoCo (Context as Covariates) uses a language model to read the text and provide simple numerical covariates that a time series foundation model can use, giving forecasts that are both context-aware and precise.

Motivation

Real-world forecasting rarely happens with numbers alone. Consider the following:

“Forecast Apple’s stock price if revenue from iPhone sales is 15% higher than the target.”

“Forecast crude oil prices given the recent tariffs on OPEC countries.”

The answer depends on the text, not just on past prices. This kind of context is everywhere. Analyst notes and policy announcements move markets, product reviews hint at future demand, and weather bulletins precede changes in traffic. These signals often foreshadow changes that past numbers simply cannot reveal.

Figure 1 – Context matters. When forecasting crude oil prices, news about recession fears lines up with a sudden price shift. CoCo uses this context to follow the shift, while a numbers-only model (Chronos2) misses it.

The Current Gap

State-of-the-art TSFMs, such as Chronos2 and TimesFM, are trained only on numbers. They were never taught to work with text or text embeddings, so even when the text clearly contains useful information, they are unable to leverage it. Large language models (LLMs), on the other hand, understand text well but lack numerical precision. We observe two obstacles that stand between these complementary capabilities and a useful multimodal forecaster:

1. Not enough paired text + time series data. Learning how a news headline affects the next few days of a series needs lots of aligned examples across many domains. Existing multi-domain datasets have only a few thousand such examples, and the larger ones are restricted to a single domain, like stock prices.

2. LLM-only approaches throw away what TSFMs do well. Many prior methods adapt the time series into text tokens and let the LLM predict the future. This uses the text, but ignores the strong forecasting instincts and uncertainty handling that dedicated TSFMs already have. The natural answer is to combine the two: let the LLM understand context, and let the TSFM do the numerical forecasting.

Introducing CoCo: Context as Covariates

CoCo works in two stages: the LLM forecasts with the context, and the TSFM refines that forecast. The bridge between them is a very simple interface that modern multivariate TSFMs already understand: covariates, i.e., extra time series provided alongside the target (like “temperature” when forecasting “electricity demand”).

Figure 2 – How CoCo works. The LLM reads the numerical history and the text and produces a context-aware forecast, sampled several times. The forecast and the LLM’s per-step confidence are passed as covariates to Chronos2, which refines them into the final, numerically precise forecast.

Given the numerical history and the text, the LLM first produces a forecast that is context-aware but not very numerically precise. We also sample multiple forecasts and measure how consistent they are at each step, as a sense of the LLM’s confidence. Both are then provided to the TSFM as covariates alongside the history. The forecast is laid out as a noisy preview of the next value, which the TSFM refines into a precise forecast, while the confidence tells it how much to trust the LLM at each step.

Generating Synthetic Multimodal Training Data

To fix the data shortage, we take real time series and ask an LLM to annotate them with text containing plausible events explaining each step and a hedged outlook on what comes next. The key idea is varying how much of the future the annotator sees, so some annotations carry strong signals while others are purely speculative, just like real-world text.

Figure 3 – Synthetic annotations look realistic. LLM-generated annotations for US gasoline prices in 2022. The text is relevant to the domain and foreshadows the trend in the numbers (green: rising, red: falling).

We annotated about 37,000 time series (~12 million time steps) across retail, macroeconomics, environment, and finance. Because synthetic and real annotations describe the same kinds of patterns (“a spike is coming”, “a slow decline ahead”), models trained on synthetic text transfer well to real text.

Two-Step Fine-Tuning

Training the LLM and the TSFM together would mean each is chasing a moving target. Instead, CoCo trains them one after the other:

Step 1 – Teach the LLM to give useful covariates. We keep the TSFM frozen and fine-tune the LLM with reinforcement learning (GRPO). The reward is simple: did the LLM’s forecast covariate make the TSFM’s forecast better than it would have been without it? The LLM is never directly told the right answer. It only learns to produce covariates that actually help.

Step 2 – Teach the TSFM to read the covariates. With the improved LLM, we generate the forecast and confidence covariates for the whole training set, then fine-tune the TSFM on them. This is where the TSFM learns to treat the forecast covariate as a soft preview of the future, and to use the confidence covariate to decide how much to rely on it.

Figure 4 – CoCo pipeline. The LLM (Qwen3-4B) is fine-tuned with reinforcement learning (GRPO) and is rewarded when its forecast covariate reduces the TSFM’s error. The TSFM (Chronos2) is then fine-tuned on the covariates from the fine-tuned LLM.

Results and Generalization

We use a small open model, Qwen3-4B, as the LLM and Chronos2 as the TSFM, and evaluate on 30 real-world datasets (95 settings) covering finance, macroeconomics, retail, climate, traffic, and security, against 27 baselines, including state-of-the-art TSFMs, LLM forecasters, and multimodal fine-tuning methods.

Figure 5 – Overall comparison. CoCo achieves the best average rank and the lowest average error across all 95 settings, including both unimodal and multimodal fine-tuning approaches.

CoCo reduces error by up to 12% compared to state-of-the-art TSFMs and LLM forecasters such as GPT-4o. Training on our synthetic annotations generalizes to domains not seen during training, and mixing synthetic with real data gives the best results overall. Interestingly, after CoCo fine-tuning, the small Qwen3-4B outperforms much larger LLMs even as a standalone forecaster.

The gains are largest when the text is genuinely useful, and the confidence signal helps the TSFM ignore the LLM when it is unsure. Fine-tuning both models and using both covariates gives the largest improvement, with each piece adding on top of the others. CoCo is not tied to one model. Across five LLM configurations and three TSFM architectures, adding the LLM’s forecast as a covariate improves every combination. The fine-tuned TSFM also works with hints from LLMs it was never trained with.

Figure 6 – CoCo generalizes across model choices. Error improvement over the TSFM baseline for different combinations of LLMs and TSFMs. Every combination benefits.

Where this leads

CoCo connects the LLM and TSFM through explicit numerical covariates and fine-tunes them in separate stages. One direction to explore is joint fine-tuning through soft tokens: learned continuous representations that could carry richer contextual information between the models and let them adapt together. Whether that added flexibility improves forecasting enough to justify the training complexity remains an open question. Reliability is also unresolved. Agreement across sampled LLM forecasts measures consistency, but a model can be consistently wrong, especially when the text is misleading, outdated, or outside its training distribution. CoCo’s confidence covariate is therefore not a guarantee that the context deserves to be trusted. Learning when to use textual context, when to discount it, and when to fall back on numerical history remains an important challenge.