> Known-future covariates. Some drivers of sensor behaviour are known in advance, such as production schedules, planned shutdowns, weather forecasts. Timestral can condition on them to improve forecast quality, so a scheduled load increase is expected rather than flagged as a surprise.

To me this should be the entire basis of the system, not a condition to improve forecast quality? Like if you're modelling electricity consumption, if all you have is a window of N previous samples, you're going to get completely beaten by a comparatively simple system that knows the time of day, that there's holidays coming up, the weather forecast, etc.

I would also think that these time-series forecasting models would gain from a sophisticated understanding at least the labels of the covariates? A 5M model has no idea what a "bank holiday" is, it just sees an arbitrary class go from 0 to 1, and won't have any useful inductive biases as to what it means or how to link them to other pieces of information.

E.g., wouldn't you predict €/MWh of electricity far more accurately with access to price of natural gas (itself influenced by many geopolitical factors), breakdown in composition of production of each energy source, predictions about roll-out (requiring a sophisticated LLM capable of understanding news and their implications), etc. What's the use in trying to predict it from 2 variables and a narrow window looking back at just 8192 or 32768 samples?

Or perhaps I misunderstand the use-case and this is more for extremely short term movements?

Problem is you really don't even need ML for the curve fitting tools. Just ARIMA or Holt Winters is enough and likely faster. Heck, Kubernetes HPA doesn't even implement full PID, just P.

It seems like the main challenge is going to be finding domains where accuracy is important enough to spend inference token money on, but not so important that you'd hire a team to build a custom model. Or maybe that team borrows Timestral weights to fine tune to the domain, but they probably already have a different model customized to the problem?

You're right but, if the time series looks like music then you can apply your music domain knowledge. You don't have to only apply knowledge that's general across every domain, part of the magic of large neural nets is that you can do recognition and prediction at the same time thereby using (statistically) things known from training about how to identify a sequence and then also how to continue it.

Sure adding outside info helps even more, but with enough data it should become unnecessary (or so goes the theory).

Is there a reason you don't throw in something like cross-attention on an embedding of the sequence description/data labels? I've trained numerous generative models for audio and images and it's always the case that the unconditional model performs worse, trains slower, etc. than the conditional one (with the best performance from a hybrid using classifier-free guidance or similar scheme).

Again I don't have experience with the domain of time-series forecasting directly, but I would think there are generalities between autoregressive generative models and time-series forecasting, given that both involve predicting the next sample(s) given N regularly sampled past inputs.

Won't you need to attribute a weight to the data coming from the model and the weight for your data?

Would love to know actual time series foundation model use cases!

That seems like a scenario where there wouldn't be external sources of information capable of improving your predictions. It also seemed to me like a scenario where good old fashioned statistical methods would outperform LSTMs, which was what they used.

(I didn't get the job - I was offered it, but they rescinded their offer at the last moment, supposedly because their investors had insisted they should hire a web developer instead.)

I'm sure someone who actually knows this stuff has a better easy to view it but my impression was that these time series foundation models aren't trained for a specific dataset but more they are trained on a procedure for how to look at datasets and make predictions. It doesn't seem to lump things into a bucket much more than any other model type (which also are just consistent procedures to look at datasets). It's not like you have radically different model types across different fields.

To improve the robustness of machine learning models, artificial noise is often introduced into actual training data. Something like Timestral could simplify the process of generating it.

That being said, more information is of course better, assuming it doesn't introduce bias, like focusing on specific events like holidays too much, when the information is already contained in the historical data, without having to make assumptions about what an upcoming holiday means.

The biggest challenge last time I worked with time series forecasting (which is ~5 years ago) was learning long-term shifts in the baseline of the data (e.g., the general use of electricity always goes up, a lot more solar panels and EVs influence the baseline, etc.).

I can only assume (hope) that there has been some substantial progress in that area though.

(Edited the formatting for a nicer reading experience)

Now with LLMs the process is much shorter. They can walk me through the algorithm step by step, translate it to pseudocode, explain equations, and most importantly, I can ask them questions and get clarifications, without worrying about seeming ignorant or exhausting their patience. With a paper, you’re stuck with whatever was written (and often, researchers aren’t good teachers, or intentionally over complicate simple concepts).

Yep, my eyes glaze over at the convoluted mathematical explanations for pathfinding in worldspace, but when I see the pseudocode in the appendix it make so much more sense.

"The Secret of Our Success" literally starts with how European explorers starved to death in environments that other humans thrived in because they lacked the cultural knowledge local hunter gatherers had, includes many other examples of how pure reason can be counter-productive to survival, and ends with a warning that the idea that now it's finally different may not be true.

"Punchy phrase. No X." "An open engine. Room to make it your own. "Same scene. Same instant. The same explosion. A different atmosphere." "A few things to know. The short answers."

Why is it always the same? What awful post-training, or synthetic data training pipeline, or RLHF gone wrong, produced this homogenised, low variety slop style? And it's totally possible for LLMs to write actually decent or at least varied prose and I would think Anthropic (or OpenAI) has the resources to fix the issue, they just choose not to.

Marketing copy wasn't this homogenous prior to LLMs, unless my memory fails me.

It was, or at least the most popular sites used it. LLMs just amplified it.

With structured outputs, that just doesn't happen though?

It's the equivalent of sliding window attention, a common LLM optimisation. Global attention may be O(n2), but you can avoid that bottleneck by using local attention, even throughout the whole model. This could give you another efficiency boost, or enable you to process larger resolutions with linear instead of quadratic compute scaling.

You could also have the same advantages of a draft in low resolution latent space, with an easier to learn data distribution that's more robust to perturbation, but instead of 512x512, you could get a 4096x4096 output for the same compute (assuming a 8x VAE).

There's also the advantage of being able to use a high number of diffusion/flow matching steps for the main LDM while the VAE can be single step and much smaller, since it does not have to handle language or significant scene understanding, just perceptual compression. This sounds especially important for a video model where I would be extremely hesitant to train a generative model without relying on interframe compression.

Time is "just" another dimension. There's temporal continuity between frames, a video VAE would be learning and representing those temporal shifts, but there's nothing to say that e.g. a recursively applied generative model at the pixel level also doesn't learn and represent those things.

As ever, figuring out how to train the thing is the hard bit I expect.

(Handwaving over "textures" here, VAEs encode more like somewhat macro blocks of image whose content is also conditioned on surrounding blocks, rather than tiny patches of patterned pixels.)

(And yes I'm a total imposter layman here, I just see VAEs as seeming to be a crutch that reduce data size - super super helpful of course - but being strictly speaking redundant and inhibiting correct fine detail.)

Older models like the SD1.5 work like you mentioned, in large part due to relying on PatchGan which just ensures that individual patches are plausible in isolation. There's nothing preventing a VAE from rendering tiny text with the correct architecture and training. If pushed to an extreme, it could essentially start guessing the letters themselves and write different, but still legible words instead of nonsense scribbles.

The jumping point for this no-VAE work was three-fold:

1) Our goal here is to get 32x32 token reduction to make video training and inference downstream cheaper. To date, the best open-weight Image VAEs like Flux-2 seem to cap out at 16x16 token reduction (8x8 VAE + 2x2 linear patchification). Others like H3 have pushed to 32x32 reduction but requires them swapping out the small VAE decoder with a 2B parameter decoder. So, this is a foray to get 32x32 compression without compromising quality.

2) We believe that end-to-end trained networks will tend to perform better than modularly trained networks (e.g. VAE + DiT). This hypothesis comes from work like REPA-E, where authors are able to get much better results by backpropagating through the pretrained VAE. The latent space for perception / reconstruction seems to have a different "optimal" configuration than a latent space specifically built for generation. That's why we liked the idea of trained E2E here.

3) There's work with VAEs that show that providing additional modalities (e.g. text captions) can help the VAEs improve as well. That's natural to this construction, so we thought language might actually help with the compression, not hurt. To be honest, this this is the most hypothetical of three ideas; and definitely warrants specific ablations.

But how do you plan on scaling to 1080p24 (or higher resolution) video with end to end pixel space training? For a 10 second long video, that's 1920x more tokens than your 512x512 DiT experiments, and I'd be concerned about scaling linear patchification temporally and spatially past a certain point.

The linear patchificaion is per patch, so if it works on 360p 2 seconds (really short low res) it should adapt naturally (given ROPE3D). The final layers of the DiT start moving us from latent space to pixel space.

Net net the compression is as much as H3 (32x32xTime) so it should be comparable to the best VAEs in terms of expense

The main reason the web won is that you can ship SaaS without the user having to install anything locally, and there's no expectation of it being able to work offline, so you can offload processing to servers resulting in fewer bugs, compatibility issues, and DRM problems.