> "Synthetic data helps with the basics. Meaningful synthetic time-series data can be generated using statistical models or physical simulations. These basic temporal patterns can teach the model the grammar of time series forecasting."
Can someone elaborate on what a grammar means in the context of time series forecasting?
Probably drawing an analogy to how causal pretrained models go through stages of understanding language, words -> grammar -> meaning. Gwen mentions this experience when training character level RNNs. https://gwern.net/scaling-hypothesis#why-does-pretraining-wo...
Totally. Also basic temporal scale or cyclic properties. It’s kind of mind blowing that the shape of most recorded human patterns is reducible in this way.
It's a bit of an anthropomorphism ation. I don't believe it has any formal meaning here. The idea is that there are certain kinds of underlying signals and patterns which are common to a wide range of time series data. So if a model is able to learn those signals and patterns, it can look at any time series and, with enough historical data, predict future data, without actually updating any model weights. Those signals constitute a "grammar" of sorts.
You could probably consider learning a sign wave to be a “grammar” related to periodic variations. Grammar in this context feels like “What are the core conceptual heuristics that help guide towards faster understanding”
Can someone elaborate on what a grammar means in the context of time series forecasting?