Fine-Tuned MLP-Mixer Foundation Models as data-driven Numerical Surrogates?

Scientific Machine Learning (SciML) has significantly advanced climate science by enabling precise forecasting of complex dynamical systems. While state-of-the-art models excel in domain-specific tasks, recent advancements in time series-based foundation models seek to replicate the success seen in natural language processing and computer vision. This study investigates whether a "small" MLP-Mixer-based foundation models, Tiny Time Mixers (TTMs), can be fine-tuned to accurately forecast complex real-world dynamical systems while adhering to practical resource and cost constraints. Our findings reveal that TTMs are sensitive to the dynamical characteristics present in the training data, particularly in terms of amplitude and periodicity, yet significant variations in forecast accuracy were observed within the same training distribution. These results highlight the need for further adaptation of TTMs to enhance their robustness in specialized SciML forecasting tasks.