Seq-LoRA: Sequential Bayesian Low-Rank Adaptation for Large Language Models

Low-Rank Adaptation (LoRA) enables efficient fine-tuning of large language models, but adapted models often become overconfident, especially in low-data settings and under distribution shift. Existing uncertainty-aware LoRA methods typically construct a static posterior over a fixed adaptation set and do not model how local posterior evidence varies across structured subsets of the training data. We propose Seq-LoRA, a post-hoc Bayesian framework for sequentially aggregating slice-wise posterior evidence in LoRA space. Starting from a shared maximum a posteriori LoRA anchor, Seq-LoRA builds slice-wise Kronecker-factored quadratic surrogates, projects them into a shared curvature-informed low-dimensional subspace, rewrites the resulting latent quadratic forms as Gaussian pseudo-observations, and performs exact inference in an induced linear--Gaussian state-space model via Kalman filtering. Rather than relying on a particular curriculum order, Seq-LoRA couples heterogeneous slice-wise local posterior evidence through a random-walk prior, forming a terminal Bayesian posterior for prediction. Across the in-distribution ScienceQA test split and six out-of-distribution (OOD) reasoning targets, Seq-LoRA substantially reduces deterministic LoRA overconfidence under shift. Among compared methods, it achieves the best negative log-likelihood on five OOD targets and the best expected calibration error on four, while preserving most task accuracy.