VENUS: Decoupling the Samples in MARS Benefits Small-Batch Training
VENUS evaluates the two branches of the MARS estimator on independent minibatches, adding one effective degree of freedom; with LMO-based convergence guarantees and lower final validation loss than exact MARS in small-batch GPT-2-style pretraining.
Nov 12, 2026