Tabular data are central to enterprise data infrastructure and underpin many critical predictive ML applications, from customer churn prediction to financial fraud detection. Historically, supervised tree‑based methods such as AdaBoost, XGBoost and random forests have dominated these tasks because of their robust performance on structured data.
Deploying those traditional models, however, is time‑consuming. Fitting an XGBoost model to a new dataset is not just a single .fit() call: practitioners must spend significant effort on domain‑specific feature engineering and extensive hyperparameter tuning to extract a reliable signal from raw tables.
Advances in large language models (LLMs) have shown a different approach: in‑context learning (ICL) enables zero‑shot prediction by providing examples and instructions in the model input without updating underlying weights.
TabFM is a foundation model purpose‑built for tabular classification and regression that treats tabular prediction as an ICL problem. By packing both historical training examples and target rows into a single prompt, TabFM produces high‑quality predictions on previously unseen tables in a single forward pass, removing the need for per‑dataset training, hyperparameter tuning, and manual feature engineering. The model is available on Hugging Face and GitHub.
How it works
Traditional ML updates model parameters for each dataset’s distribution. ICL bypasses this: the whole dataset becomes a single context from which the model infers relationships between columns and rows at inference time.
Applying ICL to tables is nontrivial because language models operate on one‑dimensional ordered sequences, whereas tables are two‑dimensional and inherently orderless (swapping rows or columns does not change semantics). To handle diverse tabular structures and enable scalable zero‑shot prediction, TabFM combines ideas from TabPFN and TabICL into a hybrid architecture with three key mechanisms:
- Alternating row and column attention: the raw table is first processed by a multilayer attention module that alternates attention across columns (features) and rows (examples). By attending across both dimensions, the model learns rich representations that capture complex feature interactions and dependencies, performing the heavy contextualization that manual feature engineering would otherwise attempt to accomplish.
- Row compression: after this cross‑attending stage, the contextualized information for each row is compressed into a single dense vector.
- In‑context learning (ICL): a dedicated Transformer operates on the sequence of compressed row embeddings. Following TabICL’s efficient approach, attention over compressed row vectors—rather than the full raw grid—dramatically reduces compute, keeping prediction efficient even for larger datasets.
Training on synthetic data at scale
Foundation models typically require high‑capacity networks trained on vast and diverse datasets. Public, large, and varied tabular datasets—particularly industrial‑scale tables—are scarce because many real tables contain proprietary schemas or sensitive information. Therefore, synthetic tables are the practical option for pretraining at scale.
TabFM was trained entirely on hundreds of millions of synthetic datasets generated dynamically using structural causal models (SCMs) with a wide variety of random functions. This large‑scale synthetic generation is intended to capture the range of distributions and complex feature relationships found in real tabular data, supporting generalization to unseen real‑world tables, as shown in the benchmarks.
Performance and benchmarking
To evaluate TabFM against state‑of‑the‑art methods, the authors benchmarked it on TabArena, a living benchmark that computes Elo scores from head‑to‑head win rates. The evaluation covered 38 classification and 13 regression datasets, with dataset sizes from 700 up to 150,000 samples.
Two model configurations were measured:
- TabFM: the out‑of‑the‑box model that generates predictions in a single forward pass without tuning or cross‑validation.
- TabFM‑Ensemble: an enhanced configuration that adds cross features and SVD (Singular Value Decomposition) features. A 32‑way ensemble’s optimal weights are computed using a non‑negative least squares solver. For classification tasks this variant also applies Platt scaling for calibration.
Detailed per‑fold metrics and head‑to‑head win rates against specific baselines are provided on the project’s GitHub page.
Conclusion and integration
By reframing tabular prediction as an ICL problem, TabFM’s hybrid attention architecture and massive synthetic pretraining capture complex feature interactions natively and remove traditional bottlenecks: manual feature engineering, hyperparameter optimization, and repeated model training. The approach consistently outperforms heavily tuned, industry‑standard supervised algorithms and delivers out‑of‑the‑box convenience for tabular ML workflows.
TabFM is also being integrated directly into Google BigQuery: in the coming weeks users will be able to run advanced regression and classification using a simple AI.PREDICT SQL command within BigQuery, without ML expertise.
Acknowledgements
This project is joint work with Erez Louidor Ilan, Taman Narayan, Shuxin Nie, Rajat Sen, Yichen Zhou, Joe Toth, Deqing Fu and Samet Oymak. Kimberly Schwede is thanked for designing the graphics.
