Sentence Transformers — the Python library for embedding and reranker models used in semantic search and retrieval‑augmented generation — adds a fourth model type in v6.0: MultiVectorEncoder. This component implements ColBERT‑style late interaction retrieval: it preserves one vector per token and scores query‑to‑document matches with the MaxSim operator.
Compatibility and formats
- Any PyLate checkpoint and any Stanford‑NLP ColBERT checkpoint can be loaded directly into MultiVectorEncoder.
- ColPali / colpali‑engine family models for visual document retrieval are supported through the same API; a few colpali checkpoints need a small configuration file on the Hugging Face repo until those PRs are merged.
What multi‑vector models change
A dense embedding model compresses an entire text into a single fixed‑size vector. Multi‑vector (late interaction / ColBERT‑style) models keep small projected token embeddings (classically 128‑dim) for every token. For example, a 9‑token document becomes a 9×128 matrix rather than a single 1×128 vector. The interaction is deferred until scoring time: documents are encoded independently and can be indexed offline, but scoring compares every query token to every document token.
The MaxSim operator
MaxSim computes, for each query token, the highest similarity against any document token and sums those maxima:
MaxSim(Q, D) = sum_{Qi in Q} max_{Dj in D} Qi · Dj
Token embeddings are L2‑normalized, so each dot product is cosine similarity in [‑1, 1], and the total score lies in [‑num_query_tokens, num_query_tokens]. MaxSim acts as a soft alignment: query tokens point to the document tokens that best explain them, enabling contextual matches (e.g., live ↔ inhabit) that lexical methods like BM25 miss, while still preserving exact token matches that a single vector would have to average away.
Benefits and tradeoffs
- Benefit: improved retrieval quality for queries that depend on a specific token or when multiple requirements must each find their own evidence. Particularly strong for visual document retrieval where chart/table/paragraph structure matters.
- Cost: larger index footprints since there is one vector per token (partly offset by smaller vector dimensionality and compression options).
Concrete storage example (Natural Questions, 4,874 passages):
- Dense all‑MiniLM‑L6‑v2: 4,874 vectors, 384d → 7.5 MB (float32)
- Dense gte‑modernbert‑base: 4,874 vectors, 768d → 15.0 MB
- Multi‑vector lightonai/LateOn: 608,414 token vectors (avg 124.8 per passage), 128d → 311.5 MB
With PLAID compression (fast‑plaid) those 608,414 vectors can occupy ~92 MB. Token pooling and retrieve‑and‑rerank patterns also reduce footprint.
Installation and requirements
Multi‑vector models work with the standard install:
pip install -U sentence-transformers
For ColPali‑style visual retrieval, also install the image extras:
pip install -U "sentence-transformers[image]"
Sentence Transformers v6.0 requires transformers v5.x, torch 2.2+, and huggingface‑hub v1.x.
Loading a model and inspecting configs
Loading is the same as other models:
from sentence_transformers import MultiVectorEncoder model = MultiVectorEncoder("lightonai/LateOn")
MultiVectorEncoder reads PyLate and Stanford‑NLP ColBERT formats and recovers recipe knobs from checkpoint configs: query/document marker prefixes, length caps, query expansion, skiplists for scoring, and so on. print(model) exposes those settings. document_length is important because it truncates documents before indexing; you can override it for a single encode call via processing_kwargs but be mindful of index growth.
Encoding queries and documents
Multi‑vector models are asymmetric: you must call encode_query() and encode_document() to get correct token matrices. Each returned item is a 2D tensor of shape (num_tokens, embedding_dim). For example a short query might return (10, 128) while different documents return different token counts.
Scoring with MaxSim and MeanMaxSim
model.similarity() computes the full all‑pairs MaxSim matrix. Because MaxSim sums per‑query‑token maxima, scores scale with the query token count and are not directly comparable across models with different query recipes. If you need bounded scores, use MeanMaxSim (similarity_fn_name="meanmaxsim") which averages by query token count and yields values in [-1, 1] (practically in [0,1]).
Semantic search and small‑scale exact scanning
For small corpora exhaustive MaxSim is simple and exact: encode the corpus once and score every query against all token vectors. In the example with 4,874 passages and 608,414 token vectors, encoding took ~20s on an RTX 3090 and each exact search took ~120ms end‑to‑end, most of that MaxSim scoring.
Retrieve‑and‑Rerank pattern
Use a fast bi‑encoder to retrieve top‑K candidates and rescore those with a MultiVectorEncoder. Only the candidates are encoded as multi‑vectors, so your main index remains a standard dense index and token vectors are transient. This is cheaper per candidate than a cross‑encoder, since documents are encoded in a batch and then scored via matrix multiplication.
Indexing options
Several vector stores index multi‑vectors natively and support MaxSim or equivalent:
- fast‑plaid (LightOn): local Rust PLAID implementation, approximate, no server; in the example: 4,874 documents ingested in 5s, query 11ms, index ~92 MB.
- Qdrant (v1.10+): needs server; example ingest 26.3s, query 18ms, exact results.
- Weaviate (v1.29+): needs server; example ingest 41s, query 17ms, exact results.
- Vespa: example ingest ~80s, query ~75–115ms; lets you express MaxSim as a ranking pipeline.
- LanceDB, VectorChord, Milvus (v2.6.4) and other systems also provide multi‑vector support.
All four tested indexes returned the same top passages and nearly identical scores when configured to score exhaustively; fast‑plaid was approximate and had tiny score drifts. For larger corpora, approximate indexes and partitioned retrieval pipelines become necessary.
Visual document, audio and video retrieval
Late interaction is state of the art for visual document retrieval: text queries match page images (charts/tables/layout preserved) without OCR. ColPali/ColQwen models encode page images as sequences of image patches (many tokens per page), and MaxSim scores query text tokens against image patch vectors. A single page can produce hundreds of token vectors (in an example: 755 token vectors for a page vs 25 tokens for a query). Token pooling is therefore particularly relevant for visual documents.
Multimodal models such as vidore/colqwen-omni-v0.1 accept text, image, audio and video. The library reports supported modalities via model.modalities. The same encode_query / encode_document API works for audio clips and short videos (recommendations: sample frames, chunk audio to ~30s). An example zero‑shot audio retrieval run (20 conversations, avg. 28s) retrieved the correct conversation even when the query used a different surface form (nausea vs carsickness) without transcription.
Interpretability
MaxSim decomposes a document score exactly into contributions per query token and per document token, enabling precise explanations (heatmaps for images, token‑level attribution for text). Sentence Transformers provides utilities to generate ColPali heatmaps and text similarity maps showing which document token each query token matched and its share of the total score.
Token pooling to reduce index size
HierarchicalTokenPooling clusters token vectors within a document (Ward linkage on cosine distance) and replaces each cluster with its mean, roughly keeping 1/pool_factor of tokens. Example reductions on the 608k token vectors:
- pool_factor=1: 608,414 tokens (311.5 MB)
- pool_factor=2: 305,438 tokens (156.4 MB)
- pool_factor=3: 204,407 tokens (104.7 MB)
- pool_factor=4: 153,936 tokens (78.8 MB)
Pooling degrades match quality gradually; earlier experiments report ~100.6% of unpooled retrieval at pool_factor=2 on average and ~99.0% at pool_factor=3 on BEIR. Training with hierarchical pooling regularization can make pooling cheaper; LightOn provides checkpoints trained with that regularizer that load normally and pool like other checkpoints.
Speeding up inference
MultiVectorEncoder benefits from the same backend optimizations as other Sentence Transformers models: torch, onnx, openvino, half precision, Flash Attention, torch.compile. GPU fp16 + Flash Attention gave ~2.44× throughput vs fp32 in the authors' benchmarks. Some checkpoints that rely on non‑attending query expansion reject Flash Attention; use sdpa for those. On CPU, OpenVINO and int8 quantization can provide additional speedups at modest accuracy cost.
Evaluation
MultiVectorNanoBEIREvaluator runs 13 NanoBEIR subsets with MaxSim out of the box. A direct comparison of LightOn models trained on the same backbone (lightonai/LateOn multi‑vector 128d vs lightonai/DenseOn dense 768d, both 149M params) showed mean NanoBEIR NDCG@10 of 0.6868 (LateOn) vs 0.6764 (DenseOn), i.e. about 1 NDCG point advantage for the late interaction approach. LateOn outperformed DenseOn on 9 of 13 datasets; the losses on a few datasets illustrate the tradeoff between quality and index footprint.
Migration notes from PyLate and colpali‑engine
MultiVectorEncoder incorporates modeling, inference, training, and evaluation features from PyLate and colpali‑engine. PyLate, Stanford‑NLP ColBERT, and colpali‑engine checkpoints load into MultiVectorEncoder, and the release documents the changed call patterns (e.g., encode_query / encode_document instead of encode(..., is_query=True)). Save compatibility is one‑way: MultiVectorEncoder.save_pretrained output is not loadable back into PyLate/colpali.
Supported models and resources
The Hub models tagged "multi-vector" and "sentence-transformers" are the canonical list; the project is working to add the tag to all compatible checkpoints. The release includes tables of tested text retrieval and visual document retrieval models with model sizes, dimensionalities and NanoBEIR / NanoViDoRe proxy scores. Model sizes range from tens of millions of parameters to multi‑billion models for the visual cases.
Acknowledgements and further reading
The work builds on ColBERT (Omar Khattab, Matei Zaharia) and on LightOn's PyLate and fast‑plaid (Antoine Chaffin, Raphael Sourty, Paulo Moura, Amélie Chatelain). ColPali authors and token pooling researchers are acknowledged. Documentation, example scripts and longer guides are provided for usage, model creation, speeding inference, token pooling, evaluation and training.
Summary
Sentence Transformers v6.0 makes ColBERT‑style late interaction retrieval a native option in its API via MultiVectorEncoder. The release unifies multiple checkpoint formats, supports multimodal retrieval (text, image, audio, video), documents indexing backends and compression strategies, and provides tooling for interpretability and evaluation. The approach improves retrieval quality in many cases at the cost of larger indexes; the release documents practical mitigations (pooling, approximate indexes, retrieve‑and‑rerank) and performance tradeoffs for real deployments.
