About three months ago Papers with Code began a revival aimed at making open AI research more accessible and easier to navigate: so users can quickly find artifacts related to a paper, identify state-of-the-art results across AI domains, share interesting research, and build on each other’s work. A fast, accurate search engine is central to that mission for both humans and automated agents (via the website, pwc search CLI, or an agent Skill).
Why research search needs different handling
Searching research papers is not the same as ordinary text search. The engine must find exact titles or arXiv identifiers, but also handle conceptual queries such as “small language models for code generation” even when those terms do not co-occur in a document. It must interpret navigational requests (e.g., “the original BERT paper”), tolerate incomplete titles and typos, and remain responsive even when a model service is cold or temporarily unavailable.
The approach: hybrid search
Papers with Code implemented a hybrid search system that combines PostgreSQL full-text lexical search with semantic recall from pgvector embeddings. The two branches’ ranks are merged using reciprocal rank fusion (RRF). This hybrid approach typically outperforms keyword-only or vector-only systems because lexical search captures exact mentions while vector search finds semantically similar items. Rerankers can further improve results at the cost of extra latency and compute.
Hugging Face services used
Three Hugging Face services are central to the pipeline:
- Hugging Face Jobs: burstable GPU compute for batch embedding of the corpus.
- Hugging Face Storage Buckets: durable object storage to hand off artifacts between database exports, Jobs, and importers.
- Hugging Face Inference Endpoints: authenticated, low-latency embeddings for live queries and small incremental updates.
The system currently maintains embeddings for more than 110,000 active papers, sourced from arXiv and Daily Papers.
Architectural separation: offline corpus build vs. online search
The team deliberately separated expensive, throughput-oriented corpus builds from latency-sensitive online query handling. Jobs perform the heavy batch work, Buckets hold durable artifacts, and only a small query-embedding step runs on the request path behind a protected Inference Endpoint. If that Endpoint is cold, busy, or unhealthy, the application falls back immediately to full-text retrieval so users still get results quickly.
A strict embedding contract
Embedding pipelines can fail subtly (model revisions, prompt mix-ups, truncation differences, mismatched content). To avoid this, Papers with Code treats the embedding format as a versioned contract. Each paper is encoded as: normalized title + "\n\n" + normalized abstract. For every vector generation they record model repository and exact revision, output dimension, input-format version, whether it’s a query or document embedding, normalization method, and a content hash of the source title and abstract.
Production uses Qwen/Qwen3-Embedding-0.6B pinned to an exact revision and emits 256-dimensional L2-normalized vectors. Qwen3 models support two helpful features: dynamic embedding size (Matryoshka Representation Learning, MRL) to trade quality for speed/storage, and separate instruction prompts for documents and queries. The team chose 256 dimensions as a speed/storage trade-off.
Jobs: turning a database snapshot into a vector corpus
Full-corpus embedding is a batch workload that benefits from GPUs and high throughput without consuming resources between runs. Hugging Face Jobs fit this profile: a Job runs a command with a hardware flavor (for example l4x1 — an NVIDIA L4 with 24 GB VRAM). The corpus build workflow:
- Export the latest version of every paper from a repeatable-read PostgreSQL snapshot; stream rows and write bounded JSONL shards with a manifest and SHA-256 checksums.
- Sync the immutable run directory to a private Storage Bucket and mount it into a Job using hf-mount.
- The Job verifies manifests and checksums, loads the pinned model revision, sorts texts by length to reduce padding, calls encode_document in batches, reduces batch size on OOM, truncates the Matryoshka representation to 256 dims and normalizes it, writes float16 Parquet shards atomically, and records throughput, package versions, hardware and checksum metadata.
Each shard marks completion so restarted Jobs can skip already verified work. In a 5,000-paper pilot the Qwen Job encoded about 75 papers per second at 1024 dimensions on an L4 GPU. The same run could be deterministically materialized at 512 and 256 dimensions for storage/quality trade-offs without extra inference cost.
Buckets as connective tissue
Storage Buckets are mutable, S3-like storage on the Hub that can be mounted read-write in Jobs. For Papers with Code the Bucket is the boundary between three lifecycles: production database exports, ephemeral Jobs that produce vectors, and the importer that validates results before touching the search index. Artifacts live under immutable run prefixes (runs/<run-id>/ input/ and output/) and each artifact is covered by a manifest and checksum to enforce reproducibility, safe retries, cheap experiments, controlled rollouts, and simple rollbacks.
Only after importer rechecks (schemas, checksums, dimensions, normalization, unique paper IDs, and content hashes) do they load vectors into PostgreSQL, build a separate HNSW index for the new generation, and atomically mark it active when every eligible paper is covered.
Inference Endpoints on the request path
Batch embeddings handle documents; queries still need embedding at request time with the same model contract. The team deploys the pinned model as an authenticated Inference Endpoint backed by Text Embeddings Inference (TEI). The Endpoint accepts query text and returns a normalized 256-dimensional vector using the model’s query prompt.
The query client enforces strict behavior: a one-second production timeout, non-blocking concurrency limit, response validation (dimension/finite/norm), a short cache keyed by query and embedding generation, a circuit breaker after repeated failures, and no raw query text in logs (only a normalized fingerprint). If the Endpoint scales up, times out, returns malformed vectors, or lacks concurrency, the client skips the semantic branch and returns lexical results.
On the 5,000-paper pilot the 256-dim Qwen HNSW index achieved 0.9955 Recall@20 versus exact search, with 1.31 ms p50 and 2.21 ms p95 HNSW lookup latency. Its table and index used about 27% of the storage of the 1024-dim version while maintaining essentially the same ANN recall in that test.
Hybrid retrieval: combining lexical and semantic results
For each query the lexical branch retrieves up to 50 candidates via weighted PostgreSQL full-text search and the semantic branch retrieves up to 50 candidates from pgvector. The system combines ranks using weighted reciprocal rank fusion (RRF): they currently use equal branch weights and k=60 as the rank constant. RRF is robust because it fuses ranks rather than disparate scores from systems on different scales.
Dense retrieval helps conceptual queries; full-text remains excellent for exact terminology, identifiers and rare names. The system preserves deterministic identity behavior: exact titles and arXiv IDs remain top-ranked; a method taxonomy recognizes navigational searches; incomplete titles and bounded spelling mistakes use conservative trigram candidates; ambiguous fuzzy matches prefer abstention to a wrong result. The team notes hybrid search is not always necessary and recommends starting with keyword search as a cheap baseline.
One Endpoint, two update paths
Large initial corpus embeddings are produced by Jobs, but the catalog changes continuously. Launching a GPU Job for a small set of changed rows would be inefficient. Instead, an hourly incremental process gathers missing or content-changed papers and sends a bounded delta (at most 500 papers, in batches of 16) to the same TEI Endpoint using the document prompt. Each row is locked and rechecked via content hash before writing; if the paper changed during inference the vector is discarded and retried in the next run.
This creates a clear division of labor:
- Jobs: full rebuilds, new model generations, large backfills.
- Inference Endpoints: interactive query embeddings and small incremental document updates.
- Buckets: preserve large-build artifacts, make runs resumable and auditable.
The hourly path keeps the active index close to the live catalog without forcing the online endpoint into unbounded batch processing.
Related papers become cheap online
The same document embeddings power related-paper recommendations on each paper page. Because the source paper already has a stored vector, related-paper retrieval is a single nearest-neighbor query over the active generation and typically requires no model call. If a vector is missing temporarily, the application can fall back to an earlier arXiv version or task-/citation-based results.
Lessons learned
- Separate throughput work from latency-sensitive work: corpus embedding and query embedding are the same model but different infrastructure problems.
- Make storage the explicit contract between compute and production: Buckets and checksummed artifacts form a reviewable handoff.
- Pin more than the model name: revision, dimension, prompt, normalization and input formatter all affect retrieval—store and validate them.
- Design for cold starts: scale-to-zero saves cost when traffic is intermittent, but requires a fast fallback (here, lexical search).
- Smaller vectors can be a systems feature: Matryoshka embeddings allow trading quality, memory, index size and latency; in the pilot 256 dims preserved ANN recall while greatly reducing storage compared to 1024 dims.
- Activation should be boring: new generations are indexed beside the current one, coverage-checked, then atomically activated so rollback is a configuration change, not an emergency recomputation.
Try it
You can try the search at https://paperswithcode.co and the chat interface at https://paperswithcode.co/chat. The team welcomes feedback.
