OpenAI has developed Pando, a new backend for Git repositories designed to scale large numbers of concurrent writers and readers while remaining compatible with the Git protocol and client. The system was created because agent-driven workflows such as those enabled by Codex increased parallel development, producing many more branches, pushes, fetches and clones than the previous infrastructure could reliably handle.
Why build a new backend?
Last year OpenAI’s monorepo saw a rapid rise in size and concurrent activity, which caused slow fetches and pushes, failed merges, and periods when reliability dipped below 90%. The team wanted to keep existing Git workflows and client tools, but rebuild the backend so unrelated branch pushes could proceed in parallel and the same history could be served to many readers efficiently.
Design goals and approach
Pando targets two main goals: horizontally scalable writes (many concurrent pushes to different refs) and very high read throughput (allowing clients to saturate their network links when fetching). To achieve these without changing the Git client, Pando makes two important design choices:
- It relaxes repository‑wide, instantaneous consistency in favor of causal consistency: updates to a single ref are serializable, but a fetch spanning multiple refs may observe them at different points in time.
- It reuses packfiles received from writers rather than generating a unique pack per client, allowing the same bytes to be served to many clients with little extra computation.
When a client pushes, Pando streams the incoming packfile into a blob store as an immutable blob and records a ref version in a horizontally scalable metadata store (Turtle). Each ref version links a ref tip to the packfile that supplied it and any dependent earlier ref versions. Fetches are served by mapping the client’s have/want commit IDs to ref versions, selecting stored packfiles that cover the needed objects, and returning packfile URIs for download (an empty packfile is returned inline to satisfy the Git protocol). Pack transfers are served through regional caches (Express).
Alternatives and why Pando differs
The team evaluated two conventional scaling strategies: a three‑phase‑commit (3PC) style design with full replicas and quorumed ref updates, and a write‑ahead log (WAL) or shared manifest approach stored in blob storage. The 3PC model serializes ref updates and puts replica latency and disk performance on the push path; the WAL model allows parallel pack uploads but forces competing updates on a shared manifest and requires replicas to index packs from blob storage before serving them.
Pando avoids both bottlenecks by allowing independent ref updates to proceed without global ordering and by reusing packfiles to reduce server CPU and I/O during fetches.
Advertising refs, pack reuse and tradeoffs
Conventional Git servers advertise a large set of refs to help clients exclude objects they know the server already has. Pando instead advertises a small, controlled set (by default a designated trunk branch). That keeps advertisement size constant as branches proliferate, and it lets clients produce packfiles whose dependencies Pando can track and reuse. The tradeoff is that stacked or repeated pushes to feature branches may send redundant objects; for OpenAI’s monorepo the median packfile size under this policy was under 20 KiB and the 99th percentile under 3 MiB.
Pando records a synchronous metadata update during a push to make the new ref version immediately fetchable, and also schedules asynchronous work (like compaction) as part of the same transaction. The only contended metadata key is the ref pointer, which is protected with an atomic compare-and-swap–like operation, so unrelated refs can be pushed independently and write throughput scales by adding servers and storage units.
Fetch planning and compaction
When serving a fetch, Pando resolves have/want commit IDs to ref versions and selects previously stored packfiles that cover the missing objects. The server returns URIs for those packs; clients download them from the Express cache tier. Because a full history of L0 packs can become large, Pando compacts packs in the background using a hierarchical scheme similar to log-structured merge trees: L0 packs are gradually combined into larger covers (up to L4 in the monorepo). Compaction improves compression ratios and lets Pando plan fetches in logarithmic time across ref version distance. Unlike traditional Git servers, Pando retains packfiles at every level for serving clients rather than immediately garbage‑collecting them.
Performance and rollout
Pando was built from kickoff to production in 10 weeks and began serving production traffic in April ("April of this year"). It was implemented in Rust as a greenfield project, includes a Git conformance test suite, and was validated with deterministic simulation and continuous agent-driven testing.
Measured outcomes on OpenAI’s monorepo include:
- Median push latency reduced by nearly 5× (the team also describes this as nearly an 80% reduction in median push latency).
- Pando can deliver nearly a petabyte of Git data per day.
- In a peak example, Pando served over 57 TB of Git packs in a single hour, with more than 99% of those bytes served from local disk cache.
- Express regional caches achieve cache-hit rates above 99%, and using standard cloud VMs they observed clone/fetch operations saturating 10 Gbps links.
Pando servers perform negotiation and pack selection without unpacking packs into a local repository, which enables them to run on commodity hardware and scale to client, network, and blob store limits.
Contributions to Git and future work
Because Pando reuses packs and serves them directly to clients, the project surfaced edge cases in concurrent downloads, interrupted transfer cleanup, and resuming already completed downloads. OpenAI contributed fixes and improvements upstream to Git — including better handling for packfile-uri downloads and the concept of explicit haves — and many changes are in their public repositories or the upstream project.
Pando began as a read‑and‑write‑through proxy to the previous source control backend to enable safe migration without downtime. The transition included shipping their Git distribution to developer machines and the ChatGPT desktop app, adapters around the git CLI for phased migration, and additional tooling for code review, integrations and automations. Codex was used to help migrate checkouts and prepare environments.
Summary
By relaxing global repository consistency and reusing packfiles while keeping Git protocol compatibility, Pando enables OpenAI to scale its monorepo to thousands of concurrent writers and readers without changing developer workflows. The system significantly improved push latency, merge reliability, and throughput in production; the team is continuing to contribute upstream improvements to Git and exploring broader uses of Pando where many agents collaborate on shared repositories.
