Skip to content
H/ACCH/ACC
Computing infrastructure ／ Technology

OpenAI balances shared storage speed and isolation

H/ACC · AI-assisted editingSource published Edited 2026-10-23

What changed

OpenAI reports that the global median latency of Habitat storage reads fell from about 120 milliseconds to about 10 milliseconds. Its engineering retrospective connects request priorities, data placement and caching, while showing how shared resources can reduce the capacity-planning burden on product teams.

The takeawayAI-assisted analysis

A shared platform centralizes capacity allocation and cache management so product teams can reuse engineering capabilities. Read performance improves through the combined work on data placement, caching and service overhead, while workload priorities help more services share resources reliably.

The breakdown

Median read latency falls to about 10 milliseconds

In its second Habitat engineering retrospective, published on October 23, OpenAI says the global p50 latency of storage reads fell from about 120 milliseconds to about 10 milliseconds after a focused performance effort began earlier this year. The p50 is the median: half of requests take no longer than this value.

The post identifies several changes: placing data closer to requests, adding cross-region replicas for a small amount of critical data, integrating caching and rewriting services in Rust. These figures measure internal storage reads. Multiple changes contributed, and the article does not provide a full controlled comparison of each change’s individual contribution.

The tradeoffs of shared and dedicated resources

Habitat initially placed different workloads in shared, default Azure Cosmos DB containers. According to the retrospective, workloads on the same physical partition competed for throughput, allowing hot requests to affect other workloads. The team later assigned dedicated containers to critical workloads to limit the impact of failures, but each product team then had to estimate and provision its own capacity in advance.

The team subsequently worked with Azure to rate-limit hot keys, lower the priority of background requests such as bulk imports and data extraction, and improve load-based scaling. With these measures in place, the platform returned to shared containers as its default, moving more capacity planning back into jointly maintained infrastructure.

Caches stay close to requests and expire when data changes

Habitat stores copies of recently accessed data in Valkey caches close to where requests are handled. When the underlying database changes, change data capture propagates cache invalidations globally; the original post describes this propagation as taking seconds. A subsequent cache miss retrieves the updated version from storage and repopulates the cache.

Extensive replication is reserved for smaller datasets that are particularly sensitive to latency, such as authentication metadata. The post also notes that local data placement is constrained by regional storage, power and cooling capacity. Some of the slowest requests still require cross-region reads of uncached data.

H/ACC perspective

Turn waiting order into rules for sharing

H/ACC editorial interpretation: the significance of this work is that it gives shared capacity clearer allocation rules. Interactive requests and large background jobs have different service needs. Limiting hot spots and deciding which work can wait allows spare capacity to remain available to all, while reducing the chance that a sudden surge in one task will slow other workloads.

isolation can therefore come from how requests are scheduled, as well as from dedicating resources to each workload. This requires the system to recognize workloads, enforce priorities and monitor anomalies. Capacity utilization and the stability of critical requests need to be evaluated together under the same stress scenarios.

Give product teams access to shared engineering capabilities

bringing data placement, cache updates and capacity management into a shared platform gives product teams access to an established set of engineering capabilities. New features may require fewer recurring infrastructure problems to be solved, while database and cloud-platform teams can concentrate on scaling, hot spots and recovery. This is one concrete way infrastructure can improve product-development efficiency.

the next question is what happens beyond fast requests—whether waits remain predictable during cache misses, cross-region access and competition from background jobs. The median shows improvement for common requests; tail latency and data freshness determine which use cases benefit from that improvement.

Limits of the evidence

  • The performance figures and operational experience are self-reported by OpenAI; H/ACC has not independently reproduced the experiments. October 23 is the retrospective’s publication date and does not mean all the changes were completed that day.
  • The reduction from about 120 milliseconds to about 10 milliseconds is the global p50 for Habitat storage reads. It cannot be translated into an equivalent improvement in model-generation speed or ChatGPT’s overall response time.
  • Cache invalidations propagating within seconds do not guarantee strongly consistent reads. Cross-region cache misses and regional physical-capacity constraints still require separate evaluation.

What to watch nextAI-assisted analysis

  1. Will future public data break down storage-read p95 and p99 by cache hits and misses, and by same-region and cross-region access, to test whether the median improvement extends to slow requests?
  2. When hot keys and bulk writes coincide in stress tests, can interactive requests maintain stable error rates and tail latency, and how long must background jobs wait?

Sources

Reporting and research used in this story.

  1. Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)OpenAI · Jon Lee, Hrachya Hakobyan, Chaomin Yu · Source published 2026-10-23

Keep following

Read next

OpenAI uses offline training and ongoing red teaming for containmentOpenAI gives routine decisions a dedicated API
Back to feed