Habitat storage platform
OpenAI’s online storage platform handles access, isolation and caching between products and underlying data resources
What changed
OpenAI reports that the global median latency of Habitat storage reads fell from about 120 milliseconds to about 10 milliseconds. Its engineering retrospective connects request priorities, data placement and caching, while showing how shared resources can reduce the capacity-planning burden on product teams.
Key readingAI-assisted analysis
A shared platform centralizes capacity allocation and cache management so product teams can reuse engineering capabilities. Read performance improves through the combined work on data placement, caching and service overhead, while workload priorities help more services share resources reliably.
In focusOpenAI balances shared storage speed and isolation
This breakdown starts with a story focused on this topic. It is not a complete assessment.
The facts
01Median read latency falls to about 10 milliseconds
In its second Habitat engineering retrospective, published on October 23, OpenAI says the global p50 latency of storage reads fell from about 120 milliseconds to about 10 milliseconds after a focused performance effort began earlier this year. The p50 is the median: half of requests take no longer than this value.
The post identifies several changes: placing data closer to requests, adding cross-region replicas for a small amount of critical data, integrating caching and rewriting services in Rust. These figures measure internal storage reads. Multiple changes contributed, and the article does not provide a full controlled comparison of each change’s individual contribution.
02The tradeoffs of shared and dedicated resources
Habitat initially placed different workloads in shared, default Azure Cosmos DB containers. According to the retrospective, workloads on the same physical partition competed for throughput, allowing hot requests to affect other workloads. The team later assigned dedicated containers to critical workloads to limit the impact of failures, but each product team then had to estimate and provision its own capacity in advance.
The team subsequently worked with Azure to rate-limit hot keys, lower the priority of background requests such as bulk imports and data extraction, and improve load-based scaling. With these measures in place, the platform returned to shared containers as its default, moving more capacity planning back into jointly maintained infrastructure.
03Caches stay close to requests and expire when data changes
Habitat stores copies of recently accessed data in Valkey caches close to where requests are handled. When the underlying database changes, change data capture propagates cache invalidations globally; the original post describes this propagation as taking seconds. A subsequent cache miss retrieves the updated version from storage and repopulates the cache.
Extensive replication is reserved for smaller datasets that are particularly sensitive to latency, such as authentication metadata. The post also notes that local data placement is constrained by regional storage, power and cooling capacity. Some of the slowest requests still require cross-region reads of uncached data.
H/ACC insights
Turn waiting order into rules for sharing
H/ACC editorial interpretation: the significance of this work is that it gives shared capacity clearer allocation rules. Interactive requests and large background jobs have different service needs. Limiting hot spots and deciding which work can wait allows spare capacity to remain available to all, while reducing the chance that a sudden surge in one task will slow other workloads.
isolation can therefore come from how requests are scheduled, as well as from dedicating resources to each workload. This requires the system to recognize workloads, enforce priorities and monitor anomalies. Capacity utilization and the stability of critical requests need to be evaluated together under the same stress scenarios.
Give product teams access to shared engineering capabilities
bringing data placement, cache updates and capacity management into a shared platform gives product teams access to an established set of engineering capabilities. New features may require fewer recurring infrastructure problems to be solved, while database and cloud-platform teams can concentrate on scaling, hot spots and recovery. This is one concrete way infrastructure can improve product-development efficiency.
the next question is what happens beyond fast requests—whether waits remain predictable during cache misses, cross-region access and competition from background jobs. The median shows improvement for common requests; tail latency and data freshness determine which use cases benefit from that improvement.
What to watchAI-assisted analysis
- Will future public data break down storage-read p95 and p99 by cache hits and misses, and by same-region and cross-region access, to test whether the median improvement extends to slow requests?
- When hot keys and bulk writes coincide in stress tests, can interactive requests maintain stable error rates and tail latency, and how long must background jobs wait?
Limits of this assessment
- The performance figures and operational experience are self-reported by OpenAI; H/ACC has not independently reproduced the experiments. October 23 is the retrospective’s publication date and does not mean all the changes were completed that day.
- The reduction from about 120 milliseconds to about 10 milliseconds is the global p50 for Habitat storage reads. It cannot be translated into an equivalent improvement in model-generation speed or ChatGPT’s overall response time.
- Cache invalidations propagating within seconds do not guarantee strongly consistent reads. Cross-region cache misses and regional physical-capacity constraints still require separate evaluation.