How OpenAI ingests petabytes of logs every day

In June 2025, ClickHouse wrote about OpenAI's logs.^[1] OpenAI ingests petabytes of log data every day.

Its engineers faced two problems:

1. The logs had to be kept, and searched quickly.
2. A viral launch could push the load up by half overnight.

In this blog we will see where OpenAI's logs go, and how one line of code saved its logging system.

1. Where do the logs come from?

OpenAI's two major technical problems are training large language models and serving customer queries, both of which scale to millions of GPUs, generating logs in the order of petabytes of data.

These logs are then routed to a central repository, where a load balancer evenly distributes them amongst 90 shards.

The central repository is a ClickHouse cluster. A shard holds a part of the logs, and the load balancer spreads new logs evenly over all 90.

2. Which logs do people read?

Do you read last month's logs? Most people read today's.

Since most users want to see recent logs, the last two days' worth of data is kept on disk for fast access, and older logs go into blob storage.

Each shard has two replicas which can be used if things go wrong and every shard can be horizontally scaled as per requirement.

ClickHouse's post puts a number on it: about 80% of the queries touch only the last two days. Keeping recent data close and old data cheap is called tiered storage.

3. The day image generation launched

What happens to the logs when a feature goes viral?

A recent incident with the logging system was on the 25th of March when millions of users started generating Ghibli images on ChatGPT.

That was 25 March 2025, the day OpenAI launched image generation in GPT-4o. People turned their photos into pictures in the style of Studio Ghibli.

Log volume surged by 50% overnight. The cluster had gone to bed with 50% headroom in CPU, and by morning it was gone.

OpenAI sampled the logs aggressively, keeping only some of them. It also added a third replica to every shard, for queries only. Lookups stopped slowing down ingestion.

But this wasn't enough.

4. One line of code

Why wasn't it enough? The engineers profiled ClickHouse itself to find where the time went.

More than half the CPU time went into building Bloom filters. A Bloom filter is an index that lets a query skip blocks of data that can't hold what it's looking for.

Each log added to the filter sets a bit. The bit's position is the log's hash modulo the filter's size. That modulo is a division, done once for every insert.

ClickHouse's post says a division is about 30 times slower than an addition or a bitwise operation. But the size never changes. The division can be replaced with a multiplication and a bit shift.

Here is that trick in Python. The constant is worked out once from the size. After that, every insert multiplies and shifts.

import random

size = 8 * 1_000_003  # bits in the filter: never changes
constant = (1 << 64) // size + 1  # worked out once


def by_division(h):
    return h % size


def by_multiplication(h):
    low = (constant * h) & ((1 << 64) - 1)  # the low 64 bits
    return (low * size) >> 64  # a multiplication and a bit shift


random.seed(0)
hashes = [random.getrandbits(32) for _ in range(100_000)]
print("same bit for every hash:", all(by_division(h) == by_multiplication(h) for h in hashes))

same bit for every hash: True

The answers are the same, with no division. In ClickHouse it was "almost a one-line change". CPU usage fell by 40%.

So, did the logging system hold?

The result was the logging system stabilized, and the viral event of Ghibli images went through successfully.

A teammate then upstreamed the change to ClickHouse. Every user of it got the fix.

Summary

1. OpenAI ingests petabytes of logs every day into a ClickHouse cluster of 90 shards.
2. A load balancer spreads the logs evenly over the shards.
3. The last two days stay on disk, where about 80% of queries look. Older logs go to blob storage.
4. Each shard has two replicas. After the image generation launch, a third was added for queries only.
5. The launch of 25 March 2025 raised log volume by 50% overnight. Sampling and the extra replica weren't enough.
6. Half the CPU time went into Bloom filters, and one division per insert was the cost.
7. A multiplication and a bit shift gave the same answer, and CPU usage fell by 40%.

Sources

1. Why OpenAI chose ClickHouse for petabyte-scale observability, ClickHouse, 30 June 2025.
2. BloomFilter.cpp, ClickHouse source code.
3. Faster remainder by direct computation, Daniel Lemire and others, 2019.