The symptom

While setting up monitoring for the self-hosted Weezlytics deployment, I compared its storage growth with the AWS deployment it replaced. On AWS the platform grew by about 0.5 GB a day. The new host was growing by 3 to 5 GB a day, with the same application and similar traffic. On a 1.5 TB disk, that is the difference between five years of headroom and less than one.

Looking at what was actually stored

Most of the growth was in OpenSearch, and most of that was in the log index rather than the event data. The main log index held about 40.8 million entries that had been written in roughly a day. The business events in the same cluster took about 258 MB. Everything else on the disk, such as PostgreSQL, container images and build cache, was static or small.

A sample of the log documents explained it quickly. Every one carried the DBG prefix, and they were gRPC internals: every call start, every subchannel pick, every message sent and received. Fluent Bit was doing its job, tailing container output and shipping all of it into OpenSearch, so the problem sat upstream of the pipeline. The applications were running with their minimum log level at DEBUG.

Putting numbers on it

About 470 DEBUG lines per second works out to roughly 28,000 a minute and 40 million a day. At around 80 bytes per entry that is 3.2 GB a day, or about 1.15 TB a year. I projected the disk at both log levels. At DEBUG it would be around 80% full after 12 months. At INFO, matching the AWS baseline of roughly 500,000 entries a day, it would sit near 7% after five years. A small table of the two trajectories makes the case on its own.

Three fixes, in order

  • Set the application's minimum level to Information, with Microsoft, System and Grpc overridden to Warning. Expected effect: 95 to 98% less volume.
  • If the application change takes time, drop the DEBUG lines at the edge with a Fluent Bit grep filter so they never reach OpenSearch.
  • Add an index lifecycle policy in OpenSearch that deletes logs after 30 days, so a future mistake costs a month of storage instead of a disk.

The report listed these as recommendations with expected impact for each, ordered by how fast they take effect.

What I would check earlier next time

Compare the storage growth of a new environment with its predecessor in the first week, per index and not only per disk. A log index that dwarfs the data it describes is visible long before a disk alarm fires. And put a retention policy on log indices from day one, because it is the cheapest safety net there is.