SharedMergeTree vs ReplicatedMergeTree: The Business Case

If you run ClickHouse® software at scale on your own hardware, you are paying a tax that never shows up as a line item. The fix is an engine called SharedMergeTree, and the case for it starts with your storage bill.

Here is the first part of it. ReplicatedMergeTree keeps a full copy of your data on every replica. Run two replicas, and you store everything twice. Run three, and you store it three times. A table that grows by half a terabyte a month adds a full terabyte a month to your footprint on a two-replica setup, and that gap compounds every month you keep the data around.

Most teams we talk to have never put a name to this. They know the cluster is expensive, but they have not traced the cost back to the engine. So let us trace it, because there are actually four taxes stacked on top of each other, and then let us show you what changes if you stop paying them.

This is the business case, and by the end you will know whether the alternative, decoupling compute and storage, is worth it for your workload, and you will have a checklist you can keep.

What SharedMergeTree actually is

Start with the one idea everything hangs on: storage and compute come apart.

In a classic ClickHouse deployment, every node owns the whole dataset on local disk. Compute and storage sit on the same machine, so when you want more of one, you get more of the other whether you need it or not. SharedMergeTree breaks that link. The data lives once in object storage, S3 or an equivalent, and the compute nodes on top are stateless. We have written up both halves of that: how the data moves off local disk and into object storage, and how stateless compute stays fast reading from it.

The part your engineers feel first: one table is one table. No Distributed table sitting on top of a sharded local table. No sharding key to pick. No ON CLUSTER bolted onto every schema change and read query. You point at one endpoint, and it behaves the same whether there is one node behind it or fifty.

To make that concrete, here is the same logical table in both worlds. Self-hosted, it is a set of objects that have to stay in step:

-- a physical table per shard, replicated for durability
CREATE TABLE events_local ON CLUSTER main (...)
ENGINE = ReplicatedMergeTree(...)
ORDER BY (...);

-- a Distributed table to fan queries across the shards
CREATE TABLE events ON CLUSTER main AS events_local
ENGINE = Distributed(main, default, events_local, rand());

On shared storage, it is one object:

CREATE TABLE events (...)
ENGINE = SharedMergeTree
ORDER BY (...);

The first version bakes the shard layout into your schema, your client, and every migration you ever run. The second has no shard layout to leak.

On ObsessionDB, we run our own version, alloy, built against the same SharedMergeTree API, so the developer experience is identical while the implementation underneath is ours. None of this is exotic, either. Separating storage from compute is the default shape of cloud data systems now. We did not invent it, and neither did ClickHouse. The interesting question is what it does to your bill and your on-call rotation.

Why ReplicatedMergeTree breaks your cost structure

Four mechanics, and they stack.

The first is the one we opened with: you pay for the same data more than once. Replication means copies, not shards. Every replica holds a full copy of the data it is responsible for, so durability costs you a multiple of your real data size. SharedMergeTree stores one copy in object storage. When you add a compute node, you are not adding another copy of the data, you are adding a reader. Replica count no longer drives the storage bill.

The second is quieter, and it wastes the most hardware. When compute and storage are welded together, you cannot scale them separately. Say your workload needs more storage but the same compute. On a replicated cluster, you add whole nodes anyway, and now you are paying for compute you will never touch just to hold bytes. It runs the other way too: a spiky query load pushes you to provision storage-heavy machines for what is really a compute problem. We hear this constantly from teams looking at a move, the over-provisioning they never chose. Decoupling is the fix. Storage grows by adding bytes to object storage. Compute grows by adding stateless nodes. Two dials instead of one.

The third tax is operational, and people underestimate it because it stays invisible until you try to grow. Adding capacity to a sharded cluster is not a button. There is no automatic shard rebalancing: the data already sitting on your old shards does not move to a new one until you move it yourself, by hand. In practice that is copying large tables between nodes, a job measured in hours and days, run carefully so you do not knock the cluster over. Add the coordination layer you have to keep healthy and the shard-awareness that leaks into your application, and you get the real shape of it: a replicated cluster is a genuinely elegant design that still takes several people to keep even a small one alive. Under SharedMergeTree, adding a node is a metadata operation. The node announces itself and starts serving. There is nothing to copy.

The fourth is how the two models behave under load, and here we will show you our own numbers. On a classic replicated setup, doubling throughput has cost us somewhere around 2.3 to 2.5 times the resources, because the coordination and duplication overhead rides along with it. On shared storage, we see closer to 1.2 times for the same doubling. The difference is that we get there without re-copying the dataset, and in the time it takes a node to boot rather than the time it takes to rebalance a cluster.

Scaling out is not the only lever, either. Because compute is separate from storage, you can run more than one compute pool against the same data, each sized for its own job: one for ingest, one for the API, one for the heavy analytical queries. On a replicated cluster, those workloads share the same nodes and compete, so the batch job that pins every core is also why your dashboards stall. Give it a pool of its own and the serving path never feels it. That is compute-compute separation, and it works precisely because the data underneath is shared, not copied.

The other side of the ledger

We do not trust a pitch with no downside, so here it is. SharedMergeTree is not free, and the cost lands somewhere people do not expect.

Object storage is not slow for lack of capacity. It is slow because of latency and request limits. AWS puts S3 first-byte latency at "roughly 100-200 milliseconds", against microseconds for a local NVMe disk. And there is a ceiling on how often you can ask: S3 gives you, in AWS's words, at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned Amazon S3 prefix. That sounds like plenty until you remember that a ClickHouse part is a directory full of small files, and a careless layout can turn a single insert into thousands of tiny objects, all fighting over that same request budget. This is the real engineering problem a shared-storage engine has to solve, and it is why one cannot exist without a serious caching layer in front of the object store.

And there is a workload where none of this pays off. If your dataset is small, static, and fits comfortably on one fast disk, a single self-hosted node is faster. In our own launch notes we say the same thing: small workloads can run slightly slower on separated storage than on a single machine with a local disk. Decoupling earns its keep at scale.

When SharedMergeTree is the right call

Here is the part to save. Reach for SharedMergeTree when most of these are you:

• Your dataset is large and growing, and replication is quietly multiplying your storage.
• You run real-time ingestion, serving, and heavy analytics against the same data, and they fight over the same nodes.
• Your load is spiky or seasonal, and you want to add compute without touching storage.
• Resharding downtime and babysitting the coordination layer are costing your team real hours.

Stay on a single node or plain ReplicatedMergeTree when:

• Your data is small and static and fits on one fast local disk.
• You need to completely avoid the latency delay.
• You would rather have full control than elasticity.

A rough rule: you do not need all four. If even one of those top bullets is you, you are paying the tax every month, and that alone is reason enough to look hard at ObsessionDB.

Where this leaves ObsessionDB

You can probably guess where we sit. ObsessionDB gives you the SharedMergeTree architecture without ClickHouse Cloud's pricing model or its lock-in. Same API, our own engine underneath, built by people who ran ClickHouse at near-petabyte scale for Numia and got tired of paying the replication tax in storage and weekends.

We are not the only ones, and we are not going to pretend otherwise. Our argument is narrower: the same architecture, hosted where you want it, at a better ratio of performance to what you pay. On a ten-billion-row benchmark, we measured 35% faster queries and up to 9x better performance per dollar than ClickHouse Cloud, and that comparison has its own writeup rather than being re-run here.

If the storage math in this piece reads like your cluster, that is the conversation to have. We run workload assessments: bring your real queries and your real data shape, and we will put the side-by-side on hardware in front of you.