Comparisons

How the Receiver compares to ClickHouse's TTL moves and storage policies and to a collector rule by service and severity, and what holding the rows written with ClickHouse tools alone takes.

How does this compare to TTL moves and storage policies

A TTL move puts older parts on a cheaper volume after ClickHouse has written and merged them. The Receiver decides per log statement which lines are written at all, before the insert.

On self-managed ClickHouse, TTL <expr> TO VOLUME or TO DISK under a storage policy moves parts to a cold volume, such as an S3 disk, as they age (MergeTree TTL). That is the right tool for storage, and the move runs during background merges, after each row has been parsed, indexed and merged. ClickHouse Cloud lists manual hot, warm and cold tiering with TTL as unsupported (Cloud compatibility), because it keeps one copy of the data on object storage.

	TTL move	Receiver
Unit	A part, by age	A log statement, by its cap and its service's total per window
Decided	In ClickHouse, after the write	On the forwarder, before the insert
Acts on	Storage	Rows written, and the insert and merge work they cost
ClickHouse Cloud	Not available	The decision sits in the collector, outside the service

How does this compare to a collector rule by service and severity

A collector rule chooses by service and severity, and on the same capture it cut more write CPU than the per-statement cap. The Receiver chooses by statement and counts each one against a number.

The OpenTelemetry Collector's routing connector sends records that match a condition on their attributes to another pipeline. ClickStack's collector build includes it, with the filter, probabilistic sampler, log dedup and transform processors (builder config). Sending the two chattiest services' INFO and DEBUG lines to the bucket on the OpenTelemetry demo capture, on one ClickHouse container, the rule cost 2.7 seconds of insert-plus-merge CPU, against 8.1 for the per-statement cap with its overflow offloaded and 10.1 with no offload (collector-rule run). The rule kept more rows and cost less, because the Receiver folds a multi-line event into one larger row and ClickStack indexes the text of every row.

What the rule cannot do is name the statement a row came from, count one statement's lines over a window, or hold a service at a total. Its condition is a fixed list, and a collector function keeps no count across records (OTTL functions). The routing connector and the S3 exporter sit in both designs; the Receiver adds the statement identity and the count that decide which lines they route.

Holding the rows written with ClickHouse tools alone

ClickHouse supplies the parts; the loop is the team's to build. Holding the rows written at a set rate per message type, with error lines never moved and the excess kept in the team's bucket, takes these pieces on ClickHouse Cloud:

1. A landing table on the Null engine that ClickStack's exporter writes to. The table stores nothing, and the materialized views on it still run over every inserted block.
2. Statement identity at insert: a regex replace that masks numbers, IDs and quoted values in the body, and a hash of the service and the masked text. The logs table has no column that names the statement (ClickStack schema). Event patterns cluster a search result at query time.
3. A counting materialized view from the landing table into an hourly count per identity and severity (incremental materialized views).
4. A budget table of rows per hour per identity, with a default, read through a dictionary with dictGetOrDefault.
5. The decider: a refreshable materialized view that rewrites the list of non-error identities over their budget every minute, read through a second dictionary with a short lifetime.
6. An admit view to otel_logs for error lines and identities under budget, and a move view to an S3 table in the team's bucket for the rest.
7. The read path: the S3 table, or a Merge table over the logs table and a view of the S3 table.
8. The meter: NewPart rows in system.part_log for the logs table by hour of event_time, read through clusterAllReplicas; written_rows in system.query_log splits between the Insert and AsyncInsertFlush entries of asynchronous inserts (the gate run). The usage cost API gives the daily compute cost.
9. Idling turned off, because an idle service suspends refreshes of refreshable materialized views (idling).

The limits of that loop:

• Every line still reaches ClickHouse and runs through the views, so the ingest work tracks raw volume.
• Routing before ClickHouse needs a second collector: ClickStack's collector build has no S3 exporter.
• Quotas key by user, address, client key or query hash, never by message.
• The mask splits a statement whose values it misses, and it merges two statements that print the same fixed text.
• The cap holds to within one refresh and one dictionary reload.

What the Receiver changes is where the pieces sit. Identity comes from a symbol library compiled from the code. The count runs per window on the forwarder, and a line over its service's total goes to the bucket before ClickHouse receives it. The landing table, the mask, the counting view and the decider are then not needed. The routing collector, the S3 table and the Merge table are the pieces that stay. The total is a number per service, set by the account or rewritten every hour by the rows run (log10x_setup_recurring with rows_per_hour) from the rows system.part_log counts; in the published run the rows written held at 0.99 to 1.04 of the set rate in every complete 10-minute window but one (1.36, when one service printed 1,422 new statements in four minutes) while load grew 2.5 times (benchmarks/clickstack-rows).

ClickHouse's docs as read for this page