Skip to content

Optimization

What the Receiver does on ClickHouse and ClickStack, what that changes on a ClickHouse Cloud bill, how to read it in ClickHouse's own system tables, and what ClickHouse's own tools cover.

What does the Receiver do on ClickHouse

The Receiver holds each service's rows at a total per window on each node, and the lines over it go to the account's own S3 bucket before ClickHouse receives them. On ClickStack the Receiver sits beside the OpenTelemetry Collector and returns every line marked with its route. A second, stock contrib collector, beside ClickStack's own, sends the lines marked offload to the bucket and everything else to ClickStack unchanged (the end to end run). An offloaded line is never inserted, indexed or merged by ClickHouse. On ClickHouse the lever is offload; compact and tier down serve other destinations.

The held number is the service total: with the regulator counting events, it caps the lines a service passes per 4-minute window on that node, and one returned record is one row in otel_logs (a multi-line event folds into one record). Once a service reaches its total, every further line takes the service's action. Three kinds of line pass and count toward the total: the baseline sample (five per pattern per window), the error keep counts (50 per pattern per window for ERROR, CRITICAL and FATAL) and lines a mute entry passes. The floor therefore grows with the number of nodes and patterns, and an error storm on one node can exceed that node's total. A fixed total holds each node. The rows run (log10x_setup_recurring with rows_per_hour) holds the fleet at a rows-per-hour figure: every hour it reads the NewPart rows ClickHouse wrote into the logs table from system.part_log and rewrites the per-statement limits and the service totals in lines. In the published run the rows written held at 0.99 to 1.04 of the set rate in every complete 10-minute meter window but one, from the first growth step to the end of the hold at 2.5 times the starting load, through an error storm and a new service, and every line over a total reached the bucket before ClickHouse received it. That window read 1.36: one service printed 1,422 new statements in four minutes, and each passed its sample. Scored against its registered criteria, the run fails on three counts: that window, the windows above 1.00 (the registered 10-minute bound is the set rate with no allowance for the floor), and 33 rows ClickStack inserted twice. The README scores each criterion. The floor is the sample of five lines per statement per window on each node, which passes whatever the limit; the run subtracts it from the allowance and reports it beside the rate. A service that prints many new statements in one window passes the sample of each, over its total, until the next run rewrites the limits.

Lines over the total are kept in the account's own bucket, written as objects (JSON by default) with the service and the day in the path. The shipped recipe passes lines from the collector to the Receiver in memory, so lines in flight when either restarts are lost (restart run).

Built with ClickHouse tools alone, holding rows per message type takes:

  • a Null-engine landing table that ClickStack's exporter writes to;
  • a regex mask and a hash on every row for statement identity;
  • a counting materialized view;
  • a budget table read through a dictionary;
  • a refreshable materialized view that rewrites the over-rate list every minute;
  • an admit view and a move view to an S3 table;
  • idling turned off.

ClickStack's collector build has no S3 exporter, so on that build every line still reaches ClickHouse and runs through those views. A second contrib collector can route lines to the bucket first. Collector functions keep no count across records, so its list is pushed into every collector's config. The Receiver counts each statement on the forwarder, from identity compiled from the code. The landing table, the mask, the counting view and the decider are then not needed. The pieces that stay are the routing collector, the S3 table and the Merge table. Each piece with ClickHouse's doc behind it: Comparisons.

What does it change on a ClickHouse Cloud bill

Fewer rows written means less insert and merge work. The invoice moves when that lets the autoscaler hold a smaller size or lets the service idle longer.

ClickHouse Cloud meters compute per minute in 8 GiB units and storage on compressed bytes (billing). On one capture, with rows removed from the same set, half the rows cost 39% of insert-plus-merge CPU and a quarter cost 17% (compute-vs-rows).

The autoscaler scales a service down when CPU stays under 37.5% of its allocation (scaling recommendations), and never below the minimum the account sets (vertical autoscaling). On a service already at its minimum, the compute line stays at that minimum however few rows the service writes. How much of a bill is writing rows, as against queries, is a figure each estate's own system.query_log answers. For Managed ClickStack, ClickHouse puts ingest compute at under a cent per GB per month and sizes query compute separately (Managed ClickStack).

A collector rule by service and severity cut insert-plus-merge CPU further on the same capture: 2.7 seconds, against 8.1 for the per-statement cap and 10.1 with no offload (collector-rule run). What that rule cannot do is name the statement a row came from, or count one statement's lines against a number.

How to read it in ClickHouse's own tables
  • Rows written per hour: sum rows of NewPart events in system.part_log for the logs table, grouped by hour of event_time. On one run this equalled the table's count to the row, while written_rows in system.query_log moved between the Insert and AsyncInsertFlush entries of ClickStack's asynchronous inserts (the gate run). On ClickHouse Cloud each node holds its own log, so read it through clusterAllReplicas.
  • Rows stored: active rows in system.parts for the logs table, as a cross-check.
  • The Receiver's side: the emitted_events and all_events counts by routeState show what each service passed and what it sent to the bucket.
  • The bill: the Cloud usage cost API returns compute cost per service and storage cost per warehouse, per day (UTC), and its records can change until they are locked. Read it beside the rows written. The rows written are the control signal; the day's cost is the report.

Reading system tables keeps a Cloud service from idling (system tables), so one read an hour is the meter's own cost.

What ClickHouse handles, and reading the bucket
  • Storage. TTL moves to a cold volume are the right tool for storage on self-managed ClickHouse. ClickHouse Cloud keeps one copy on object storage and has no storage tier a TTL can move parts to (managing data).
  • Lossless compaction. ClickStack's own codecs and text index absorb most of the repetition, so compact is not a lever on ClickHouse (clickhouse-clickstack).
  • Insert batching. Batch size and asynchronous inserts are set in the collector and in ClickHouse, and they cut write CPU for every row. They are collector and ClickHouse settings, and they apply alongside the Receiver.
  • Reading the bucket. ClickHouse reads the objects in place through an S3 table, and a Merge table over the hot table and the S3 table answers a query from both sides. A query filtered on time alone opens every object in range, so dashboards over the Merge table filter on service or day. The day in the path is the upload day, not the record's day. The S3 table is created after the first object exists under its prefix: on ClickHouse releases without the fix for issue 116888, a table read while its prefix is empty answers path filters with zero rows.
  • Counts and alerts. Once part of a statement goes to the bucket, the hot table alone undercounts it. Alerts and charts that count lines read the Merge table or the counts-per-type table the recipe creates (counts run).

One-day test

How to test it on one ClickStack deployment in a day

Nothing in the path until step 3, and every number read from ClickHouse's own tables.

  1. Census, with nothing installed. Run the cost POC with siem: clickhouse against the logs table. The connector reads CLICKHOUSE_URL with CLICKHOUSE_USER and CLICKHOUSE_PASSWORD, or CLICKHOUSE_API_KEY. The ClickStack schema is mapped by hand: clickhouse_table: otel_logs, clickhouse_timestamp_column: Timestamp, clickhouse_message_column: Body, clickhouse_service_column: ServiceName, clickhouse_severity_column: SeverityText. The local engine runs the analysis, so the pulled events stay on the machine. The top view names the statements that write the rows.

  2. Baseline the meter. Rows written into the logs table per hour, over a day:

    SELECT toStartOfHour(event_time) AS hour, sum(rows) AS rows
FROM system.part_log
WHERE event_type = 'NewPart'
  AND database = 'default' AND table = 'otel_logs'
  AND event_time >= now() - INTERVAL 24 HOUR
GROUP BY hour ORDER BY hour

    On ClickHouse Cloud, read clusterAllReplicas(default, system.part_log), since each node holds its own log. This count equalled the table's own count to the row in the gate run; written_rows in system.query_log did not, because ClickStack's asynchronous inserts split the rows across two entry kinds.

  3. Receiver in read-only mode beside the ClickStack collector (receiverReadOnly): events pass through untouched and the Receiver publishes per-statement metrics to the account's own metrics backend. Top patterns ranks them. Pick one chatty service.

  4. One service, one total. Give that service a total per window in events, action offload, and switch the Receiver to read-write. The Retriever advisor renders the ClickHouse recipe: the second collector that routes routeState == "offload" to the bucket, the S3 table, the counts table and the Merge table, in the order the recipe gives (the S3 table after the first object exists).
  5. Read the three numbers after an hour. The meter from step 2, the active rows in the table (SELECT sum(rows) FROM system.parts WHERE database = 'default' AND table = 'otel_logs' AND active), and one statement's count through the Merge table against the hot table alone (count() on otel_logs_all and on otel_logs where LogAttributes['tenx_hash'] is the statement's hash). Hot rows plus rows in the objects should equal the records the Receiver returned.
  6. Restart it. Stop the Receiver for twenty seconds mid-feed and count again. The shipped recipe passes lines from the collector to the Receiver in memory, and lines in flight were lost in every run measured (restart run). The deployment's own collector settings decide what is held, so the number to carry is the one measured there.
  7. Turn on the rows run after the day. log10x_setup_recurring with rows_per_hour: auto reads the meter from step 2 every hour, reports the rows the last hour wrote and holds nothing; with a rate in place of auto, each run rewrites the limits and totals to hold that rate.

The day shows which statements write the rows, one service held at a total, the rows written per hour, and what a restart costs. The day does not show the bill: ClickHouse Cloud compute moves through the autoscaler (what it changes on the bill), and a rule by service and severity at the collector cuts more write CPU on one capture (Comparisons).

Names on code the team does not own

How to check the pattern names on a deployment's own code and images

Each line is named from a symbol library compiled from source, binaries and container images (how a line gets its name). The shipped library covers the repositories and images in its manifest. A line with no library word is named template_ plus a hash of its shape, so the names a deployment gets are a thing to read, not to take on trust.

  1. Read the census from step 1 above. A name made of words (Error_syncing_pod) came from a statement the library knows. A name that starts with template_ is a line no library word covered. The share of rows under template_ names is this deployment's coverage figure.
  2. Compile the deployment's own images. The compiler scans GitHub repositories and container images (Docker pull) and links the symbols into a library the engine loads beside the shipped one; the compile tool runs it. Rerun the POC and read the template_ share again.
  3. Hold a name across days. Run the POC on a second day. A statement's name is the same string while the engine version, the library and the configuration stay the same. With the Receiver installed, Pattern diff returns the new, retired and persistent names across any boundary.
  4. Hold a name across nodes. Two engine instances fed different halves of one stream named 299,997 of 300,000 probe lines the same, with the reproduce command in benchmarks/pattern-identity. The template hash of a multi-line event can depend on the lines read before it; the pattern name does not.

A value made of library words enters the name, so one statement that prints an identifier built from known words yields one name per identifier (benchmarks/intent). The census shows that too.