Optimization

How the 10x Engine affects hot/warm/cold tiering, data-node count reductions, handling fields whose values change per event, and search-time overhead from the L1ES plugin.

Does the 10x Engine help with hot/warm/cold tier optimization

Yes, on self-hosted Elasticsearch or OpenSearch. Compact mode shrinks the indexed footprint losslessly in place, so each tier holds fewer bytes. Measured on the OpenTelemetry demo sample in Elasticsearch 8.17.0, compact removed 51.7% of the raw index's bytes on disk, from 28.6 MB to 13.8 MB with the templates included. On managed Elasticsearch (Elastic Cloud, OpenSearch Service), compact is a no-op at the destination, so the levers there are tier_down, offload, and sample/drop instead.

Adding Retriever (offload) moves the slices you choose out of the cluster, keeping only the hot working set indexed. Offloaded data lands in your own S3 and stays queryable on demand from there; Elasticsearch does not re-ingest it. Compact and offload are non-destructive, so the data stays reachable; sampling and drop are lossy choices you opt into.

Bonus: Offloaded events land in your own S3 at $0.023/GB-month and the Retriever returns the exact offloaded events on demand, with no rehydration and no Elasticsearch re-ingest.

How does volume reduction translate to fewer data nodes

For most self-managed clusters, data node count is driven by storage capacity, so fewer indexed bytes means fewer data nodes, and each decommissioned node retires both its EC2 instance and its EBS volume. Compact shrinks in place only on self-hosted ES/OpenSearch, where it removed 51.7% of the raw index's bytes on disk in the measurement above.

Offloaded events land in your own S3 at $0.023/GB-month, and the Retriever returns the exact offloaded events on demand rather than re-ingesting them into the cluster. Master nodes, coordinating nodes, and Kibana instances stay the same regardless of data volume.

Enter a monthly bill and node count on the homepage for a figure on your own cluster.

How does the 10x Engine handle fields whose values change per event and cause mapping explosions

The 10x Engine separates the stable structure of each log event from the variable data inside it (timestamps, IDs, UUIDs) that drives mapping explosions. Fewer distinct field values reach Elasticsearch, so the mappings stay small.

Each event is matched to its pattern name, the symbols of the scope most likely to have written it, such as Error_syncing_pod or Receive_ListRecommendations_for_product_ids. That pattern is the unit of tracking, not the raw field values.

Once cost is tracked per pattern, budget-based sampling can target the noisiest patterns before they reach Elasticsearch. Sampling is a lossy choice you opt into; it cuts ingestion volume and the count of distinct field values your mappings handle, but it does not keep every line.

What is the search-time overhead in Elasticsearch

The L1ES Lucene plugin expands compact events transparently during the fetch phase, so search returns the full original events. The expansion overhead is modest and offset by the smaller index: fewer data nodes, less SSD, lower compute. For managed Elasticsearch (Elastic Cloud, OpenSearch Service), Retriever returns the exact offloaded events on demand.

Expansion runs in the fetch phase, so its cost scales with the number of documents a query returns rather than with index size, and it is offset by the smaller index the compact format produces. For resource requirements and Kubernetes resource specs, see Node counting.

Routing Key

What does 10x add to Elastic's own tiering

Where compact does not apply, as on Elastic Cloud, Elastic's frozen tier or your own S3 stores the cheaper slice, and today a person writes the rules that choose it: which data stream a line lands in, and the ILM policy that moves that stream's indices to the frozen data tier. 10x keeps those rules current per pattern: the rules key on the pattern, which is the same on every pod from its first line, whatever lines that pod has seen and whenever it started. A miner at the edge depends on both: two Drain3 instances that each see half of one stream name a line seen 2 to 10 times differently 39.78% of the time (pattern-identity benchmark). The compiler runs in CI or on a schedule, so new log statements, including ones a coding agent wrote, are named as the code ships; a scheduled check (tenx-recur) costs each pattern, routing changes arrive as pull requests to the policy repo, and caps act on their own.