Compatibility
Kibana and existing queries, Logstash/Beats, supported forwarders and platforms, how to test on your cluster, and Data Streams + ILM integration with Retriever.
Does 10x work with my existing Kibana dashboards and queries
Yes. The 10x Engine operates pre-ingestion. Events that pass through arrive in standard Elasticsearch format. Kibana dashboards, saved searches, KQL queries, visualizations, and alerts all work unchanged. Zero reconfiguration.
All field mappings, index patterns, and document structure are preserved. Reporter and Receiver output standard Elasticsearch documents. Receiver (Compact mode) compacts events losslessly, expanded transparently by the L1ES plugin for self-hosted Elasticsearch 8.17 (Kibana dashboards and KQL queries work unchanged) and OpenSearch 2.19 (OpenSearch Dashboards work unchanged), or via Retriever for managed services.
Does this work with Logstash and Beats
Yes. The 10x Engine runs as a sidecar alongside your existing forwarder. It intercepts events at the forwarder output, optimizes them, then forwards to the Elasticsearch Bulk API. Your Logstash pipelines, Beats modules, and Elasticsearch ingest nodes continue unchanged.
Supported: Filebeat, Metricbeat, Logstash 7.x/8.x, Fluent Bit, Fluentd, and OpenTelemetry Collector.
Which forwarders can I put the 10x Engine in front of
All major log forwarders are supported. The 10x Engine runs as a Kubernetes sidecar container next to the forwarder and exchanges events over loopback TCP. No custom plugins, no forks of the forwarder image, and no subprocess management by the forwarder.
| Forwarder | Integration Type | Wire Protocol | Helm Setup |
|---|---|---|---|
| Fluent Bit | Sidecar container | Fluent Forward (TCP) | Values overlay on fluent/fluent-bit |
| Fluentd | Sidecar container | Fluent Forward (TCP) | Values + kustomize overlay on fluent/fluentd |
| Logstash | Sidecar container | NDJSON over TCP | Values overlay on elastic/logstash |
| OTel Collector | Sidecar container | OTLP/gRPC (TCP) | Values overlay on open-telemetry/opentelemetry-collector |
| Vector | Sidecar container | NDJSON in / Fluent Forward out (TCP) | Values overlay on vector/vector |
| Filebeat | Embedded process | Unix socket | Image swap (log10x/filebeat-10x) + values overlay on elastic/filebeat |
| Datadog Agent | File relay via Fluent Bit | File-based | See Fluent Bit setup |
| Splunk UF | File relay via Fluent Bit | File-based | See Fluent Bit setup |
Lifecycle: Kubernetes manages both containers. If either process exits, the kubelet restarts that container automatically via the pod's default restartPolicy.
Which Elasticsearch platforms are supported
All of them. The 10x Engine works pre-ingestion. It optimizes events before they reach any Elasticsearch-compatible backend.
| Platform | Versions |
|---|---|
| Elastic Cloud | All regions, all tiers |
| Self-hosted Elasticsearch | 7.x, 8.x (including open-source) |
| OpenSearch | 1.x, 2.x (AWS and self-hosted) |
| Coralogix | Elasticsearch-compatible ingestion |
| Logz.io | Elasticsearch-compatible ingestion |
The same Receiver sidecar runs everywhere, but the available levers differ by destination. On self-hosted Elasticsearch and OpenSearch, the L1ES plugin expands compacted events in place, so Compact mode shrinks storage losslessly. On managed backends like Coralogix and Logz.io, there is no expansion plugin, so compact is a no-op there. The cost levers on those destinations are tier_down, offload to your own S3, and the lossy choices you opt into (sample and drop).
How do I test this on my Elasticsearch cluster
- Dev, Run on your Elasticsearch log files locally. One-line install, results in minutes. No account, no credit card.
- Reporter, Deploy as a DaemonSet for pre-SIEM cost visibility. Alternatively, use the MCP server's SIEM-sample tool for agentless SIEM-side analysis via Elasticsearch REST API.
- Receiver, Deploy via Helm chart alongside your forwarder. Filter mode (sampling) or Compact mode (lossless shrink via the L1ES plugin).
- Retriever, Route events to S3, stream selected data to Elasticsearch on-demand.
Each step is independent, start with Dev to see your reduction ratio, then move to production when ready.
Do Retriever fetches break my Data Streams and ILM rollover
Retriever fetches queried events from S3 on-demand and sends them back into Elasticsearch. It is not the live path. When you fetch into a Data Stream managed by Index Lifecycle Management (ILM), the events land cleanly: Retriever outputs standard Elasticsearch JSON with original timestamps and fields intact, and the data stream routes each event to the backing index that matches its timestamp.
How it works:
- Retriever queries S3 for matching events (via Bloom filter index)
- Matched events are transformed back to original JSON, then sent to the Elasticsearch Bulk API with their original timestamps
- The data stream routes each event to the backing index matching its timestamp
- ILM rollover decisions run on index age and size, unaware of event origin
Will fetched events disrupt my ILM rollover schedule?
No. ILM policies operate on index metadata (creation time, size), not event timestamps. Fetched events with historical timestamps land in the appropriate backing index without triggering unintended rollovers. Example:
Today's live logs (timestamp: 2024-02-27T14:00:00Z)
↓
Data stream: logs-prod-default
↓
Backing index: .ds-logs-prod-default-2024.02.27-000001 (age-based ILM policy)
↓
Your S3 bucket (Retriever fetch: "last 24h of ERROR")
└─ Offloaded events with timestamps from yesterday (2024-02-26T14:00:00Z)
↓
Streamed to data stream
↓
Routed to backing index: .ds-logs-prod-default-2024.02.26-000001 (or appropriate index by timestamp)
ILM continues: Index rollover happens on schedule based on index age, not event timestamps.
Can I fetch offloaded events into the same data stream as live logs?
Yes. The typical split is:
- Live logs: the forwarder writes to Elasticsearch via Receiver (Compact mode)
- Offloaded logs: fetched back via Retriever on-demand (e.g., "pull last 24h of ERROR level")
- Target: same data stream (e.g.,
logs-prod-default)
Events from both sources land in the same data stream, so searches, dashboards, and alerts work on the combined set with no KQL changes.
Example workflow:
Retriever is customer-hosted (in-cluster or Lambda), so queries run against your own S3 bucket. The console submits a query over SQS and follows progress:
# Query S3 for ERROR events in the last 24h, then stream results to Elasticsearch
python3 console.py \
--search 'severity_level=="ERROR"' \
--since 24h \
--bucket my-log-archive \
--queue-url "$RETRIEVER_QUEUE_URL" \
--follow
The destination data stream is set in Retriever config, not on the query. Historical ERROR events from S3 appear in Kibana alongside today's live logs in the same data stream index.
What if I want to fetch into a different destination index?
You can configure Retriever to route fetched events to a separate index (e.g., logs-prod-restored). Set the target index in Retriever config and events are ingested there instead. ILM policies on the live data stream and the fetch-target index work independently.
Query latency depends on result set size (see Retriever FAQ).