Elasticsearch

Cut Elastic costs 50-80%. Reduce ingestion costs via edge optimization and on-demand S3 streaming.

Compatibility

Does 10x work with my existing Kibana dashboards and queries

Yes. The 10x Engine operates pre-ingestion -- events that pass through arrive in standard Elasticsearch format. Kibana dashboards, saved searches, KQL queries, visualizations, and alerts all work unchanged. Zero reconfiguration.

All field mappings, index patterns, and document structure are preserved. Edge Reporter and Edge Regulator output standard Elasticsearch documents. Edge Optimizer compacts events losslessly -- expanded transparently by the L1ES plugin for self-hosted Elasticsearch 8.17 and OpenSearch 2.19 (Kibana dashboards and KQL queries work unchanged), or via Storage Streamer for managed services.

Does this work with Logstash and Beats

Yes. The 10x Engine runs as a sidecar alongside your existing forwarder -- it doesn't replace Logstash, Beats, or any part of your pipeline.

Your Logstash pipelines, Beats modules, and Elasticsearch ingest nodes all continue unchanged. The 10x Engine intercepts events at the forwarder output, optimizes them, then forwards to the Elasticsearch Bulk API. Nothing to reconfigure in Logstash, Beats, or your ingest pipelines.

Supported: Filebeat, Metricbeat, Logstash 7.x/8.x, Fluent Bit, Fluentd, and OpenTelemetry Collector.

Which forwarders are supported with detailed versions

All major log forwarders are supported. The 10x Engine integrates via standard input/output protocols — no custom plugins or agents required.

Forwarder	Minimum Version	Integration Type	Socket Type	Notes
Fluent Bit	None documented	Lua filter + subprocess	TCP or Unix	Launches 10x on first event
Fluentd	None documented	exec_filter plugin	Unix socket	Auto-restart child process
Filebeat	None documented	JavaScript processor	Unix socket	Cannot use console output
Logstash	7.x+	Pipe output plugin	Unix/TCP (5160)	Works with Logstash pipelines
OTel Collector	0.143.0+	Syslog RFC5424 exporter	Unix/TCP	Separate service, not sidecar
Datadog Agent	Any	File relay (via Fluent Bit)	File-based	Indirect integration
Splunk UF	Any	File relay (via Fluent Bit)	File-based	Indirect integration

Key note: All integrations use standard protocols (syslog RFC5424, Fluentd forward protocol, JSON). The only explicit version requirement is OTel Collector v0.143.0+ for Unix socket support in the syslog exporter.

Which Elasticsearch platforms are supported

All of them. The 10x Engine works pre-ingestion -- it optimizes events before they reach any Elasticsearch-compatible backend.

Platform	Versions
Elastic Cloud	All regions, all tiers
Self-hosted Elasticsearch	7.x, 8.x (including open-source)
OpenSearch	1.x, 2.x (AWS and self-hosted)
Coralogix	Elasticsearch-compatible ingestion
Logz.io	Elasticsearch-compatible ingestion

Same optimization pipeline, same forwarder sidecar, same results -- regardless of where your Elasticsearch runs.

Storage Streamer: Data Streams & ILM Integration

Storage Streamer streams queried events from S3 back into Elasticsearch. If you're using Data Streams with Index Lifecycle Management (ILM) for automated index rollover, you may have questions about integration:

Q: Will streamed events from S3 integrate cleanly with my existing data stream?

Yes. Storage Streamer outputs standard Elasticsearch JSON documents with original timestamps and fields intact. These events integrate cleanly into your data stream's backing indices -- no special configuration needed.

How it works:

Storage Streamer queries S3 for matching events (via Bloom filter index)
Matched events are transformed back to original JSON format (full field fidelity)
Events are sent to Elasticsearch Bulk API with timestamps from the original events
Data stream receives events and routes them to appropriate backing index based on timestamp
ILM policies continue unchanged -- rollover decisions based on index age/size, unaware of event origin

Q: Will streamed events disrupt my ILM rollover schedule?

No. ILM policies operate on index metadata (creation time, size), not event timestamps. Streamed events with historical timestamps integrate into the appropriate backing index without triggering unintended rollovers. Example:

Today's live logs (timestamp: 2024-02-27T14:00:00Z)
     ↓
Data stream: logs-prod-default
     ↓
Backing index: .ds-logs-prod-default-2024.02.27-000001  (age-based ILM policy)
     ↓
S3 archive (queried: "last 24h of ERROR")
└─ Events with timestamps from yesterday (2024-02-26T14:00:00Z)
     ↓
Streamed to data stream
     ↓
Routed to backing index: .ds-logs-prod-default-2024.02.26-000001 (or appropriate index by timestamp)

ILM continues: Index rollover happens on schedule based on index age, not event timestamps.

Q: Can I stream regulated events into the same data stream as live logs?

Yes. The typical use case is:

Live logs: Stream continuously to Elasticsearch via Edge Optimizer
Archived logs: Query via Storage Streamer on-demand (e.g., "pull last 24h of ERROR level")
Target: Same data stream (e.g., logs-prod-default)

Events from both sources merge seamlessly in Kibana. Searches, dashboards, and alerts work on the combined dataset without any changes to your KQL queries or visualization logic.

Example workflow:

# Query S3 for errors in last 24h, stream to same data stream as live logs
curl -X POST https://streamer.log10x.com/query \
  -d '{
    "from": "now(\"-24h\")",
    "to": "now()",
    "search": "severity_level == \"ERROR\"",
    "target": "logs-prod",
    "elasticsearch_endpoint": "https://your-elastic-cloud.es.us-central1.gcp.cloud.es.io:9243"
  }'

Results: Historical ERROR events from S3 appear in Kibana alongside today's live logs in the same data stream index.

Q: What if I want to stream to a different destination index?

You can configure Storage Streamer to route streamed events to a separate index (e.g., logs-prod-archive-restored) if needed. Set the target index in Storage Streamer config; events will be ingested there instead. ILM policies on both the live data stream and archive index work independently.

Performance notes:

Data stream ingestion continues at normal rate (unaffected by Storage Streamer queries)
Streamed events use Elasticsearch Bulk API, processed at same rate as HEC events
ILM index rollover adds no overhead (operates on index metadata, not per-event)
Typical query: ~2-30 seconds depending on result set size (see Storage Streamer FAQ)

Local Development & Testing

Docker Compose Setup for L1ES Plugin Testing

For local development and testing of the L1ES plugin, use this Docker Compose setup:

version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.17.0
    environment:
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.security.enabled=false  # For dev only; enable in production
    ports:
      - "9200:9200"
    volumes:
      - ./l1es-plugin-0.3.0.es.8.17.0.zip:/tmp/l1es-plugin.zip:ro
      - es-data:/usr/share/elasticsearch/data

  setup:
    image: curlimages/curl:latest
    depends_on:
      elasticsearch:
        condition: service_started
    entrypoint: >
      sh -c "
      sleep 10 &&
      echo 'Installing L1ES plugin...' &&
      curl -X POST 'http://elasticsearch:9200/_plugins/_install?location=file:///tmp/l1es-plugin.zip' || true &&
      sleep 5 &&
      echo 'Setting up plugin indices...' &&
      curl -X POST 'http://elasticsearch:9200/_l1es/setup' &&
      echo 'Done!'
      "

volumes:
  es-data:

Usage:

docker-compose up -d

# Wait for setup to complete, then verify:
curl -s http://localhost:9200/_cat/indices?v | grep l1es

# Stop when done:
docker-compose down -v

What this sets up:

Single-node Elasticsearch 8.17.0 with L1ES plugin
Two internal indices: l1es_dml (templates) and l1es_dml_indices (field mappings)
Plugin endpoints: _l1es/setup, _l1es/add-dml-index, _l1es/query, etc.
Ready for development and testing without x-pack security

Configuration & Capacity Planning

What does dmlSizeToSearch do and when do I need to increase it

dmlSizeToSearch limits how many templates the plugin scans per query. Default: 10,000.

How it works:

When you query an index with encoded events, the plugin: 1. Fetches templates from l1es_dml index (up to dmlSizeToSearch limit) 2. Matches incoming events against those templates 3. Decodes matched events; leaves unmatched events as-is

What happens when you exceed the limit:

If you have 50,000 templates but dmlSizeToSearch=10000: - Plugin scans only first 10,000 templates - Events matching templates #10,001-50,000 won't decode (silent — no error) - Result: Partial decoding, some events remain encoded

When to increase it:

Monitor template count:

GET l1es_dml/_count

Under 10,000: No action needed
10,000-50,000: Increase dmlSizeToSearch to 50000 in l1es.yml
Over 50,000: Contact engineering; consider template partitioning

Trade-off: Increasing dmlSizeToSearch uses more memory and may slow queries. Recommended max: 100,000.

Query Semantics: Why do I need to include the 'fields' parameter

The fields parameter in Elasticsearch queries triggers the fetch phase, which is when L1ES decoding happens.

Without fields parameter:

GET my-logs/_search
{
  "query": { "match": { "message": "error" } }
}

Response: _id and _score only. Encoded events remain encoded (L1ES doesn't run).

With fields parameter:

GET my-logs/_search
{
  "query": { "match": { "message": "error" } },
  "fields": ["message", "timestamp", "level"]
}

Response: Specified fields are returned (AND L1ES plugin decodes encoded events during fetch phase).

Kibana automatically includes fields — if you're using Kibana, this is handled transparently. For direct API calls or custom applications, always include fields if you want decoding to happen.

Performance note: Including fields adds modest overhead (~1-2ms per event) for the fetch phase. This is where decoding happens.

Operational Procedures

How do I monitor L1ES plugin health in production

Health check endpoint:

curl http://localhost:9200/_l1es

Expected response:

{
  "name": "l1es-plugin",
  "version": "0.3.0",
  "elasticsearch_version": "8.17.0",
  "status": "active"
}

Set up Elasticsearch Watcher alert (Kibana):

POST _watcher/watch/l1es_plugin_health
{
  "trigger": { "schedule": { "interval": "5m" } },
  "input": {
    "http": {
      "request": {
        "url": "http://localhost:9200/_l1es",
        "method": "GET"
      }
    }
  },
  "condition": {
    "compare": { "ctx.payload.status": { "ne": "active" } }
  },
  "actions": {
    "notify_team": {
      "email": {
        "to": "ops@company.com",
        "subject": "L1ES Plugin Health Alert"
      }
    }
  }
}

Monitor template index health:

# Check shard status
GET _cat/shards/l1es_dml?v
# All shards should be GREEN (STARTED)

# Check index stats
GET l1es_dml/_stats
# Look for: docs.count (template count), store.size_in_bytes

Backup and disaster recovery for L1ES plugin

Monthly backup procedure:

L1ES stores two critical indices: templates and field mappings.

# Backup 1: Templates (monthly)
curl -X GET "localhost:9200/l1es_dml/_search?size=10000&scroll=5m" \
  -H "Content-Type: application/json" > templates_backup_$(date +%Y%m%d).json

# Backup 2: Field mappings (monthly)
curl -X GET "localhost:9200/l1es_dml_indices/_search?size=1000" \
  -H "Content-Type: application/json" > mappings_backup_$(date +%Y%m%d).json

# Store in S3 or backup system
aws s3 cp templates_backup_*.json s3://my-backups/l1es/

Recovery: l1es_dml index deleted

Check what's lost:

GET l1es_dml/_search
# Should return 0 if index is gone

Restore from backup:

# Re-create and bulk-load templates from backup
curl -X POST "localhost:9200/_bulk" \
  -H "Content-Type: application/json" \
  --data-binary @templates_backup_20260227.json

Verify restoration:

GET l1es_dml/_count
# Should match pre-deletion count

Impact during recovery: Queries will NOT decode until templates are restored (encoded events remain encoded). Recovery time: ~5 minutes for 100K templates.

Recovery: l1es_dml_indices deleted

Re-register field mappings:

POST _l1es/add-dml-index
{
  "index_name": "my-logs",
  "source": "message",
  "dest": "decoded_message"
}

Verify:

GET l1es_dml_indices/_search
# Should show your index mapping

Impact: Decoding stops for that index until re-registered. Recovery is immediate (< 1 second).

Rolling upgrade: Elasticsearch version bump with L1ES plugin

For 3-node Elasticsearch cluster upgrade (e.g., 8.16 → 8.17):

# Step 1: Disable shard allocation (prevents rebalancing during upgrade)
PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "none"
  }
}

# Step 2: Stop and upgrade node 1
# (On node 1)
curl -X POST "localhost:9200/_nodes/self/_shutdown"

# Upgrade ES binary
# (e.g., apt upgrade elasticsearch)

# Upgrade L1ES plugin
bin/elasticsearch-plugin remove l1es-plugin
bin/elasticsearch-plugin install file:///path/to/l1es-0.3.0.es.8.17.0.zip

# Start node 1
systemctl start elasticsearch

# Step 3: Wait for node 1 to rejoin cluster and recover shards
GET _cluster/health
# Watch until status=green, relocating_shards=0

# Step 4: Re-enable shard allocation
PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "all"
  }
}

# Step 5: Verify cluster is healthy
GET _cluster/health
# Should show: status=green, active_shards=<all shards>

# Repeat for nodes 2 and 3

CRITICAL: Plugin version must match Elasticsearch version

L1ES Version	Elasticsearch	Status
0.3.0	8.17.0	✓ Tested
0.3.0	8.16.x	⚠ Likely works, not officially tested
0.2.x	7.10.0	❌ Legacy

If plugin version mismatch: - Cluster will refuse to start or may behave unpredictably - Always verify plugin version matches ES version before starting node

Optimization

Does the 10x Engine help with hot/warm/cold tier optimization

Yes -- the 10x Engine dramatically reduces hot tier requirements:

Without the 10x Engine (100TB/day):

Hot tier: 100TB x 7 days = 700TB SSD storage
Warm tier: 700TB x 4 weeks = 2.8PB storage
Monthly hot tier cost: ~$35,000

With the 10x Engine (80% reduction):

Hot tier: 20TB x 7 days = 140TB SSD storage
Warm tier: 140TB x 4 weeks = 560TB storage
Monthly hot tier cost: ~$7,000

Bonus: Full raw data in S3 (~$2,300/month) for compliance and on-demand analysis. Total savings: 75%+.

How does volume reduction translate to fewer data nodes

For most self-managed clusters, data node count is driven by storage capacity. The relationship is roughly linear: 50% less indexed volume ≈ 50% fewer data nodes, because each decommissioned node removes both its EBS volume and its EC2 instance.

Example — 30 data nodes ingesting 10TB/day, 7-day hot retention, 1 replica:

	Before	Edge Optimizer (50%)	+ Storage Streamer (80%)
Daily indexed volume	10TB	5TB	2TB
Hot storage (7d × 1 replica)	140TB	70TB	28TB
Data nodes	30	15	6
Monthly data node cost	~$25,000	~$12,500	~$5,000
S3 archive (30d retention)	—	—	~$6,900
Monthly total	~$25,000	~$12,500	~$11,900

Master nodes (typically 3), coordinating nodes, and Kibana instances stay the same regardless of data volume.

The numbers above assume r6g.2xlarge data nodes with 5TB gp3 EBS each. Your instance types and storage sizes will differ, but the scaling relationship holds: half the indexed volume → half the hot-tier storage → half the data nodes.

Use the ROI Calculator with your actual cluster size for a personalized estimate.

How does the 10x Engine handle high-cardinality fields that cause mapping explosions

The 10x Engine uses automatic message enrichment to separate the stable, low-cardinality structure of log events from the high-cardinality variable data (timestamps, IDs, UUIDs) that causes mapping explosions.

Instead of indexing every dynamic field value, the 10x Engine identifies the symbol pattern of each event -- the stable message structure that remains constant across instances. This extracts consistent identifiers like Error_syncing_pod or Receive_ListRecommendations_for_product_ids regardless of the variable values embedded in them.

The rate regulator then uses these stable identifiers to track cost per event type and apply budget-based sampling -- targeting the noisiest patterns before they reach Elasticsearch, reducing both ingestion volume and the number of unique field values your mappings must handle.

What is the search-time overhead in Elasticsearch

The L1ES Lucene plugin expands compact events during search at ~1.25x search time. 50% less indexed volume offsets the expansion cost — fewer data nodes, less SSD, lower compute. For managed Elasticsearch (Elastic Cloud, OpenSearch Service), Storage Streamer expands and re-indexes on-demand.

The 10x Engine processes events at sub-millisecond per event — 100+ GB/day on a single node (512 MB heap, 2 threads). For resource requirements, scaling tables, and architecture details, see Performance FAQ.

Getting Started

How do I test this on my Elasticsearch cluster

Dev — Run on your Elasticsearch log files locally. One-line install, results in minutes. No account, no credit card.
Cloud Reporter — Connect to your Elasticsearch cluster via REST API. See which event types cost the most — no agent changes.
Edge apps — Deploy optimizer and regulator via Helm chart alongside your forwarder. ~30 min setup.
Storage Streamer — Route events to S3, stream selected data to Elasticsearch on-demand.

Each step is independent — start with Dev to see your reduction ratio, then move to production when ready.

Comparisons

How does this compare to Elastic ILM (Index Lifecycle Management)

ILM moves data to cheaper tiers after you've already paid to index every byte. The 10x Engine cuts what gets indexed -- fewer nodes, less compute, less hot-tier SSD from day one.

	Elastic ILM	10x Engine
When	After ingestion	Before ingestion
What	Move data to cheaper tiers	Reduce volume 50%+ before indexing
Ingestion cost	Full volume indexed	Only optimized volume indexed
Hot tier	Full volume on SSD	50%+ less SSD required

How does Log10x compare to Elastic's Logsdb index mode and searchable snapshots

Both kick in after ingestion -- you still pay full compute and node costs to index every byte. Neither reduces your node count.

	Logsdb	Searchable Snapshots	10x Engine
When	After indexing	After indexing	Before indexing
Reduces	Disk footprint (columnar encoding)	Lifecycle storage (S3 frozen tier)	Ingestion compute + storage + nodes
Ingestion cost	Unchanged	Unchanged	50%+ less
Node count	Unchanged	Unchanged	Fewer nodes needed