Skip to content

Edge Optimizer

Cut log analytics and storage costs by over 50% by optimizing log/trace events at the edge before shipping them to output.

Edge Optimizer Console

Cut Storage Costs

Reduce storage costs by 50-75% by forwarding compact events to object storage (e.g., AWS S3, Azure Blobs). Expand events on-the-fly using the Storage Streamer app to forward raw data to log analytics and metric outputs periodically or on-demand.

Cut Log Analytics Costs

Reduce ingestion and licensing costs by over 50% by forwarding compact events to Splunk and Elasticsearch. The open-source 10x for Splunk app and L1ES Elasticsearch plugin expand events in real-time, maintaining full querying, dashboard, and alerting capabilities without data loss.

Workflow

The Edge Optimizer app processes events from a variety of log forwarders, such as Fluentd, Fluent Bit, Filebeat, and Logstash. Configure the app to process all or a subset of the events, allowing for targeted analysis, regulation, and optimization.

graph LR
    A["<div style='font-size: 14px;'>🚙 Forwarder</div><div style='font-size: 10px; text-align: center;'>Sidecar Process</div>"] --> B["<div style='font-size: 14px;'>📡 Receive</div><div style='font-size: 10px; text-align: center;'>Stream Events</div>"]
    B --> C["<div style='font-size: 14px;'>🔄 Transform</div><div style='font-size: 10px; text-align: center;'>into TenXObjects</div>"]
    C --> D["<div style='font-size: 14px;'>🎁 Enrich</div><div style='font-size: 10px; text-align: center;'>Add Context</div>"]
    D --> E["<div style='font-size: 14px;'>🗜️ Optimize</div><div style='font-size: 10px; text-align: center;'>Encode Events</div>"]
    E --> F["<div style='font-size: 14px;'>📤 Output</div><div style='font-size: 10px; text-align: center;'>Return to Forwarder</div>"]

    classDef deploy fill:#7c3aed88,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef receive fill:#9333ea88,stroke:#7c3aed,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef transform fill:#2563eb88,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef enrich fill:#059669,stroke:#047857,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef optimize fill:#f59e0b,stroke:#d97706,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#ea580c88,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A deploy
    class B receive
    class C transform
    class D enrich
    class E optimize
    class F output

🚙 Forwarder: Runs 10x as a sidecar process to log forwarders for real-time event analysis

📡 Receive: Read events continuously from log forwarders via IPC

🔄 Transform: Structure raw events into well-defined TenXObjects

🎁 Enrich: Applies enrichment rules to augment TenXObjects with intelligent context

📈 Report: Publishes cost insight metrics for visualization and alerting

🚦 Regulate: Filters events using local or environment policies to prevent over-billing

🗜️ Optimize: Losslessly compacts events using templates the runtime engine builds from AOT compiler symbols

📤 Output: Returns optimized events to forwarder to ship to destination analyzers or storage

Architecture

Edge optimizers execute as a forwarder sidecar process to optimize events before they ship to log analyzer or Object Storage.

Forwarders ship verbose events containing repetitive elements such as JSON/KV field names, messages, severity levels and app-specific low-cardinality values, increasing cost of by over 50%.

Image title
❌ Forwarders ship costly duplicative events to log analyzers

Forward optimized events to low-cost storage (e.g., AWS S3, Azure Blobs). The Storage Streamer app expands and streams selected events to log analyzers and metric outputs.

Image title
✅ Edge Optimizers losslessly compact events before they upload to storage

Forward optimized events to Splunk. The open-source 10x for Splunk app expands events on-the-fly at search time, displaying them in full JSON/text form in dashboards and queries.

Image title
✅ Edge Optimizers losslessly compact events before they ship to Splunk

Forward optimized events to Elasticsearch or OpenSearch. The open-source L1ES plugin transparently rewrites standard queries and decodes _source at search time — Kibana dashboards, saved searches, and alerts work unchanged.

Image title
✅ Edge Optimizers losslessly compact events before they ship to ElasticSearch

Safety & Reliability

The Edge Optimizer runs as a sidecar alongside your log forwarder with fail-open design — if the optimizer crashes or stops, your logs continue flowing normally to your analyzer.

Key topics:

See the Edge FAQ for complete operational details, capacity planning, and deployment guidance.

Next: Run