Skip to content

Compact

Cut log analytics and storage costs by over 50% by losslessly compacting log/trace events at the edge before shipping them to output. Compact mode is one of the Reducer's two execution modes (the other is Filter). Events ship as a compact wire-form that downstream SIEM plugins or Retriever expand at query time.

Edge architecture — forwarders route events through the 10x sidecar before reaching your analyzer

10x Sidecar — Reporter, Reducer, and Compact mode savings per app

Per-pattern volume reduction — Compact mode compacted 44.6 TB to 31.5 TB, saving $79K with zero data loss

Cut Storage Costs

Reduce storage costs by 50-75% by forwarding compact events to object storage (e.g., AWS S3, Azure Blobs). Expand events on-the-fly using the Retriever app to forward raw data to log analytics and metric outputs periodically or on-demand.

Cut Log Analytics Costs

Reduce ingestion and licensing costs by over 50% by forwarding compact events to Splunk and Elasticsearch. The open-source 10x for Splunk app and L1ES Elasticsearch plugin expand events in real-time, maintaining full querying, dashboard, and alerting capabilities without data loss.

Workflow

Compact mode processes events from a variety of log forwarders, such as Fluentd, Fluent Bit, Filebeat, and Logstash. Configure the app to process all or a subset of the events, allowing for targeted analysis, regulation, and volume reduction.

graph LR
    A["<div style='font-size: 14px;'>🚙 Forwarder</div><div style='font-size: 10px; text-align: center;'>Sidecar Process</div>"] --> B["<div style='font-size: 14px;'>📡 Receive</div><div style='font-size: 10px; text-align: center;'>Stream Events</div>"]
    B --> C["<div style='font-size: 14px;'>🔄 Transform</div><div style='font-size: 10px; text-align: center;'>into TenXObjects</div>"]
    C --> D["<div style='font-size: 14px;'>🎁 Enrich</div><div style='font-size: 10px; text-align: center;'>Add Context</div>"]
    D --> E["<div style='font-size: 14px;'>🗜️ Compact</div><div style='font-size: 10px; text-align: center;'>Encode Events</div>"]
    E --> F["<div style='font-size: 14px;'>📤 Output</div><div style='font-size: 10px; text-align: center;'>Return to Forwarder</div>"]

    classDef deploy fill:#7c3aed88,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef receive fill:#9333ea88,stroke:#7c3aed,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef transform fill:#2563eb88,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef enrich fill:#059669,stroke:#047857,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef compact fill:#f59e0b,stroke:#d97706,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#ea580c88,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A deploy
    class B receive
    class C transform
    class D enrich
    class E compact
    class F output

🚙 Forwarder: Runs 10x as a sidecar process to log forwarders for real-time event analysis

📡 Receive: Read events continuously from log forwarders via IPC

🔄 Transform: Structure raw events into well-defined TenXObjects

🎁 Enrich: Apply enrichment rules to augment TenXObjects with intelligent context

🚦 Regulate: Filter events via per-node budget sampling or a declarative field-set mute file to prevent over-billing

🗜️ Compact: Losslessly compact events using templates the runtime engine builds from AOT compiler symbols

📤 Output: Return compact events to forwarder to ship to destination analyzers or storage

Architecture

Compact mode executes inside the Reducer sidecar, compacting events before they ship to a log analyzer or object storage.

Forwarders ship verbose events containing repetitive elements such as JSON/KV field names, messages, severity levels and app-specific low-cardinality values, increasing cost by over 50%.

Architecture diagram: Log forwarders ship verbose events with repetitive JSON field names and low-cardinality values to log analyzers, increasing costs by over 50%
❌ Forwarders ship costly duplicative events to log analyzers

Forward compact events to low-cost storage (e.g., AWS S3, Azure Blobs). The Retriever app expands and streams selected events to log analyzers and metric outputs.

Architecture diagram: Compact mode losslessly compacts events before uploading to S3 or Azure Blob storage, where Retriever expands and streams them on demand
✅ Compact mode losslessly compacts events before they upload to storage

Forward compact events to Splunk. The open-source 10x for Splunk app expands events on-the-fly at search time, displaying them in full JSON/text form in dashboards and queries.

Architecture diagram: Compact mode losslessly compacts events before shipping to Splunk, where the 10x for Splunk app expands them at search time
✅ Compact mode losslessly compacts events before they ship to Splunk

Forward compact events to Elasticsearch or OpenSearch. The open-source L1ES plugin transparently rewrites standard queries and decodes _source at search time — Kibana dashboards, saved searches, and alerts work unchanged.

Architecture diagram: Compact mode losslessly compacts events before shipping to Elasticsearch or OpenSearch, where the L1ES plugin expands them at search time
✅ Compact mode losslessly compacts events before they ship to ElasticSearch

Safety & Reliability

Compact mode runs inside the Reducer sidecar with fail-open design — if the reducer crashes or stops, your logs continue flowing normally at full volume to your analyzer.

Topic Detail
Fail-open design Logs continue flowing if 10x goes down
Backpressure handling Disk buffering prevents data loss during spikes
Resource requirements 512MB heap + 2 threads handles 100+ GB/day
Rollback helm uninstall takes ~1 minute, no data loss

See the Reducer FAQ for complete operational details, capacity planning, and deployment guidance.

Next: Splunk