Logstash

Runs 10x Engine as a sidecar to Logstash for reporting, receiving, and optimizing events before they ship to their destination (Elasticsearch, OpenSearch, Splunk, S3, Kafka, …). Logstash and Log10x run as peer processes and exchange events as newline-delimited JSON over TCP — Logstash's built-in tcp input and output plugins with codec => json_lines on both legs. Works against any stock Logstash build (OSS or Elastic distribution) and the official elastic/logstash Helm chart with a values overlay.

Architecture

graph LR
    A["<div style='font-size: 14px;'>📂 Sources</div><div style='font-size: 10px;'>file, beats, http</div>"] --> F["<div style='font-size: 14px;'>🧪 ingest pipeline</div><div style='font-size: 10px;'>enrichment filters</div>"]
    F --> B["<div style='font-size: 14px;'>📤 tcp output</div><div style='font-size: 10px;'>json_lines :5044</div>"]
    B --> E["<div style='font-size: 14px;'>⚡ 10x Engine</div><div style='font-size: 10px;'>Report/Receive/Optimize</div>"]
    E --> C["<div style='font-size: 14px;'>📥 tcp input</div><div style='font-size: 10px;'>json_lines :5045</div>"]
    C --> D["<div style='font-size: 14px;'>📤 Destinations</div><div style='font-size: 10px;'>ES, Splunk, S3, Kafka</div>"]

    classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A input
    class B filter
    class C filter
    class D output
    class E engine
    class F filter

Data Flow

📂 Sources — Your existing Logstash inputs (file, beats, tcp, http, …) feed into the ingest pipeline.
🧪 ingest pipeline — Your enrichment filters (grok, mutate, kv, geoip, date, …) run here exactly once before the event is handed off to Log10x. The pipeline also stamps a tag field that survives the round trip and surfaces as the event's source inside Log10x.
📤 tcp output → Log10x — Logstash's tcp output plugin (codec => json_lines) ships the enriched event to the Log10x sidecar on TCP :5044 (or a Unix socket on Linux/macOS via the unix output plugin).
⚡ 10x Engine — The Receiver app applies rate/policy-based filtering and optionally compacts events for volume reduction.
📥 tcp input → destinations pipeline — Processed events come back to Logstash on :5045 and are picked up by a separate destinations pipeline whose output block holds your real destinations. Filters defined under ingest are not re-applied — Logstash's multi-pipeline routing keeps each pipeline's filter chain isolated.
📤 Destinations — The original tag field survives the round trip, so destinations that route on it (Splunk index, S3 prefix, Kafka topic, …) behave the same as if Log10x weren't in the path.

What an event looks like on the way back

The record structure of the original Logstash event is preserved end-to-end — every field comes back to your destinations pipeline with the same name and same position. What changes depends on the Receiver app mode:

Mode	Difference vs the event Logstash sent in
Receive (default)	None. Same record.
Receive + `symbolMessageHashField <name>`	Adds one new field with the symbol-pattern hash (a stable identifier for the message pattern — usable as a dedup key, metric dimension, or correlation ID).
`receiverOptimize true`	The value of the message field (`message` by default, or whatever `logstashInputMessageField` is set to) is replaced with a compact encoded form. A separate event with `tag` set to `tenx-template` is emitted with the template needed to decode it. All other fields stay verbatim.
`receiverOptimize true` + `symbolMessageHashField <name>`	Both of the above.

The tag field that the ingest pipeline stamps becomes the event's source inside Log10x — used for rate-based grouping and preserved on each event as it returns to Logstash. The message text is read from the message field by default (override with logstashInputMessageField). When the Receiver app is configured with k8sExtractorName: fluentK8s, the kubernetes.* sub-object is also lifted into pod/container metadata fields used by message-pattern and rate filtering.

Key Files

File	Purpose
`stream.yaml`	Logstash JSON socket input + output stream definitions
`conf/tenx-sidecar.conf`	Reference Logstash pipeline config showing the `ingest` / `destinations` split

Quickstart

1. Run Log10x:

tenx @run/input/forwarder/logstash @apps/receiver

2. Wire up your Logstash config — split your work into two pipelines via pipelines.yml, with an ingest leg that ends in tcp { codec => json_lines } and a destinations leg that starts with tcp { codec => json_lines }:

pipelines.yml

- pipeline.id: ingest
  path.config: "${TENX_MODULES}/pipelines/run/modules/input/forwarder/logstash/conf/tenx-sidecar.conf"
- pipeline.id: destinations
  path.config: "/etc/logstash/conf.d/destinations.conf"

3. Point the destinations pipeline at your real outputs (the recipe defaults to stdout for testing):

destinations.conf

input {
  tcp {
    host  => "0.0.0.0"
    port  => 5045
    codec => json_lines
  }
}
output {
  # ... your real destinations ...
}

For Kubernetes deployment via the official elastic/logstash Helm chart see the Helm sidecar overlay.

Config Files

To configure the Logstash module, Edit these files.

Below is the default configuration from: logstash/config.yaml.

Edit Online

# 🔟❎ 'run' Logstash receiver configuration
#
# To learn more see https://doc.log10x.com/run/input/forwarder/logstash/

tenx: run

# =============================== Dependencies ================================

include:
  - run/input/forwarder/config.yaml
  - run/modules/input/forwarder/logstash

# ============================== Logstash Options =============================

logstash:

  # ----------------------------- Input Options -----------------------------

  input:

    # 'port' specifies the TCP port to listen on for events from Logstash's
    #  `tcp` output with `codec => json_lines`.
    port: 5044

    # 'path' switches the input to a Unix domain socket instead of TCP.
    #  When set, takes precedence over 'port'.
    # path: /tmp/tenx-logstash-in.sock

    # 'messageField' is the name of the JSON field in each Logstash event
    #  carrying the log line text.
    messageField: message

  # ----------------------------- Output Options ----------------------------

  output:

    # 'host' specifies the TCP host of Logstash's `tcp` input receiving
    #  processed events from Log10x.
    host: 127.0.0.1

    # 'port' specifies the TCP port of Logstash's `tcp` input. MUST differ
    #  from input.port or the two would collide on the same listener.
    port: 5045

    # 'encodeType' is the on-wire format for the socket record.
    #  - 'delimited': each top-level field of the rendered record becomes its
    #    own emitted field — preserves the original record's structure.
    #  - 'json': the whole record is wrapped as one stringified field.
    encodeType: delimited

Options

Specify the options below to configure the Logstash:

Name	Description	Category
logstashInputPort	TCP port to listen on for events from Logstash's tcp output	Input
logstashInputPath	Unix domain socket path to listen on (Linux/macOS)	Input
logstashInputMessageField	Name of the JSON field carrying the original log line	Input
logstashOutputHost	TCP host of the Logstash tcp input receiving processed events	Output
logstashOutputPort	TCP port of the Logstash tcp input receiving processed events	Output
logstashOutputEncodeType	Output format when outputFields are set. Possible values: [json, delimited]	Output

Input

`logstashInputPort`

TCP port to listen on for events from Logstash's tcp output.

Type	Default	Category
String	5044	Input

TCP port where Log10x listens for events sent by Logstash's tcp output with codec => json_lines. Match this against the port in your Logstash output { tcp { ... } } block.

`logstashInputPath`

Unix domain socket path to listen on (Linux/macOS).

Type	Default	Category
String	""	Input

When set, Log10x listens on a Unix domain socket instead of TCP — useful on Linux/macOS to avoid TCP overhead. Configure Logstash's unix output plugin with codec => json_lines and a matching path. Ignored on Windows.

`logstashInputMessageField`

Name of the JSON field carrying the original log line.

Type	Default	Category
String	message	Input

Name of the field in each Logstash event that contains the actual log message text. The default (message) matches Logstash's convention for the file, beats, tcp, and http inputs. Set to e.g. log if your sources emit the log line under a different key.

Output

`logstashOutputHost`

TCP host of the Logstash tcp input receiving processed events.

Type	Default	Category
String	127.0.0.1	Output

Hostname or IP where the Logstash tcp input is listening for processed events from Log10x. Pairs with logstashOutputPort.

`logstashOutputPort`

TCP port of the Logstash tcp input receiving processed events.

Type	Default	Category
String	5045	Output

TCP port where the Logstash tcp input is listening for processed events from Log10x. MUST differ from the port Log10x's own input listens on (default 5044) or the two would collide on the same socket.

`logstashOutputEncodeType`

Output format when outputFields are set. Possible values: [json, delimited].

Type	Default	Category
String	delimited	Output

Specifies how the combined output (main event field plus outputFields) is encoded when writing back to Logstash. Possible values:

json: formats all fields as a JSON object
delimited: formats field values separated by the output delimiter Only takes effect when logstashOutputFields is set.

This module is defined in logstash/module.yaml.