Skip to content

Vector Reducer

Read events from Vector and filter them using local/centralized reducer policy. This module is a component of the Reducer app.

Architecture

graph LR
    A["📂 Vector sources<br/>file, kubernetes_logs, …"] --> B["📤 socket sink<br/>Unix / text"]
    B --> C["🔌 Unix Socket"]
    C --> D["⚡ 10x Reducer<br/>filter policy"]
    D --> E["🔌 Unix Socket"]
    E --> F["📥 fluent source<br/>Forward / Unix"]
    F --> G["📤 Final Sinks<br/>ES, Splunk, S3"]

Data Flow

  • 📂 Vector sources — Collect logs from files, Kubernetes, journald, OTLP, etc.
  • 📤 Socket Sink — Sends ALL logs to Log10x via Unix socket (newline-delimited text or JSON).
  • ⚡ 10x Reducer — Applies rate/policy-based filtering, drops noisy events.
  • 🔌 Forward Output — Returns FILTERED events via Fluent Forward protocol.
  • 📥 Fluent Source — Vector receives filtered events.
  • 📤 Final Sinks — Only filtered events ship to final destinations.

Sidecar Relay

This module configures a Unix socket input/output pair that receives events from Vector, applies regulation policy, and returns filtered events via the Fluent Forward protocol. The sidecar relays regulated events back to Vector to ship to outputs (e.g., Splunk, S3).

Install

See the Log10x Reducer Vector run instructions

Deploy to k8s via Helm (see the Vector Helm chart fork with the tenx.* block).

Config Files

To configure the Vector reducer module, Edit these files.

Below is the default configuration from: regulate/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' Vector reducer configuration
#
# Configure an input that reads events from Vector via a Unix socket and an
# output that writes regulated TenXObjects back via the Fluent Forward
# protocol. When 'reducerOptimize' is enabled, events are encoded for
# volume reduction.
#
# Architecture:
#   Vector --[socket/unix]--> Log10x Reducer --[fluent/unix]--> Vector --> Final destinations
#
# Usage:
#   tenx run @run/input/forwarder/vector/regulate @apps/reducer

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include:

  # Parent module declaring shared forwarder options (outputEncodeField, outputWriteTemplates, encodeObjects)
  - run/modules/input/forwarder/module.yaml

  - run/modules/input/forwarder/vector/regulate/module.yaml

  # Read events from Vector via Unix socket (newline-delimited records)
  - run/modules/input/forwarder/vector/input

  # Write filtered events back to Vector via Fluent Forward protocol
  - run/modules/input/forwarder/vector/output/unix

# =============================== Mode Options ================================

# Mirrors the otel-collector / fluentbit dispatch — see fluentbit/regulate/config.yaml
# for the full explanation.

outputEncodeField: '$=yield TenXEnv.get("compactReducerLookupFile") ? "encoded=shouldEncode() ? encode() : fullText" : (TenXEnv.get("reducerOptimize") ? "encoded=encode()" : "fullText")'

outputWriteTemplates: $=yield TenXEnv.get("reducerOptimize", false) || TenXEnv.get("compactReducerLookupFile", false)

encodeObjects: $=yield TenXEnv.get("reducerOptimize", false) || TenXEnv.get("compactReducerLookupFile", false)

# ========================= Configuration Options =============================

vector:

  # 'outputFields' specifies TenXObject field names to emit alongside the main event.
  #  When set, enrichment fields are included in the sidecar output for forwarder post-processing.
  #  Example: outputFields: [level, group, symbolMessage]
  outputFields: []

  # 'outputEncodeType' controls the output format when outputFields is set.
  #  Possible values: 'json' or 'delimited'
  outputEncodeType: delimited

Options

Specify the options below to configure the Vector reducer:

Name Description
vectorOutputFields List of TenXObject field names to include alongside the main event in output
vectorOutputEncodeType Output format when outputFields are set. Possible values: [json, delimited]

vectorOutputFields

List of TenXObject field names to include alongside the main event in output.

Type Default
List []

Specifies TenXObject field names to emit alongside the main event text when writing back to the Vector forwarder. When empty (default), only the main event field is written. Example fields: level, group, symbolMessage.

vectorOutputEncodeType

Output format when outputFields are set. Possible values: [json, delimited].

Type Default
String delimited

Specifies how the combined output (main event field plus outputFields) is encoded when writing back to the Vector forwarder. Possible values:

  • json: formats all fields as a JSON object
  • delimited: formats field values separated by the output delimiter Only takes effect when vectorOutputFields is set.


This module is defined in regulate/module.yaml.