Skip to content

Regulate

Read events from a Logstash forwarder to transform them into typed TenXObjects to filter using local/centralized regulator policy. This module is a component of the Edge Regulator app.

Architecture

graph LR
    A["<div style='font-size: 14px;'>📂 Logstash</div><div style='font-size: 10px;'>inputs</div>"] --> B["<div style='font-size: 14px;'>🔧 Pipe Output</div><div style='font-size: 10px;'>plugin</div>"]
    B --> C["<div style='font-size: 14px;'>⚡ 10x Regulator</div><div style='font-size: 10px;'>filter policy</div>"]
    C --> D["<div style='font-size: 14px;'>🔌 Unix/TCP</div><div style='font-size: 10px;'>input</div>"]
    D --> E["<div style='font-size: 14px;'>📤 Logstash</div><div style='font-size: 10px;'>outputs</div>"]

    classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef socket fill:#0891b2,stroke:#0e7490,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A input
    class B filter
    class C engine
    class D socket
    class E output

Data Flow

  • 📂 Logstash Inputs - Collect logs from files, beats, TCP, or other sources
  • 🔧 Pipe Output Plugin - Pipes ALL events to 10x sidecar via stdin
  • ⚡ 10x Regulator - Applies rate/policy-based filtering, drops noisy events
  • 🔌 Unix/TCP Input - Receives FILTERED events back from the sidecar
  • 📤 Logstash Outputs - Only filtered events ship to final destinations

Key Characteristics

Feature Description
🚦 Rate Limiting Filter events based on per-template rate limits
📋 Policy-Based Apply local or centralized filtering policies
💰 Cost Control Reduce log volume and costs by dropping noisy events
🔧 Pipe Output Uses Logstash's pipe output plugin for stdin piping

Sidecar Relay

This module configures a Logstash pipe output plugin and Unix/TCP input plugin. The Logstash output plugin launches a 10x sidecar process and pipes events to it to regulate using a local/centralized policy. The sidecar relays regulated events back to Logstash Unix/TCP input plugin to ship to output (e.g., ElasticSearch).

Install

See the Log10x Edge Regulator Logstash run instructions

Currently not supported

Configuration

To configure the Logstash regulator module, Edit these settings.

Below is the default configuration from: regulate/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' Logstash regulator configuration

# Configure an input that reads events from a Logstash forwarder and
# an output that writes filtered TenXObjects back to Logstash.

# This config operates in conjunction with the instructions provided in:
# https://doc.log10x.com/run/input/forwarder/logstash/regulate/#install

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include:

   # Include output encoding format options
  - run/modules/input/forwarder/logstash/output/module.yaml

    # Include the stdin input to read events from the forwarder
  - run/modules/input/forwarder/logstash/input

    # Include Unix domain/TCP socket output to write events to the forwarder
  - '$="run/modules/input/forwarder/logstash/output/" + (TenXString.includes(TenXEnv.get("os.name"), "Windows") ? "tcp" : "unix")'

# ============================== Logstash Options ==============================

logstash:
  # 'encodeObjects' specifies the target output (Unix domain/TCP) will write
  #  TenXObjects in their original form via the intrinsic 'fullText' field.
  encodeObjects: false

Below is the default configuration from: regulate/tenx-pipe-in-unix.conf.

Edit Online

Edit tenx-pipe-in-unix.conf Locally 

# 10x Logstash Unix domain socket input configuration
#
# Add this Logstash pipeline configuration file to 'pipelines.yml' to 
# read events from a 10x subprocess via a Unix domain socket.
#
# This file is used in conjunction with:
# run/config/forwarder/logstash/regulate/tenx-pipe-out.conf
#

# Read back data from 10x via a Unix domain socket.
# The 10x 'run' pipeline captures the socket address using 'tenx_marker_pipeline_unix_address' in '/run/modules/input/forwarder/logstash/output/unix/stream.yaml'
input {
  unix {
    id => "tenx_unix_input"
    path => "${tenx_marker_pipeline_unix_address:/tmp/tenx_logstash.sock}"
    force_unlink => true
    tags => ["tenx"]
    codec => json_lines
  }
}

# Process incoming events; templates get sent to Elastic, and everything else gets
# processed by the rest of the Logstash pipeline.
#
output {
   pipeline {
    send_to => user_output_pipeline
  }
}

Below is the default configuration from: regulate/tenx-pipe-in-tcp.conf.

Edit Online

Edit tenx-pipe-in-tcp.conf Locally 

# 10x Logstash TCP socket input configuration
#
# Add this Logstash configuration to 'pipelines.yml' to launch
# an 10x subprocess from which to read events via a TCP socket.
#
# Add this file as a pipeline for your Logstash pipelines.yml config to incorporate
# it into your configuration.
#
# This file is used in conjunction with:
# run/config/forwarder/logstash/regulate/tenx-pipe-out.conf
#

# Read back data from 10x via a TCP socket.
# Port 5160 is specified as a default, as Logstash does not define a default TCP port.
# To learn more see https://www.elastic.co/guide/en/logstash/current/plugins-inputs-tcp.html 
# The port value is captured by the 10x 'run' pipeline using the 'tenx_marker_pipeline_tcp_port' marker in '/run/modules/input/forwarder/logstash/output/tcp/stream.yaml'
input {
  tcp {
    id => "tenx_tcp_input"
    port => "${tenx_marker_pipeline_tcp_port:5160}"
    tags => ["tenx"]
    codec => json_lines
  }
}

# Process incoming events; templates get sent to Elastic, and everything else gets
# processed by the rest of the Logstash pipeline.
#
output {
  pipeline {
    send_to => user_output_pipeline
  }
}

Below is the default configuration from: regulate/tenx-pipe-out.conf.

Edit Online

Edit tenx-pipe-out.conf Locally 

# 10x Logstash pipe output configuration
#
# Add this Logstash configuration file  to 'pipelines.yml' to write
# events into an 10x subprocess.
#
# This file is used in conjunction with a Logstash input '.conf' file
# used to read events back from the 10x subprocess:
#   unix: run/config/forwarder/logstash/regulate/tenx-pipe-in-unix.conf
#   win:  run/config/forwarder/logstash/regulate/tenx-pipe-in-tcp.conf


input {
  # Getting data into this Logstash pipeline.
  # When using the pipelines.yml approach, this sets Logstash's internal address
  # for data input into this pipeline.
  #
  # Alternatively, any other Logstash input plugin may be used when incorporating
  # 10x into your Logstash config
  #
  pipeline {
    address => tenx_input_pipeline
  }
}

output {
  # Send incoming events to the 10x executable via the 'pipe' output plugin.
  #
  pipe {
    id => "tenx_pipeline"

    # Don't close the 10x process due to events inactivity.
    ttl => -1

    # NIX
    command => "${TENX_BIN:/opt/tenx-edge/bin/tenx} run @run/input/forwarder/logstash/regulate/config.yaml @run/apps/edge/regulator"

    # WIN
    #command => "${TENX_BIN:'c:\program files\tenx\tenx.exe'} run @run/input/forwarder/logstash/regulate/config.yaml @run/apps/edge/regulator"
  }
}


This module is defined in regulate/module.yaml.