Policy

The policy module queries a Prometheus endpoint (e.g., 10x SaaS/self-hosted) for event types exhibiting the highest volume rates, storing the results in a lookup file locally or on Github.

Instances of the Edge regulator app running alongside log forwarders can fetch and use the lookup to filter 'noisy' events from shipping to log analyzers (e.g., Splunk, Elastic).

Workflow

The policy regulator executes the following periodic process:

graph LR
    A["<div style='font-size: 14px;'>⏰ Scheduler</div><div style='font-size: 10px; text-align: center;'>Periodic Job</div>"] --> B["<div style='font-size: 14px;'>📊 Query</div><div style='font-size: 10px; text-align: center;'>Prometheus</div>"]
    B --> C["<div style='font-size: 14px;'>🔧 Build Policy</div><div style='font-size: 10px; text-align: center;'>Lookup Table</div>"]
    C --> D["<div style='font-size: 14px;'>📤 Push</div><div style='font-size: 10px; text-align: center;'>to GitHub</div>"]
    D --> E["<div style='font-size: 14px;'>📥 Pull</div><div style='font-size: 10px; text-align: center;'>Edge Regulators</div>"]
    E --> F["<div style='font-size: 14px;'>🚫 Filter</div><div style='font-size: 10px; text-align: center;'>Collected Events</div>"]

    classDef schedule fill:#7c3aed88,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef query fill:#9333ea88,stroke:#7c3aed,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef filter fill:#2563eb88,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef Push fill:#059669,stroke:#047857,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef pull fill:#dc2626,stroke:#b91c1c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef regulate fill:#ea580c88,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A schedule
    class B query
    class C filter
    class D Push
    class E pull
    class F regulate

⏰ Periodic Invocation: Scheduled job triggers policy generation at configured intervals

📊 Prometheus Query: Retrieves cost metrics using range queries for event pattern analysis

🔧 Policy Building: Creates lookup table from high-cost event patterns exceeding thresholds

📤 GitHub Push: Commits policy lookup file to centralized repository for distribution (optional)

📥 Edge Regulator Pull: Distributed regulators fetch updated policies from central config

🚫 Event Filtering: Downstream edge regulators apply policies to regulate events before shipping

Output Lookup

The policy lookup file can either be pushed to GitHub for GitOps distribution or stored locally for custom distribution mechanisms.

GitHub Output Local Output

The module uses a GitHub output to push the calculated event rates as lookup file containing the identities of the highest-rate event patterns for use by downstream Edge regulators.

The GitHub approach enables centralized GitOps workflows where edge regulators automatically pull policy updates from the repository, ensuring consistent regulation across distributed deployments.

The module can emit the policy lookup table to a local file on disk, allowing users to distribute it using their preferred mechanism to downstream edge regulators.

Distribution Examples:

• Kubernetes ConfigMaps: Mount the lookup file as a ConfigMap volume across edge regulator pods
• S3 Distribution: Upload the file to S3 and configure edge regulators to periodically download updates
• Network File Systems: Place the file on shared storage accessible by all edge nodes
• Container Images: Bundle the lookup file into custom container images for immutable deployments

This approach provides flexibility for organizations with existing distribution infrastructure or specific compliance requirements that prevent external repository usage.

Configuration

To configure the Policy regulator lookup input module, Edit these settings.

Below is the default configuration from: policy/config.yaml.

Edit Online Edit Locally

Edit config.yaml Locally

Download and save to:

• Linux / Docker / macOS
• Custom directory
  $TENX_CONFIG/run/regulate/policy/config.yaml
• Within cloned repo
  ./pipelines/run/regulate/policy/config.yaml

Download

# 🔟❎ 'run' policy generator configuration

# Policy generators periodically query a instance Prometheus to generate a filter lookup file on GitHub
# used by Edge regulators to filter 'noisy' telemetry from shipping to outputs (e.g., Splunk, Elastic).

# To learn more see https://doc.log10x.com/run/regulate/policy

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include:
  - run/modules/regulate/policy

# =========================== Prometheus Policy Input =========================

policy:

  - prometheus:

      # 'endpoint' defines the address of the Prometheus HTTP API.
      endpoint: https://prometheus.log10x.com

      # 'series' defines the name of the series to query for events volume in bytes
      series: all_events_summaryBytes_total

      # 'labels' specifies the list of metric labels to group by in the Prometheus 'avg_over_time' query
      labels:
        - message_pattern

      # 'start' sets the query's beginning timestamp for Prometheus data retrieval
      start: $=now() / 1000

      # 'rangeInterval' defines the time range interval for the Prometheus 'avg_over_time' query
      rangeInterval: 24h

      # 'policyStepDuration' defines the resolution step duration for the Prometheus 'avg_over_time' query
      stepDuration: 5m

      # 'topEvents' specifies the number of highest-rate event patterns to return using Prometheus topk operator
      topEvents: 50

    ingestionCostPerGB: 1.5

    output:

      path: $=path("data/sample/policy") + "/policy.csv"

      github:
        # 'repo' specifies the GitHub repo to push output lookup file to
        repo: ""

        # 'branch' specifies the GitHub repo branch to push the file to, defaults to main
        branch: ""

Options

Specify the options below to configure multiple Policy regulator lookup input:

Name	Description	Category
policyPrometheusEndpoint	Prometheus instance endpoint	Prometheus
policyPrometheusSeries	Prometheus series to query	Prometheus
policyPrometheusLabels	Prometheus metric labels for grouping	Prometheus
policyPrometheusApps	Prometheus metric app names	Prometheus
policyPrometheusStart	Query start time	Prometheus
policyPrometheusRangeInterval	Query time range interval	Prometheus
policyPrometheusStepDuration	Query resolution step duration	Prometheus
policyPrometheusTopEvents	Number of top events to return from Prometheus query	Prometheus
policyIngestionCostPerGB	Vendor ingestion cost per GB in USD for cost calculation	Cost
policyOutputPath	Output path for the generated policy lookup file	Output
policyOutputGithubRepo	GitHub repository to push the policy lookup file	Output
policyOutputGithubBranch	GitHub branch for pushing the policy lookup file	Output
policyOutputGithubSquashMerge	Use squash merge when merging GitHub pull requests	Output

Prometheus

policyPrometheusEndpoint

Prometheus instance endpoint.

Type	Default	Category
String	https://prometheus.log10x.com	Prometheus

Defines the address of the Prometheus HTTP API to query.

policyPrometheusSeries

Prometheus series to query.

Type	Default	Category
String	all_events_summaryBytes_total	Prometheus

Defines the name of the series to query for events volume in bytes.

policyPrometheusLabels

Prometheus metric labels for grouping.

Type	Required	Category
List	✔	Prometheus

Specifies the list of metric labels to group by in the Prometheus query.

NOTE: this list must match the rateRegulatorFieldNames specified by the edge rate regulator.

policyPrometheusApps

Prometheus metric app names.

Type	Default	Category
List	reporter|regulator|optimizer	Prometheus

Specifies the list of 10x edge app names to specify in the Prometheus query.

policyPrometheusStart

Query start time.

Type	Default	Category
String	1772387753	Prometheus

Sets query's beginning timestamp for Prometheus data retrieval (e.g., 2025-10-13T23:29:23Z, 1736724563).

policyPrometheusRangeInterval

Query time range interval.

Type	Required	Category
String	✔	Prometheus

Defines the time range interval for the Prometheus range query (e.g., '5m', '1h').

policyPrometheusStepDuration

Query resolution step duration.

Type	Required	Category
String	✔	Prometheus

Defines the query resolution step duration for the Prometheus range query (e.g., '30s', '1m').

policyPrometheusTopEvents

Number of top events to return from Prometheus query.

Type	Default	Category
Number	50	Prometheus

Specifies the number of highest-rate event patterns to return using the Prometheus topk operator.

Cost

policyIngestionCostPerGB

Vendor ingestion cost per GB in USD for cost calculation.

Type	Default	Category
Number	1.5	Cost

Defines the cost per GB charged by your observability vendor for log ingestion. Used to calculate cost-per-hour data for each event pattern in the generated lookup file.

Edge rate regulators in global mode use this cost data (from the lookup) to enforce cluster-wide budget policies.

Important: This value should matchrateRegulatorIngestionCostPerGB configured on your edge rate regulators.

Common vendor pricing (2025):

• Splunk Cloud: ~$1.50/GB (varies by contract, SKU)
• Datadog Logs: ~$0.10-$0.25/GB (depends on tier: standard, flex, online archives)
• Elastic Cloud: ~$0.109/GB (standard logging tier)
• New Relic: ~$0.30/GB (Data Plus)
• Sumo Logic: ~$1.50/GB (depends on plan)
• AWS CloudWatch Logs: ~$0.50/GB ingestion + $0.03/GB storage

Example: If your Splunk contract is $1.50/GB and you emit 100GB/hour across all apps, the total cost is $150/hour. The policy module calculates per-event-pattern costs (e.g., "error_login|frontend" = $20/hour) and writes them to the lookup. Edge regulators then enforce per-app or per-pattern caps based on this data.

Output

policyOutputPath

Output path for the generated policy lookup file.

Type	Required	Category
String	✔	Output

Defines the output path for the policy lookup file containing cluster-wide spend data (per field set and total) averaged over the configured time range.

Behavior depends on policyOutputGithubRepo:

• If policyOutputGithubRepo is NOT set: Local file path where the lookup will be written (e.g., /tmp/rate-policy.csv, ./policies/rate-lookup.csv)
• If policyOutputGithubRepo IS set: Path within the GitHub repository where the lookup will be pushed (e.g., policies/rate-lookup.csv, rate-policy.csv)

Edge rate regulators in global mode pull this file to enforce cluster-wide budgets.

policyOutputGithubRepo

GitHub repository to push the policy lookup file.

Type	Default	Category
String	""	Output

Defines the GitHub repository (format: owner/repo) where the generated policy lookup file will be pushed. Used for sharing the lookup with distributed edge rate regulators.

Requires GitHub authentication (via token or SSH key).

Example: my-org/observability-policies.

policyOutputGithubBranch

GitHub branch for pushing the policy lookup file.

Type	Default	Category
String	main	Output

Defines the Git branch to use when pushing the policy lookup file to GitHub.

Example: main, production, or policies.

policyOutputGithubSquashMerge

Use squash merge when merging GitHub pull requests.

Type	Default	Category
Boolean	true	Output

Specifies whether to use squash merge (true) or regular merge (false) when merging pull requests to GitHub. Squash merge combines all commits into a single commit, resulting in a cleaner commit history. This is particularly useful for automated policy updates that run frequently (e.g., every 5 minutes).

Recommendation: Enable this for production environments with frequent policy updates.

This module is defined in policy/module.yaml.