Skip to content

Slack Bot

The Log10x Slack Bot brings cost attribution into your team's workflow. Query per-pattern log costs, identify cost drivers, and look up specific log events — all from a /log10x slash command.

Install

1. Create a Slack App

  1. Go to api.slack.com/apps and click Create New App > From scratch
  2. Name it Log10x and select your workspace
  3. Under Basic Information > Display Information, upload the bot icon (512x512 PNG) and set the background color to #0d1117
  4. Under Slash Commands, click Create New Command:
    • Command: /log10x
    • Request URL: your Lambda Function URL or API Gateway endpoint
    • Short Description: Log cost attribution
    • Usage Hint: [top | <service> | <label>=<value> | event <text>]
  5. Under OAuth & Permissions, install the app to your workspace
  6. Copy the Signing Secret from Basic Information

2. Configure Environment Variables

The bot runs as two AWS Lambdas: a thin Python entry point that acknowledges Slack within 3 seconds, and a Java worker that queries Prometheus and posts the result. Set these environment variables on the respective Lambda:

Variable Lambda Required Description
SLACK_SIGNING_SECRET Ack Yes Slack app signing secret (from Basic Information)
SLACK_WORKER_FUNCTION Ack Yes Name of the worker Lambda function
TENX_SLACK_TENANT Worker Yes Your Log10x environment ID
TENX_SLACK_API_KEY Worker Yes Your Log10x API key (used for cost settings and AI enrichment)
TENX_SLACK_ENV_ID Worker No Environment ID for AI enrichment
SLACK_CONSOLE_URL Worker No Console URL for drill-down links (default: https://console.log10x.com)

The bot reads your analyzer cost ($/GB) from your Console profile settings at startup and refreshes it hourly. To change it, update the cost in your profile — the bot picks up the new value within an hour.

Multi-Environment Setup

To query multiple Log10x environments from a single Slack workspace, use TENX_SLACK_ENVS instead of the single-env variables:

[
  {
    "nickname": "prod",
    "tenant": "abc-123",
    "apiKey": "your-api-key",
    "envId": "abc-123"
  },
  {
    "nickname": "staging",
    "tenant": "def-456",
    "apiKey": "your-staging-api-key",
    "envId": "def-456"
  }
]

Each environment requires an apiKey. The bot fetches the analyzer cost from your Console profile settings via the API key — no cost configuration is stored in the Slack config.

With multiple environments, prefix any command with the nickname: /log10x prod cart or /log10x staging top.

Commands

Command Description
/log10x Show usage and available commands
/log10x top Top 5 patterns by cost across all services
/log10x <service> Cost drivers for a specific service
/log10x <label>=<value> Cost drivers filtered by any metric attribute
/log10x <service> <timeframe> Cost drivers with custom time range
/log10x list List services ranked by cost
/log10x list <label> List values for an attribute ranked by cost
/log10x list labels Show all available attributes
/log10x event <log text> Look up a log event in the console
/log10x setup Show current bot configuration
/log10x help Show help message

Timeframes

Append a timeframe to any query. Default is 7d.

Timeframe Label Baseline
1d Last 24h Avg of prior 3 days
7d This week Avg of prior 3 weeks
30d Last 30d Avg of prior 3 months

Filtering by Attribute

By default, /log10x <name> filters by tenx_user_service. To filter by other metric attributes, use label=value syntax:

/log10x k8s_namespace=production
/log10x severity_level=ERROR
/log10x k8s_container=api-server 30d

Multiple filters use AND logic — combine them to narrow down:

/log10x k8s_namespace=production severity_level=ERROR 7d

Available attributes are discovered from your Prometheus data, not hardcoded. Run /log10x list labels to see what's available. Common attributes include tenx_user_service, k8s_namespace, k8s_container, severity_level, and http_code, but any attribute your Enrich pipeline produces is queryable — including custom enrichment modules.

Discovering Services and Attributes

Use list to explore what data is available before querying:

/log10x list                    # Services ranked by cost (default: 7d)
/log10x list 30d                # Services ranked by cost (last 30 days)
/log10x list k8s_namespace      # Namespace values ranked by cost
/log10x list severity_level 1d  # Severity levels, last 24h
/log10x list labels             # All available attributes

Cost Driver Analysis

When you query a service or attribute, the bot runs a cost driver algorithm:

  1. Query current window — bytes per pattern for the selected timeframe
  2. Query baseline — average of the 3 prior windows of the same size
  3. Compute delta — cost_this_period - cost_baseline per pattern
  4. Apply gates — a pattern is a cost driver when it passes both:
    • Dollar floor: delta exceeds $500/period (configurable)
    • Contribution floor: delta is at least 5% of the total service increase (configurable)
  5. Sort by delta descending

The output shows:

  • Cost drivers — patterns whose cost increase passes both gates, with before/after dollar amounts and event counts
  • Also costing you — top patterns by absolute cost that aren't drivers (expensive but stable)
  • Footer — driver attribution percentage and a link to the Console for the full pattern list

When no single pattern dominates the increase, the bot reports "broad traffic growth" instead of attributing to individual drivers.

Event Lookup

Paste a raw log line to look it up in your metrics:

/log10x event 2024-01-15T10:23:45.678Z ERROR [cart-service] AddItemAsync failed: Connection timeout

The bot extracts the log pattern using the 10x Engine, queries Prometheus for cost data across all services, and returns:

  • Pattern name (AI-generated when enrichment is configured)
  • Cost per service with baseline trend
  • Event count
  • A drill-down link to the Console

AI Enrichment

When TENX_SLACK_API_KEY and TENX_SLACK_ENV_ID are set, the bot calls the Log10x AI service to generate human-readable names for log patterns. For example, a pattern like cart_cartstore_ValkeyCartStore becomes Valkey Cart Store Access.

AI enrichment is optional — without it, the bot falls back to rule-based pattern name prettification. Names that are ambiguous or duplicated across patterns are automatically discarded.

Architecture

The bot uses a two-Lambda pattern for reliability:

  1. Ack Lambda (Python) — receives the Slack webhook, verifies the signing secret, returns 200 OK within Slack's 3-second deadline, then invokes the worker Lambda asynchronously
  2. Worker Lambda (Java) — queries Prometheus, runs the cost driver algorithm, enriches patterns with AI names, and posts the result to Slack via response_url

This separation ensures the user always gets an immediate acknowledgment ("Analyzing..."), and the heavy Prometheus work runs reliably as a separate Lambda invocation with a 60-second timeout. The worker Lambda uses SnapStart for sub-second Java cold starts.

Security

  • All requests are verified using Slack's signing secret (HMAC-SHA256)
  • Requests older than 5 minutes are rejected (replay protection)
  • Response URLs are validated to ensure they point to *.slack.com
  • The bot queries read-only Prometheus metrics — it cannot modify your data