Compatibility

Splunk dashboards, Universal Forwarders, testing on your own environment, HEC integration, Splunk Enterprise on-premises, and the full Splunk Cloud KV-Store pilot checklist.

Does 10x work with my existing Splunk dashboards and queries

Yes. Log10x preserves all Splunk field mappings and metadata.

Fields preserved:

  • _time, host, source, sourcetype
  • _raw, index
  • All field extractions (props.conf, transforms.conf)
  • Custom metadata and tags

Functionality preserved:

  • Saved searches work identically
  • Dashboards and alerts require zero changes
  • SPL queries return same results
  • Report scheduling unchanged

How optimization works: The 10x Engine uses template-based deduplication, not field removal. Repeated log events are compacted via references to templates while maintaining full searchability in Splunk.

Does this work with Universal Forwarders

Yes. The Log10x Reducer deploys as a sidecar to Universal Forwarders; the Reporter deploys as a DaemonSet alongside them.

Kubernetes deployment:

  • DaemonSet alongside Universal Forwarder
  • Logs forwarded to 10x via file output or stdout
  • 10x optimizes and forwards to Splunk HEC

VM deployment:

  • Local process alongside Universal Forwarder
  • Reads from forwarder output directory
  • Optimizes and forwards to HEC

Compatibility: Universal Forwarder 8.x, 9.x. No changes to Universal Forwarder configuration required. Works with existing HEC endpoints and token authentication.

How do I test this on my Splunk environment
  1. Dev — Run on your Splunk log files locally. One-line install, results in minutes. No account, no credit card.
  2. Reporter — Deploy as a DaemonSet alongside your forwarder for pre-SIEM cost visibility. Alternatively, the MCP server's SIEM-sample tool connects to Splunk via REST API for agentless SIEM-side analysis (no DaemonSet).
  3. Reducer — Deploy via Helm chart alongside your forwarder. Filter mode (sampling) or Compact mode (lossless shrink via the 10x for Splunk app). ~30 min setup.
  4. Retriever — Route events to S3, stream selected data to Splunk on-demand.

Each step is independent — start with Dev to see your reduction ratio, then move to production when ready.

How does this integrate with Splunk HEC

The 10x Engine sits between your log forwarder and Splunk HEC:

App → Fluentd/UF → 10x Reducer (Compact mode) → Splunk HEC → Indexer

Execution (Reducer):

  • Deploys as sidecar to Universal Forwarders or alongside Fluentd/OTel
  • Filters (lossy) or compacts (lossless) logs in-memory before forwarding to HEC
  • Uses standard HEC token authentication
  • Can also forward via splunktcp to indexers (standard UF protocol)

Observability (Reporter):

  • Deploys as a DaemonSet alongside your forwarder (not in the critical log path)
  • Emits pre-SIEM cost insight metrics to Prometheus / ROI Analytics
  • Fails independently — logs continue flowing to Splunk if Reporter goes down

Streaming from S3:

  • Retriever streams from S3 archive to HEC on-demand

Agentless SIEM analysis: The MCP server's SIEM-sample tool queries Splunk REST API for cost analysis without deploying a DaemonSet.

Compatibility: Works with Splunk Cloud Platform (all regions), Splunk Enterprise 8.x and 9.x. No changes to HEC configuration required.

Works with Splunk Enterprise on-premises

Yes. Works with both Splunk Cloud and Splunk Enterprise on-premises.

Splunk Enterprise compatibility:

  • Versions: 8.x, 9.x
  • Deployment: Single instance, distributed, clustered
  • Authentication: HEC token or Heavy Forwarder S2S
  • Integration: Reducer sidecar + Reporter DaemonSet forward to HEC or Heavy Forwarders

Agentless SIEM analysis via log10x-mcp for on-premises:

  • SIEM-sample tool queries Splunk Enterprise REST API
  • Requires admin or power user credentials
  • Runs wherever you run an MCP server (workstation, CI, infrastructure)
  • No data egress to external systems

Retriever for on-premises: Works with AWS S3, Azure Blobs, and any S3-compatible object storage. Streams archived logs to Splunk Enterprise HEC endpoints on-demand.

Splunk Cloud: KV Store Setup & Pilot Checklist

The 10x for Splunk app expands compact events at search time using a KV Store collection. Here's the complete setup and pilot validation checklist for Splunk Cloud.

Before You Start

  • Admin or Power User access to your Splunk Cloud instance
  • Splunk Cloud supporting KV Store (all modern instances do)
  • Ability to create HTTP Event Collector (HEC) tokens
  • Two HEC tokens configured: one for templates (tenx_dml_raw_json sourcetype), one for encoded events

Day 1: App Installation & KV Store Setup

  1. Install 10x for Splunk app - [ ] Download from GitHub - [ ] Upload via Settings > Apps > Install app from file - [ ] Restart (if prompted by Splunk Cloud) - [ ] Verify: Settings > Apps > Confirm "10x for Splunk" appears in app list

  2. Create KV Store collection - [ ] Go to Settings > Advanced Search > Collections - [ ] Create new collection named kvdml - [ ] Schema fields (automatically generated, verify all present): 

    _key (primary key)
pattern_hash (string)
pattern (string)
pattern_parts (array)
part_0 (string)
pattern_terminator (string)
timestamp_format (string)
    - [ ] Verify: | inputlookup tenx-dml-lookup | stats count returns 0 (empty)

  3. Create indexes for template data - [ ] Create tenx_dml index (required for storing templates): 

    Settings → Indexes → New Index
Name: tenx_dml
Data type: Events
Max size: 10GB (adjust based on your expected template volume)
Retention: 30+ days (templates are reference data, not logs)
    Verify: | rest /services/data/indexes | search title="tenx_dml" - [ ] Optional: Create separate index for encoded events: 
    Settings → Indexes → New Index
Name: encoded_events (or your preferred name)
Data type: Events
Max size: Depends on log volume
    Or use main index if preferred (encoded events are searchable until expansion)

  4. Verify props.conf and transforms.conf - [ ] Check Settings > Field Extractions > Verify tenx_encoded sourcetype has REPORT-tenx extraction - [ ] Verify transforms.conf has tenx-hash-vars-extraction and tenx-dml-lookup defined - [ ] If missing, manually add via Settings > Add data > Source type settings

Phase 4: Enable HTTP Event Collector (HEC) globally

4a. Enable HEC globally (required before creating tokens): - [ ] Settings → Data Inputs → HTTP Event Collector → Global Settings - [ ] Toggle "All Tokens" to ENABLED - [ ] Set Default Input Port: 8088 (or your custom port) - [ ] Enable SSL: YES (recommended for production) - [ ] Click Save - [ ] Verify: | rest /services/data/inputs/http | search disabled=0

4b. Create HEC Token 1 (for templates): - [ ] Settings → Data Inputs → HTTP Event Collector → New Token - [ ] Name: tenx-templates - [ ] Source Type: tenx_dml_raw_json - [ ] Index: tenx_dml (created in step 3) - [ ] Indexes allowed: tenx_dml (restrict to this index only) - [ ] Disabled: NO - [ ] Click Save Token - [ ] Copy the token value (save it for later)

4c. Create HEC Token 2 (for encoded events): - [ ] Settings → Data Inputs → HTTP Event Collector → New Token - [ ] Name: tenx-encoded - [ ] Source Type: tenx_encoded - [ ] Index: Your target index (where searchable events go) - [ ] Indexes allowed: Your target index - [ ] Disabled: NO - [ ] Click Save Token - [ ] Copy the token value (save it for later)

Week 1: Data Ingestion & KV Store Population

  1. Send template data via HEC (or via your log forwarder) - [ ] Via curl (for testing): 

    SPLUNK_HOST="your-splunk-cloud.splunkcloud.com"
SPLUNK_PORT="8088"
HEC_TOKEN="<your-tenx-templates-token>"

curl -k https://$SPLUNK_HOST:$SPLUNK_PORT/services/collector/event \
  -H "Authorization: Splunk $HEC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "event": {
      "templateHash": "abc123def456",
      "template": "User %s logged in from %s",
      "templateParts": ["User", "logged in from"]
    },
    "sourcetype": "tenx_dml_raw_json",
    "index": "tenx_dml"
  }'
    - [ ] Via Fluentd/Fluent Bit: Configure your forwarder output to send to this HEC endpoint with same token and sourcetype - [ ] Verify templates arriving: index=tenx_dml sourcetype=tenx_dml_raw_json | head 10

  2. Send encoded events via HEC (or via your log forwarder) - [ ] Via curl (for testing): 

    SPLUNK_HOST="your-splunk-cloud.splunkcloud.com"
SPLUNK_PORT="8088"
HEC_TOKEN="<your-tenx-encoded-token>"

curl -k https://$SPLUNK_HOST:$SPLUNK_PORT/services/collector/event \
  -H "Authorization: Splunk $HEC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "event": "~abc123def456,admin,192.168.1.1",
    "sourcetype": "tenx_encoded",
    "index": "main"
  }'
    - [ ] Via Fluentd/Fluent Bit: Configure your forwarder output to send to this HEC endpoint with same token and sourcetype - [ ] Verify encoded events arriving: index=main sourcetype=tenx_encoded | head 10

  3. Wait for KV Store population - [ ] The "Consume KV" saved search runs every 2 minutes (automatic) - [ ] Check: index=_internal savedsearch_name="Consume KV" | table _time, status, result_count (verify no errors) - [ ] Check KV store: | inputlookup tenx-dml-lookup | stats count (should be > 0 after 2-3 min)

  4. Monitor template consumption - [ ] Run: index=tenx_dml sourcetype=tenx_dml_pure | stats count (confirms templates are searchable) - [ ] Run: | inputlookup tenx-dml-lookup | head 5 | table _key, pattern, timestamp_format (verify structure)

Week 2: Expansion Validation & Performance Testing

  1. Test basic expansion - [ ] Run: index=your_target_index sourcetype=tenx_encoded | head 10 | \tenx-inflate`- [ ] Verify: All fields (raw,_time,host,source, etc.) are restored to original values - [ ] Check: Notenx_hash,tenx_var*` fields remain in final output (cleanup working)

  2. Test debug mode - [ ] Run: index=your_target_index sourcetype=tenx_encoded | head 1 | \tenx-inflate-debug` | table *- [ ] Verify:_rawfield matches original (unencoded) log format - [ ] Check:tenx_ts_sec` correctly detects timestamp precision (milliseconds vs nanoseconds)

  3. Test field extractions and searches post-expansion - [ ] Run existing saved search/dashboard on expanded data - [ ] Verify: All field extractions work (extractions applied post-expansion) - [ ] Check: Alerts trigger correctly on expanded events - [ ] Compare: Results match pre-optimization historical logs (sample query on same time range)

  4. Measure search performance - [ ] Run: index=your_target_index sourcetype=tenx_encoded earliest=-1h | \tenx-inflate` | stats count` - [ ] Note: Search time (should be ~1-3 seconds for expansion overhead) - [ ] Compare: Same query without expansion vs with expansion - [ ] Acceptable: <5 second overhead for 10M+ event searches

Post-Pilot: Production Deployment

  1. Enable analytics dashboard - [ ] Open: App launcher > 10x for Splunk > Analytics Dashboard - [ ] Verify: Shows total compact events, reduction ratio, storage savings - [ ] Check: Updates every minute (confirms scheduled searches running)

  2. Set up monitoring and alerts - [ ] Monitor KV store size: | inputlookup tenx-dml-lookup | stats count (alert if > 1M entries or error) - [ ] Monitor expansion failures: Check tenx app logs for errors - [ ] Optional: Set up dashboard for Splunk license impact (GB before/after)

  3. Risk Mitigation & Rollback - [ ] Rollback procedure: Simply disable 10x for Splunk app:

    1. Settings > Apps > 10x for Splunk > Disable
    2. Re-run searches without \tenx-inflate`` macro (searches work on encoded raw data until disabled)
    3. KV Store collection remains; can re-enable app without data loss - [ ] Zero data loss: Encoded events remain in index; templates preserved in KV Store - [ ] Retention: Configure KV Store collection retention if needed (Settings > Collections)

Splunk Cloud Limitations & Workarounds

  • No custom Python alert actions — Covered. App uses standard KV Store and Search hooks (no custom Python required)
  • Limited app customization — App config available in local/default folders; can override via local/ without modifying default/
  • Network egress — All data stays within Splunk Cloud. No external calls needed after app installation
  • KV Store max size — Typical: 10M-50M entries. Monitor via | inputlookup tenx-dml-lookup | stats count. If approaching limit, consider archiving old templates

Forwarder Configuration Examples

Fluentd: 

<match encoded_events>
  @type http_buffered
  endpoint_url https://<splunk-host>:8088/services/collector/event
  serializer json
  auth_type basic
  auth_key "Splunk <your-hec-token>"
  <buffer>
    flush_interval 10s
  </buffer>
</match>

Fluent Bit: 

[OUTPUT]
Name http
Match *
Host <splunk-host>
Port 8088
URI /services/collector/event
header Authorization Splunk <your-hec-token>
header Content-Type application/json
json_date_key timestamp
Format json

Universal Forwarder: Configure in $SPLUNK_HOME/etc/apps/TA-log10x/local/outputs.conf: 

[tcpout]
defaultGroup = log10x_hec

[tcpout:log10x_hec]
server = <splunk-host>:8088
clientCert = $SPLUNK_HOME/etc/auth/mycerts/client.pem
sslVerifyServerCert = true

Support & Troubleshooting

  • Templates not in KV Store: Check saved search logs: index=_internal savedsearch_name="Consume KV"
  • Expansion returns empty: Verify template format in tenx_dml_raw_json index, ensure KV Store has matching hash
  • Performance degradation: Limit time ranges in searches; filter by tenx_hash before expansion for large datasets
  • HEC token rejected: Verify token is enabled and not deleted: | rest /services/data/inputs/http
  • For detailed troubleshooting: See 10x for Splunk Troubleshooting Guide on GitHub