Compatibility

Splunk dashboards, Universal Forwarders, testing on your own environment, HEC integration, Splunk Enterprise on-premises, and the full Splunk Cloud KV-Store pilot checklist.

Does 10x work with my existing Splunk dashboards and queries

Yes. 10x preserves all Splunk field mappings and metadata.

Fields preserved:

_time, host, source, sourcetype
_raw, index
All field extractions (props.conf, transforms.conf)
Custom metadata and tags

Functionality preserved:

Saved searches, dashboards, and alerts require zero changes
SPL queries return the same results

How optimization works: The 10x Engine uses template-based encoding, not field removal. Each repeated log event is replaced by a compact reference to its template while maintaining full searchability in Splunk.

Does this work with Universal Forwarders

Yes. The 10x Receiver deploys as a sidecar to Universal Forwarders; the Reporter deploys as a DaemonSet alongside them.

In Kubernetes, 10x runs as a DaemonSet alongside the Universal Forwarder; on a VM, it runs as a local process. In both cases it reads the forwarder output, optimizes, and forwards to Splunk HEC.

Compatibility: Universal Forwarder 8.x, 9.x. No changes to Universal Forwarder configuration required. Works with existing HEC endpoints and token authentication.

How do I test this on my Splunk environment

Dev, Run on your Splunk log files locally. One-line install, results in minutes. No account, no credit card.
Reporter, Deploy as a DaemonSet alongside your forwarder for pre-SIEM cost visibility. Alternatively, the MCP server's SIEM-sample tool connects to Splunk via REST API for agentless SIEM-side analysis (no DaemonSet).
Receiver, Deploy via Helm chart alongside your forwarder. Filter mode (sampling) or Compact mode (lossless shrink via the 10x for Splunk app). ~30 min setup.
Retriever, Route events to S3, stream selected data to Splunk on-demand.

Each step is independent, start with Dev to see your reduction ratio, then move to production when ready.

How does this integrate with Splunk HEC

The 10x Engine sits between your log forwarder and Splunk HEC:

App → Fluentd/UF → 10x Receiver (Compact mode) → Splunk HEC → Indexer

Execution (Receiver):

Deploys as sidecar to Universal Forwarders or alongside Fluentd/OTel
Filters (lossy) or compacts (lossless) logs in-memory before forwarding to HEC
Uses standard HEC token authentication
Can also forward via splunktcp to indexers (standard UF protocol)

Observability (Reporter):

Deploys as a DaemonSet alongside your forwarder (not in the critical log path)
Emits pre-SIEM cost insight metrics to Prometheus / ROI Analytics
Fails independently, logs continue flowing to Splunk if Reporter goes down

Fetching from S3:

Retriever routes events to your own S3 and returns the exact offloaded events to HEC on-demand

Agentless SIEM analysis: The MCP server's SIEM-sample tool queries Splunk REST API for cost analysis without deploying a DaemonSet.

Compatibility: Works with Splunk Cloud Platform (all regions), Splunk Enterprise 8.x and 9.x. No changes to HEC configuration required.

Works with Splunk Enterprise on-premises

Yes. Works with both Splunk Cloud and Splunk Enterprise on-premises (8.x, 9.x), across single-instance, distributed, and clustered deployments.

On-premises, the Receiver sidecar and Reporter DaemonSet forward to HEC or to Heavy Forwarders, using either HEC token or Heavy Forwarder S2S authentication.

Agentless SIEM analysis via log10x-mcp for on-premises:

SIEM-sample tool queries Splunk Enterprise REST API
Requires admin or power user credentials
Runs wherever you run an MCP server (workstation, CI, infrastructure)
No data egress to external systems

Retriever for on-premises: Works with AWS S3, Azure Blobs, and any S3-compatible object storage. Returns the exact offloaded events to Splunk Enterprise HEC endpoints on-demand.

Splunk Cloud: KV Store Setup & Pilot Checklist

The 10x for Splunk app expands compact events at search time using a KV Store collection. Here's the complete setup and pilot validation checklist for Splunk Cloud.

Before You Start

Admin or Power User access to your Splunk Cloud instance
Splunk Cloud supporting KV Store (all modern instances do)
Ability to create HTTP Event Collector (HEC) tokens
Two HEC tokens configured: one for templates (tenx_dml_raw_json sourcetype), one for encoded events

Day 1: App Installation & KV Store Setup

Install 10x for Splunk app - [ ] Download from GitHub - [ ] Upload via Settings > Apps > Install app from file - [ ] Restart (if prompted by Splunk Cloud) - [ ] Verify: Settings > Apps > Confirm "10x for Splunk" appears in app list
Create KV Store collection - [ ] Go to Settings > Advanced Search > Collections - [ ] Create new collection named tenx_dml - [ ] Schema fields (automatically generated, verify all present):
```
_key (primary key)
pattern_hash (string)
pattern (string)
pattern_parts (array)
part_0 (string)
pattern_terminator (string)
timestamp_format (string)
```
- [ ] Verify: | inputlookup tenx-dml-lookup | stats count returns 0 (empty)

Create indexes for template data - [ ] Create tenx_dml index (required for storing templates):

Settings → Indexes → New Index
Name: tenx_dml
Data type: Events
Max size: 10GB (adjust based on your expected template volume)
Retention: 30+ days (templates are reference data, not logs)

Verify: | rest /services/data/indexes | search title="tenx_dml" - [ ] Optional: Create separate index for encoded events:

Settings → Indexes → New Index
Name: encoded_events (or your preferred name)
Data type: Events
Max size: Depends on log volume

Or use main index if preferred (encoded events are searchable until expansion)

Verify props.conf and transforms.conf - [ ] Check Settings > Field Extractions > Verify tenx_encoded sourcetype has REPORT-tenx extraction - [ ] Verify transforms.conf has tenx-hash-vars-extraction and tenx-dml-lookup defined - [ ] If missing, manually add via Settings > Add data > Source type settings

Phase 4: Enable HTTP Event Collector (HEC) globally

4a. Enable HEC globally (required before creating tokens): - [ ] Settings → Data Inputs → HTTP Event Collector → Global Settings - [ ] Toggle "All Tokens" to ENABLED - [ ] Set Default Input Port: 8088 (or your custom port) - [ ] Enable SSL: YES (recommended for production) - [ ] Click Save - [ ] Verify: | rest /services/data/inputs/http | search disabled=0

4b. Create HEC Token 1 (for templates): - [ ] Settings → Data Inputs → HTTP Event Collector → New Token - [ ] Name: tenx-templates - [ ] Source Type: tenx_dml_raw_json - [ ] Index: tenx_dml (created in step 3) - [ ] Indexes allowed: tenx_dml (restrict to this index only) - [ ] Disabled: NO - [ ] Click Save Token - [ ] Copy the token value (save it for later)

4c. Create HEC Token 2 (for encoded events): - [ ] Settings → Data Inputs → HTTP Event Collector → New Token - [ ] Name: tenx-encoded - [ ] Source Type: tenx_encoded - [ ] Index: Your target index (where searchable events go) - [ ] Indexes allowed: Your target index - [ ] Disabled: NO - [ ] Click Save Token - [ ] Copy the token value (save it for later)

Week 1: Data Ingestion & KV Store Population

Send template data via HEC (or via your log forwarder) - [ ] Via curl (for testing):

SPLUNK_HOST="your-splunk-cloud.splunkcloud.com"
SPLUNK_PORT="8088"
HEC_TOKEN="<your-tenx-templates-token>"

curl -k https://$SPLUNK_HOST:$SPLUNK_PORT/services/collector/event \
  -H "Authorization: Splunk $HEC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "event": {
      "templateHash": "abc123def456",
      "template": "User %s logged in from %s",
      "templateParts": ["User", "logged in from"]
    },
    "sourcetype": "tenx_dml_raw_json",
    "index": "tenx_dml"
  }'

- [ ] Via Fluentd/Fluent Bit: Configure your forwarder output to send to this HEC endpoint with same token and sourcetype - [ ] Verify templates arriving: index=tenx_dml sourcetype=tenx_dml_raw_json | head 10

Send encoded events via HEC (or via your log forwarder) - [ ] Via curl (for testing):

SPLUNK_HOST="your-splunk-cloud.splunkcloud.com"
SPLUNK_PORT="8088"
HEC_TOKEN="<your-tenx-encoded-token>"

curl -k https://$SPLUNK_HOST:$SPLUNK_PORT/services/collector/event \
  -H "Authorization: Splunk $HEC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "event": "~abc123def456,admin,192.168.1.1",
    "sourcetype": "tenx_encoded",
    "index": "main"
  }'

- [ ] Via Fluentd/Fluent Bit: Configure your forwarder output to send to this HEC endpoint with same token and sourcetype - [ ] Verify encoded events arriving: index=main sourcetype=tenx_encoded | head 10

Wait for KV Store population - [ ] The "Consume KV" saved search runs every 2 minutes (automatic) - [ ] Check: index=_internal savedsearch_name="Consume KV" | table _time, status, result_count (verify no errors) - [ ] Check KV store: | inputlookup tenx-dml-lookup | stats count (should be > 0 after 2-3 min)
Monitor template consumption - [ ] Run: sourcetype=tenx_dml_pure | stats count (confirms templates are searchable; this sourcetype lands in the index set by dest_dml_index, main by default) - [ ] Run: | inputlookup tenx-dml-lookup | head 5 | table _key, pattern, timestamp_format (verify structure)

Week 2: Expansion Validation & Performance Testing

Test basic expansion - [ ] Run: index=your_target_index sourcetype=tenx_encoded | head 10 | \tenx-inflate`- [ ] Verify: All fields (raw,_time,host,source, etc.) are restored to original values - [ ] Check: Notenx_hash,tenx_var*` fields remain in final output (cleanup working)
Test debug mode - [ ] Run: index=your_target_index sourcetype=tenx_encoded | head 1 | \tenx-inflate-debug` | table *- [ ] Verify:_rawfield matches original (unencoded) log format - [ ] Check:tenx_ts_sec` correctly detects timestamp precision (milliseconds vs nanoseconds)
Test field extractions and searches post-expansion - [ ] Run existing saved search/dashboard on expanded data - [ ] Verify: All field extractions work (extractions applied post-expansion) - [ ] Check: Alerts trigger correctly on expanded events - [ ] Compare: Results match pre-optimization historical logs (sample query on same time range)
Measure search performance - [ ] Run: index=your_target_index sourcetype=tenx_encoded earliest=-1h | \tenx-inflate` | stats count- [ ] Note: Thetenx-inflate` macro adds a per-event decode step at search time; expect a small overhead relative to searching un-encoded events - [ ] Compare: Same query without expansion vs with expansion, and validate the overhead on your own data volume

Post-Pilot: Production Deployment

Enable analytics dashboard - [ ] Open: App launcher > 10x for Splunk > Analytics Dashboard - [ ] Verify: Shows total compact events, reduction ratio, storage savings - [ ] Check: Panels populate when opened (the dashboard runs its searches on view)
Set up monitoring and alerts - [ ] Monitor KV store size: | inputlookup tenx-dml-lookup | stats count (alert on growth or error against the threshold you set) - [ ] Monitor expansion failures: Check tenx app logs for errors - [ ] Optional: Set up dashboard for Splunk license impact (GB before/after)
Risk Mitigation & Rollback - [ ] Rollback procedure: Simply disable 10x for Splunk app:
1. Settings > Apps > 10x for Splunk > Disable
2. Re-run searches without \tenx-inflate`` macro (searches work on encoded raw data until disabled)
3. KV Store collection remains; can re-enable app without data loss - [ ] Zero data loss: Encoded events remain in index; templates preserved in KV Store - [ ] Retention: Configure KV Store collection retention if needed (Settings > Collections)

Splunk Cloud Limitations & Workarounds

No custom Python alert actions, Covered. App uses standard KV Store and Search hooks (no custom Python required)
Limited app customization, App config available in local/default folders; can override via local/ without modifying default/
Network egress, All data stays within Splunk Cloud. No external calls needed after app installation
KV Store max size, Bounded by your Splunk KV Store limits, one entry per template. Monitor via | inputlookup tenx-dml-lookup | stats count. If approaching your limit, consider archiving old templates

Forwarder Configuration Examples

Fluentd:

<match encoded_events>
  @type http_buffered
  endpoint_url https://<splunk-host>:8088/services/collector/event
  serializer json
  auth_type basic
  auth_key "Splunk <your-hec-token>"
  <buffer>
    flush_interval 10s
  </buffer>
</match>

Fluent Bit:

[OUTPUT]
Name http
Match *
Host <splunk-host>
Port 8088
URI /services/collector/event
header Authorization Splunk <your-hec-token>
header Content-Type application/json
json_date_key timestamp
Format json

Universal Forwarder: The Universal Forwarder integration uses a file relay pattern, not a custom forwarder output. Fluent Bit with the 10x sidecar reads logs from the original folder, filters and (optionally) compacts them, and writes the processed output to a second folder. The Universal Forwarder monitors that second folder with a standard inputs.conf and ships to the indexers over its normal S2S protocol. The UF keeps its existing configuration; no custom outputs are required.

Support & Troubleshooting

Templates not in KV Store: Check saved search logs: index=_internal savedsearch_name="Consume KV"
Expansion returns empty: Verify template format in tenx_dml_raw_json index, ensure KV Store has matching hash
Performance degradation: Limit time ranges in searches; filter by tenx_hash before expansion for large datasets
HEC token rejected: Verify token is enabled and not deleted: | rest /services/data/inputs/http
For detailed troubleshooting: See 10x for Splunk Troubleshooting Guide on GitHub