KV Store

Validating the KV Store, diagnosing "Consume KV", loading templates indexed earlier, sizing, and search head clusters.

How do I validate that KV Store is working correctly

Run each search in the Log10x App.

1. Templates are arriving:

   index=tenx_dml sourcetype=tenx_dml_raw_json | stats count
   

   Expected: more than 0. If 0, check the templates HEC token points at tenx_dml.

2. Templates are stored:

   | inputlookup tenx-dml-lookup | stats count
   

   Expected: close to the number of distinct templates sent.

3. "Consume KV" is running:

   index=_internal sourcetype=scheduler savedsearch_name="Consume KV"
   | stats latest(status) as status, latest(_time) as last_run
   | eval last_run=strftime(last_run, "%F %T")
   

   Expected: status=success, last_run within the last 5 minutes.

The Diagnostics dashboard runs these checks and more on one page.

\"Consume KV\" is not storing templates

"Consume KV" runs every 5 minutes over the last 7 minutes of tenx_dml_raw_json and hands the results to the tenx_dml_to_kv alert action, which writes the KV Store.

Symptom	Cause	Fix
No scheduler entries	The search is disabled	Enable it in Settings → Searches, reports, and alerts
success, KV count stays 0	No templates in the last 7 minutes	Check the templates token, then run the backfill search below
success, KV count stays 0, templates present	The alert action failed	Read $SPLUNK_HOME/var/log/splunk/tenx_dml_to_kv.log

The alert action's log is also searchable:

index=_internal source=*tenx_dml_to_kv.log* (ERROR OR WARNING)

Templates were indexed before the app was installed

"Consume KV" reads only the last 7 minutes, so templates indexed earlier, or while it was disabled or failing, are never stored and their events stay compact.

Run this search once, as an admin or power user:

index=tenx_dml sourcetype=tenx_dml_raw_json earliest=-30d | sendalert tenx_dml_to_kv

Widen earliest to reach older templates. Templates already stored are skipped, so running it again is safe. The app's Backfill KV saved search holds the same search, but the Run button does not fire its alert action. Encoded events need no re-indexing: they expand as soon as their template is in the KV Store.

How large does the KV Store get

One row per template, not per event, so size follows the number of distinct templates. The Analytics dashboard shows the count as Active Templates; the same number from SPL:

| inputlookup tenx-dml-lookup | stats count

The tenx-inflate macro reads the single tenx_dml collection; keep every active template there.

Search head clusters

The tenx_dml collection is defined in the app's default/collections.conf and replicates with the rest of the cluster's KV Store. Deploy the app to every search head with the deployer. Each member reports the same count:

| inputlookup tenx-dml-lookup | stats count

Only admins and power users can write the collection. The scheduled "Consume KV" search runs on one member, and its writes replicate to the others.