Skip to content

SIEM connector for an offload bucket

10x lands offload as NDJSON in your own S3 bucket, and the SIEM reads it from there. This tool emits the connector that makes Datadog or Splunk pull those objects: the AWS plumbing (an SNS fan-out plus IAM) and the SIEM-native input config, matched to the offload layout (one JSON event per line, Hive-partitioned keys).

It never pushes to the SIEM. Vendor-driven re-ingest fights Datadog Rehydration billing, Splunk HEC permissions, and Elastic _bulk limits, so the bucket stays the system of record and the SIEM subscribes to it. Call it once an offload destination is already writing to the bucket (see configure engine).

Example

You

wire Splunk to read my offload bucket acme-logs-offload

Log10x

SQS-based S3 input (the recommended mode at scale). Apply the plumbing, then add the input.

# S3 ObjectCreated -> SNS -> SQS. The bucket already notifies the 10x
# indexer, and S3 allows one notification config per prefix, so fan out via
# SNS and subscribe both the indexer queue AND the new Splunk queue.
resource "aws_sqs_queue" "tenx_offload_splunk" { name = "tenx-offload-splunk" }
resource "aws_sns_topic"  "tenx_offload_fanout" { name = "tenx-offload-fanout" }
resource "aws_s3_bucket_notification" "offload_to_sns" {
  bucket = "acme-logs-offload"
  topic {
    topic_arn     = aws_sns_topic.tenx_offload_fanout.arn
    events        = ["s3:ObjectCreated:*"]
    filter_prefix = "app/"
  }
}

[aws_sqs_based_s3://tenx-offload]
sqs_queue_url = https://sqs.us-east-1.amazonaws.com/<account>/tenx-offload-splunk
sourcetype    = log10x:offload
index         = main

IAM for the add-on role is the SQS actions plus s3:GetObject and kms:Decrypt. One caveat: the SQS-based input only collects objects created after this is wired, so backfill older data with a one-shot Generic S3 input over the same prefix.

More to ask

  • "Datadog instead, I already run the Forwarder"
  • "both Datadog and Splunk off the same bucket"
  • "why doesn't it push straight to Datadog"

Prerequisites

  • An offload destination already writing NDJSON to the bucket (the offload action from configure engine).
  • The SIEM's own ingest agent in your account: a deployed Datadog Forwarder Lambda, or a Splunk instance with the Splunk Add-on for AWS. 10x emits the wiring; the vendor agent runs the pull.
  • Permission to add an S3 notification plus an SNS topic on the bucket. When the bucket already notifies the 10x indexer, the recipe fans out via SNS so both consumers receive events.

Logs, not metrics

The SIEM ingests the offload as log lines. For per-pattern metrics in Datadog, use backfill metric, which writes an S3 cohort straight to the Datadog metrics API.