Skip to content

Object Storage Input

Object storage inputs provide a method for querying events in an Object Storage (e.g., AWS S3, Azure blobs) directly, without requiring data to be copied to a log analyzer.

The object storage module comprises of index output, query input and storage access sub-modules. These components execute jointly to index, query and stream event data with speed at any scale.

Index

graph LR
    A["<div style='font-size: 14px;'>📥 Trigger</div><div style='font-size: 10px;'>S3 → SQS</div>"] --> B["<div style='font-size: 14px;'>📦 Fetch Blob</div><div style='font-size: 10px;'>new upload</div>"]
    B --> C["<div style='font-size: 14px;'>⚡ 10x Engine</div><div style='font-size: 10px;'>Transform to TenXObjects</div>"]
    C --> D["<div style='font-size: 14px;'>🏷️ Generate Filters</div><div style='font-size: 10px;'>TenXTemplate</div>"]
    D --> E["<div style='font-size: 14px;'>💾 Store Index</div><div style='font-size: 10px;'>metadata</div>"]

    classDef trigger fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef blob fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef filter fill:#0891b2,stroke:#0e7490,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef store fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A trigger
    class B blob
    class C engine
    class D filter
    class E store

Index outputs execute when S3 event notifications are sent directly to SQS queues, triggering index workers to fetch and transform log/trace events enclosed within new blobs into typed TenXObjects.

A dedicated output stream utilizes the TenXObjects to generate lightweight TenXTemplate Filters to enable fast querying of the blob's enclosed event in-place, without first requiring its contents to be duplicated to a log analyzer.

Query

graph LR
    A["<div style='font-size: 14px;'>🔍 Query</div><div style='font-size: 10px;'>REST / scheduled</div>"] --> B["<div style='font-size: 14px;'>🏷️ Filter Index</div><div style='font-size: 10px;'>TenXTemplate</div>"]
    B --> C["<div style='font-size: 14px;'>📦 Fetch Ranges</div><div style='font-size: 10px;'>byte-range reads</div>"]
    C --> D["<div style='font-size: 14px;'>⚡ 10x Engine</div><div style='font-size: 10px;'>Transform/Aggregate</div>"]
    D --> E["<div style='font-size: 14px;'>📤 Output</div><div style='font-size: 10px;'>Analyzers, TSDB</div>"]

    classDef query fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef filter fill:#0891b2,stroke:#0e7490,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef blob fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A query
    class B filter
    class C blob
    class D engine
    class E output

Queries can execute periodically or in response to ad-hoc REST API requests to stream selected TenXObjects from a storage container (e.g., S3 bucket) to log analyzers and time-series DBs.

Queries utilizes storage-filters to pinpoint blob byte-ranges that match specified criteria to transform into TenXObjects for further filtering, aggregation and output.

Storage Access

Access to an object storage (e.g., AWS S3, Azure Blobs, Google, IBM Cloud Storage) is defined by implementing of the ObjectStorageIndexAccessor interface.

This interface is designed to have minimal requirements for put, store and list operations to support virtually any on-premises/hosted key-value object storage.

For an example implementation, see AWSIndexAccess.

Modules

  • Object Storage Index Outputs

    Index files uploaded to a object storage container (e.g. AWS S3 bucket).

    More info

  • Object Storage Input

    Query and stream events from object storage (e.g., AWS S3).

    More info

  • Object Storage Query Inputs

    Query an object storage container for events matching target criteria.

    More info


This module is defined in objectStorage/module.yaml.