Object Storage Fetch Blob
Fetch specified byte range(s) from an object storage container (e.g., AWS S3 bucket) to transform into TenXObjects on which to perform a set of actions.
End-users do not commonly configure fetch object directly but instead use queries to scan through a target container for matching byte ranges to fetch.
The query coordinator dispatches each stream worker with bootstrap args carrying its slice bounds (queryObjectSliceFrom, queryObjectSliceTo) and output-mode flags (queryObjectWriteResults, queryObjectWriteSummaries). Slice bounds are encoded into output S3 keys: events at qr/{queryId}/{sliceFrom}_{sliceTo}/{worker}.jsonl, summaries at qrs/{queryId}/{sliceFrom}_{sliceTo}/{worker}.jsonl.
Options
Specify the options below to configure multiple Object storage fetch blob:
| Name | Description | Category |
|---|---|---|
| queryObjectQueryName | Logical name to give this query | General |
| queryObjectPrintProgress | Sets whether this input prints throughput stats to the console | General |
| queryObjectWriteResults | Sets whether matched events are buffered to JSONL and uploaded to object storage | General |
| queryObjectWriteSummaries | Sets whether per-slice TenXSummary aggregates are buffered to JSONL and uploaded | General |
| queryObjectSliceFrom | Slice from-bound (epoch ms) for this dispatch | General |
| queryObjectSliceTo | Slice to-bound (epoch ms) for this dispatch | General |
| queryObjectUntimestamped | Whether events that carry no timestamp are admitted by this stream worker | General |
| queryObjectObjectStorageName | Object storage logical name | Container |
| queryObjectContainer | Storage container of the target blob | Container |
| queryObjectIndexContainer | Object storage container containing index objects | Container |
| queryObjectTarget | Logical name identifying the origin of 'queryObjectTargetObject' | Container |
| queryObjectResultTarget | Tier-1 result-sink redirect prefix forwarded by the coordinator | Container |
| queryObjectTargetObject | The target object to retrieve from storage | Container |
| queryObjectByteRanges | Array of byte ranges to fetch | Container |
| queryObjectFilter | Filter applied on objects in range | Filter |
| queryObjectFrom | Beginning of the search time range (inclusive) | Filter |
| queryObjectTo | End of the search time range (exclusive) | Filter |
| queryObjectObjectStorageArgs | Custom object storage args | Advanced |
| queryObjectID | Unique ID of host query | Advanced |
| queryObjectElapseTime | Epoch after which processing times out | Advanced |
| queryObjectLogLevels | Log levels to send to CloudWatch Logs for query progress tracking | Advanced |
| queryObjectLogGroup | CloudWatch Logs log group for query event logging | Advanced |
General
queryObjectQueryName
Logical name to give this query.
| Type | Required | Category |
|---|---|---|
| String | ✔ | General |
Specifies a logical name associated with the query. This value identifies this query operation in aggregated metrics reports.
queryObjectPrintProgress
Sets whether this input prints throughput stats to the console.
| Type | Default | Category |
|---|---|---|
| Boolean | false | General |
Sets whether this input prints throughput stats to the console This value is commonly used when testing an integration to a remote endpoint.
queryObjectWriteResults
Sets whether matched events are buffered to JSONL and uploaded to object storage.
| Type | Default | Category |
|---|---|---|
| Boolean | false | General |
Controls whether a second output alongside the byte-count marker writer buffers matched TenXObject events to a local temp file as JSONL and uploads them to object storage on close.
Defaults to false so the extra JSONL upload only runs when a caller
explicitly opts in. When set to true, query results are written
under a slice-keyed prefix
{indexObjectPath(queryResults)}/{queryId}/{sliceFrom}_{sliceTo}/*.jsonl
so out-of-band consumers (e.g. the Log10x MCP server) can retrieve
partial progress by polling per-slice subdirectories. The fields
serialized per event default to timestamp + text plus every name
registered in the global enrichmentFields option.
The MCP server opts in by sending writeResults=true on its query
request; the coordinator forwards the flag to stream workers as a
queryObjectWriteResults=true bootstrap arg on the stream pipeline
launch. Deployments that rely solely on the byte-count marker
backstop used by the query coordinator leave this off and pay no
results-writer overhead.
queryObjectWriteSummaries
Sets whether per-slice TenXSummary aggregates are buffered to JSONL and uploaded.
| Type | Default | Category |
|---|---|---|
| Boolean | false | General |
Controls whether a second results-style output buffers TenXSummary
records (produced by the optional run/aggregate/stream aggregator)
to a local temp file as JSONL and uploads them to object storage on
close.
Defaults to false. When set to true, summaries are written under
{indexObjectPath(querySummaries)}/{queryId}/{sliceFrom}_{sliceTo}/*.jsonl
so consumers can fetch a low-volume per-slice rollup (count,
bytesum, grouping fields) for noisy patterns instead of pulling raw
events. Independent of queryObjectWriteResults; either can be on
in isolation or both together.
queryObjectSliceFrom
Slice from-bound (epoch ms) for this dispatch.
| Type | Default | Category |
|---|---|---|
| Number | 0 | General |
Set by the query coordinator to the lower bound of the timeslice
being processed by this stream worker. Always populated regardless
of queryObjectFrom/queryObjectTo (which may be zero when the
byte range fully contains the slice and per-event timestamp filtering
is unnecessary). Used by writers to key slice-grouped S3 outputs.
queryObjectSliceTo
Slice to-bound (epoch ms) for this dispatch.
| Type | Default | Category |
|---|---|---|
| Number | 0 | General |
Set by the query coordinator to the upper bound of the timeslice
being processed by this stream worker. See queryObjectSliceFrom.
queryObjectUntimestamped
Whether events that carry no timestamp are admitted by this stream worker.
| Type | Default | Category |
|---|---|---|
| Boolean | true | General |
Set by the query coordinator. A request clipped to a time window
(queryObjectFrom/queryObjectTo) cannot place an event that carries
no timestamp, so it admits every such event or none. A byte range
that straddles a slice boundary is fetched by every slice it touches;
the coordinator sets this to true on exactly one of those requests
(the slice holding the range's start, or the first slice of the query
when the range begins before the query window) and to false on the
rest, so each untimestamped event is returned once. Requests that are
not clipped always carry true.
Container
queryObjectObjectStorageName
Object storage logical name.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Container |
Identifies the object storage against which this query is made (e.g., AWS).
queryObjectContainer
Storage container of the target blob.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Container |
Specifies the object storage container (e.g., AWS S3 bucket) of the target blob (e.g., log file) byte range to fetch.
queryObjectIndexContainer
Object storage container containing index objects.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Container |
Specifies the object storage container (e.g., AWS S3 bucket) name in which to store TenXTemplate Filters.
queryObjectTarget
Logical name identifying the origin of 'queryObjectTargetObject'.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Container |
Specifies the logical name used to store index objects produced by the index app.
queryObjectResultTarget
Tier-1 result-sink redirect prefix forwarded by the coordinator.
| Type | Default | Category |
|---|---|---|
| String | "" | Container |
Bare-token prefix under which this stream worker writes query OUTPUT
(qr/, qrs/, q/ markers) inside the index container. Forwarded by
the query coordinator from queryResultTarget. Blank/absent falls back
to queryObjectTarget, byte-identical to legacy behavior.
queryObjectTargetObject
The target object to retrieve from storage.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Container |
Sets the name of the object (e.g., log file) to retrieve from queryObjectContainer.
queryObjectByteRanges
Array of byte ranges to fetch.
| Type | Default | Category |
|---|---|---|
| List | [] | Container |
Defines an array of byte-range pairs (e.g., start1,length1..startN,lengthN) to fetch from queryObjectTargetObject.
To learn more see byte range fetches.
If not set, fetch the entire object.
Filter
queryObjectFilter
Filter applied on objects in range.
| Type | Default | Category |
|---|---|---|
| String | "" | Filter |
Specifies filtering logic for TenXObjects in this query object. Combines both the querySearch (Bloom filter pre-filtering from storage) and queryFilters (in-memory JavaScript expressions) from the originating query. This ensures consistent application of both storage-level and in-memory filtering.
queryObjectFrom
Beginning of the search time range (inclusive).
| Type | Required | Category |
|---|---|---|
| Number | ✔ | Filter |
Defines the beginning of the search time range (inclusive) expressed as a UNIX millisecond epoch value. This value commonly uses the now function.
For example, setting this value to:
aml
queryObjectFrom: $=now("-1h")
express the value of one hour before the current millisecond epoch.
queryObjectTo
End of the search time range (exclusive).
| Type | Required | Category |
|---|---|---|
| Number | ✔ | Filter |
Defines the end of the search time range expressed as an UNIX millisecond epoch value (exclusive). This value commonly uses the now function.
Advanced
queryObjectObjectStorageArgs
Custom object storage args.
| Type | Default | Category |
|---|---|---|
| List | [] | Advanced |
Custom arguments passed as a map to the constructor of the underlying object storage. This list is expected to hold pairs of key values (e.g., args: [key1, value1, key2, value2]).
queryObjectID
Unique ID of host query.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Advanced |
UUID of the query which has submitted this fetch request.
queryObjectElapseTime
Epoch after which processing times out.
| Type | Required | Category |
|---|---|---|
| Number | ✔ | Advanced |
Epoch value which if exceeds the current system time will result in this request being cancelled.
queryObjectLogLevels
Log levels to send to CloudWatch Logs for query progress tracking.
| Type | Default | Category |
|---|---|---|
| List | [] | Advanced |
Specifies which query event log levels are sent to CloudWatch Logs for tracking query progress.
Valid values: ERROR, INFO, DEBUG, PERF.
If not specified, defaults to ERROR, INFO, PERF.
queryObjectLogGroup
CloudWatch Logs log group for query event logging.
| Type | Default | Category |
|---|---|---|
| String | "" | Advanced |
Specifies the CloudWatch Logs log group name for logging query lifecycle events. The log group must be pre-created (e.g. by Terraform). If not specified or empty, query event logging is disabled.
This module is defined in object/module.yaml.