Object Storage Fetch Blob
Fetch specified byte range(s) from an object storage container (e.g., AWS S3 bucket) to transform into TenXObjects on which to perform a set of actions.
End-users do not commonly configure fetch object directly but instead use queries to scan through a target container for matching byte ranges to fetch.
Options
Specify the options below to configure multiple Object storage fetch blob:
| Name | Description | Category |
|---|---|---|
| queryObjectQueryName | Logical name to give this query | General |
| queryObjectPrintProgress | Sets whether this input prints throughput stats to the console | General |
| queryObjectWriteResults | Sets whether matched events are buffered to JSONL and uploaded to object storage | General |
| queryObjectObjectStorageName | Object storage logical name | Container |
| queryObjectContainer | Storage container of the target blob | Container |
| queryObjectIndexContainer | Object storage container containing index objects | Container |
| queryObjectTarget | Logical name identifying the origin of 'queryObjectTargetObject' | Container |
| queryObjectTargetObject | The target object to retrieve from storage | Container |
| queryObjectByteRanges | Array of byte ranges to fetch | Container |
| queryObjectFilter | Filter applied on objects in range | Filter |
| queryObjectFrom | Beginning of the search time range (inclusive) | Filter |
| queryObjectTo | End of the search time range (exclusive) | Filter |
| queryObjectObjectStorageArgs | Custom object storage args | Advanced |
| queryObjectID | Unique ID of host query | Advanced |
| queryObjectElapseTime | Epoch after which processing times out | Advanced |
| queryObjectLogLevels | Log levels to send to CloudWatch Logs for query progress tracking | Advanced |
| queryObjectLogGroup | CloudWatch Logs log group for query event logging | Advanced |
General
queryObjectQueryName
Logical name to give this query.
| Type | Required | Category |
|---|---|---|
| String | ✔ | General |
Specifies a logical name associated with the query. This value identifies this query operation in aggregated metrics reports.
queryObjectPrintProgress
Sets whether this input prints throughput stats to the console.
| Type | Default | Category |
|---|---|---|
| Boolean | false | General |
Sets whether this input prints throughput stats to the console This value is commonly used when testing an integration to a remote endpoint.
queryObjectWriteResults
Sets whether matched events are buffered to JSONL and uploaded to object storage.
| Type | Default | Category |
|---|---|---|
| Boolean | false | General |
Controls whether a second output alongside the byte-count marker writer buffers matched TenXObject events to a local temp file as JSONL and uploads them to object storage on close.
Defaults to false so the extra JSONL upload only runs when a caller
explicitly opts in. When set to true, query results are written
under the queryResults index prefix so out-of-band consumers
(e.g. the Log10x MCP server) can retrieve them by polling S3 for
{indexObjectPath(queryResults)}/{queryId}/*.jsonl. The fields
serialized per event default to timestamp + text plus every name
registered in the global enrichmentFields option.
The MCP server opts in by sending writeResults=true on its query
request; the coordinator forwards the flag to stream workers as a
queryObjectWriteResults=true bootstrap arg on the stream pipeline
launch. Deployments that rely solely on the byte-count marker
backstop used by the query coordinator leave this off and pay no
results-writer overhead.
Container
queryObjectObjectStorageName
Object storage logical name.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Container |
Identifies the object storage against which this query is made (e.g., AWS).
queryObjectContainer
Storage container of the target blob.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Container |
Specifies the object storage container (e.g., AWS S3 bucket) of the target blob (e.g., log file) byte range to fetch.
queryObjectIndexContainer
Object storage container containing index objects.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Container |
Specifies the object storage container (e.g., AWS S3 bucket) name in which to store TenXTemplate Filters.
queryObjectTarget
Logical name identifying the origin of 'queryObjectTargetObject'.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Container |
Specifies the logical name used to store index objects produced by the index app.
queryObjectTargetObject
The target object to retrieve from storage.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Container |
Sets the name of the object (e.g., log file) to retrieve from queryObjectContainer.
queryObjectByteRanges
Array of byte ranges to fetch.
| Type | Default | Category |
|---|---|---|
| List | [] | Container |
Defines an array of byte-range pairs (e.g., start1,length1..startN,lengthN) to fetch from queryObjectTargetObject.
To learn more see byte range fetches.
If not set, fetch the entire object.
Filter
queryObjectFilter
Filter applied on objects in range.
| Type | Default | Category |
|---|---|---|
| String | "" | Filter |
Specifies filtering logic for TenXObjects in this query object. Combines both the querySearch (Bloom filter pre-filtering from storage) and queryFilters (in-memory JavaScript expressions) from the originating query. This ensures consistent application of both storage-level and in-memory filtering.
queryObjectFrom
Beginning of the search time range (inclusive).
| Type | Required | Category |
|---|---|---|
| Number | ✔ | Filter |
Defines the beginning of the search time range (inclusive) expressed as a UNIX millisecond epoch value. This value commonly uses the now function.
For example, setting this value to:
```yaml queryObjectFrom: $=now("-1h") ````
express the value of one hour before the current millisecond epoch.
queryObjectTo
End of the search time range (exclusive).
| Type | Required | Category |
|---|---|---|
| Number | ✔ | Filter |
Defines the end of the search time range expressed as an UNIX millisecond epoch value (exclusive). This value commonly uses the now function.
Advanced
queryObjectObjectStorageArgs
Custom object storage args.
| Type | Default | Category |
|---|---|---|
| List | [] | Advanced |
Custom arguments passed as a map to the constructor of the underlying object storage. This list is expected to hold pairs of key values (e.g., args: [key1, value1, key2, value2]).
queryObjectID
Unique ID of host query.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Advanced |
UUID of the query which has submitted this fetch request.
queryObjectElapseTime
Epoch after which processing times out.
| Type | Required | Category |
|---|---|---|
| Number | ✔ | Advanced |
Epoch value which if exceeds the current system time will result in this request being cancelled.
queryObjectLogLevels
Log levels to send to CloudWatch Logs for query progress tracking.
| Type | Default | Category |
|---|---|---|
| List | [] | Advanced |
Specifies which query event log levels are sent to CloudWatch Logs for tracking query progress.
Valid values: ERROR, INFO, DEBUG, PERF.
If not specified, defaults to ERROR, INFO, PERF.
queryObjectLogGroup
CloudWatch Logs log group for query event logging.
| Type | Default | Category |
|---|---|---|
| String | "" | Advanced |
Specifies the CloudWatch Logs log group name for logging query lifecycle events. The log group must be pre-created (e.g. by Terraform). If not specified or empty, query event logging is disabled.
This module is defined in object/module.yaml.