Object Storage Fetch Blob
Fetch specified byte range(s) from an object storage container (e.g., AWS S3 bucket) to transform into TenXObjects on which to perform a set of actions.
End-users do not commonly configure fetch object directly but instead use queries to scan through a target container for matching byte ranges to fetch.
Options
Specify the options below to configure multiple Object storage fetch blob:
| Name | Description | Category |
|---|---|---|
| queryObjectQueryName | Logical name to give this query | General |
| queryObjectPrintProgress | Sets whether this input prints throughput stats to the console | General |
| queryObjectObjectStorageName | Object storage logical name | Container |
| queryObjectContainer | Storage container of the target blob | Container |
| queryObjectIndexContainer | Object storage container containing index objects | Container |
| queryObjectTarget | Logical name identifying the origin of 'queryObjectTargetObject' | Container |
| queryObjectTargetObject | The target object to retrieve from storage | Container |
| queryObjectByteRanges | Array of byte ranges to fetch | Container |
| queryObjectFilter | Filter applied on objects in range | Filter |
| queryObjectFrom | Beginning of the search time range (inclusive) | Filter |
| queryObjectTo | End of the search time range (exclusive) | Filter |
| queryObjectObjectStorageArgs | Custom object storage args | Advanced |
| queryObjectID | Unique ID of host query | Advanced |
| queryObjectElapseTime | Epoch after which processing times out | Advanced |
General
queryObjectQueryName
Logical name to give this query.
| Type | Required | Category |
|---|---|---|
| String | ✔ | General |
Specifies a logical name associated with the query. This value identifies this query operation in aggregated metrics reports.
queryObjectPrintProgress
Sets whether this input prints throughput stats to the console.
| Type | Default | Category |
|---|---|---|
| Boolean | false | General |
Sets whether this input prints throughput stats to the console This value is commonly used when testing an integration to a remote endpoint.
Container
queryObjectObjectStorageName
Object storage logical name.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Container |
Identifies the object storage against which this query is made (e.g., AWS).
queryObjectContainer
Storage container of the target blob.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Container |
Specifies the object storage container (e.g., AWS S3 bucket) of the target blob (e.g., log file) byte range to fetch.
queryObjectIndexContainer
Object storage container containing index objects.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Container |
Specifies the object storage container (e.g., AWS S3 bucket) name in which to store TenXTemplate Filters.
queryObjectTarget
Logical name identifying the origin of 'queryObjectTargetObject'.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Container |
Specifies the logical name used to store index objects produced by the index app.
queryObjectTargetObject
The target object to retrieve from storage.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Container |
Sets the name of the object (e.g., log file) to retrieve from queryObjectContainer.
queryObjectByteRanges
Array of byte ranges to fetch.
| Type | Default | Category |
|---|---|---|
| List | [] | Container |
Defines an array of byte-range pairs (e.g., start1,length1..startN,lengthN) to fetch from queryObjectTargetObject.
To learn more see byte range fetches.
If not set, fetch the entire object.
Filter
queryObjectFilter
Filter applied on objects in range.
| Type | Default | Category |
|---|---|---|
| String | "" | Filter |
Specifies filtering logic for TenXObjects in this query object. Combines both the querySearch (Bloom filter pre-filtering from storage) and queryFilters (in-memory JavaScript expressions) from the originating query. This ensures consistent application of both storage-level and in-memory filtering.
queryObjectFrom
Beginning of the search time range (inclusive).
| Type | Required | Category |
|---|---|---|
| Number | ✔ | Filter |
Defines the beginning of the search time range (inclusive) expressed as a UNIX millisecond epoch value. This value commonly uses the now function.
For example, setting this value to:
```yaml queryObjectFrom: $=now("-1h") ````
express the value of one hour before the current millisecond epoch.
queryObjectTo
End of the search time range (exclusive).
| Type | Required | Category |
|---|---|---|
| Number | ✔ | Filter |
Defines the end of the search time range expressed as an UNIX millisecond epoch value (exclusive). This value commonly uses the now function.
Advanced
queryObjectObjectStorageArgs
Custom object storage args.
| Type | Default | Category |
|---|---|---|
| List | [] | Advanced |
Custom arguments passed as a map to the constructor of the underlying object storage. This list is expected to hold pairs of key values (e.g., args: [key1, value1, key2, value2]).
queryObjectID
Unique ID of host query.
| Type | Required | Category |
|---|---|---|
| String | ✔ | Advanced |
UUID of the query which has submitted this fetch request.
queryObjectElapseTime
Epoch after which processing times out.
| Type | Required | Category |
|---|---|---|
| Number | ✔ | Advanced |
Epoch value which if exceeds the current system time will result in this request being cancelled.
This module is defined in object/module.yaml.