Sinks

Sinks define where collected data is written. You can write to the local filesystem, S3-compatible object storage, or stdout.

Filesystem

Write output files to a directory on the local filesystem.

Configuration

FilesystemSinkSpec configures filesystem output.

• path - Optional (string)

  Path is the directory to write files to (default: current directory).

• prefix - Optional Template (string)

  Prefix is prepended to filenames.

Template variables

The prefix field supports these variables:

Variable	Description	Example
$JOB_NAME	The job’s metadata.name	my-job
$JOB_DATE_ISO8601	UTC time in ISO8601 basic format	20240115T143052Z
$JOB_DATE_RFC3339	UTC time in RFC3339 format	2024-01-15T14:30:52Z

Example

output:
  sink:
    filesystem:
      path: ./output
      prefix: $JOB_NAME/$JOB_DATE_RFC3339/

This writes files to ./output/my-job/2024-01-15T14:30:52Z/<step-id>.json.

---

S3

Write output to S3-compatible object storage. Supports AWS S3, Cloudflare R2, MinIO, and other S3-compatible services.

Configuration

S3SinkSpec configures S3-compatible object storage output. Supports AWS S3, Cloudflare R2, MinIO, and other S3-compatible services.

• bucket - Required Template (string)

  Bucket is the S3 bucket name.

• region - Optional Template (string)

  Region is the AWS region (optional, uses SDK defaults if not specified).

• endpoint - Optional Template (string)

  Endpoint is a custom endpoint URL for S3-compatible services (e.g., R2, MinIO).

• prefix - Optional Template (string)

  Prefix is prepended to object keys. Note: ${JOB_DATE_ISO8601} is recommended for S3 keys as RFC3339 contains colons which require URL encoding.

• credentials - Optional A S3Credentials block as defined below.

  Credentials provides explicit credentials (optional, uses SDK credential chain if not specified).

• force_path_style - Optional (bool)

  ForcePathStyle forces path-style addressing (required for MinIO and some S3-compatible services).

S3Credentials

S3Credentials provides explicit S3 credentials.

• access_key_id - Required Template (string)

  AccessKeyID is the AWS access key ID.

• secret_access_key - Required Template (string)

  SecretAccessKey is the AWS secret access key.

Template variables

The prefix field supports the same variables as the filesystem sink. For S3 keys, $JOB_DATE_ISO8601 is recommended since RFC3339 contains colons that require URL encoding.

Examples

AWS S3

output:
  sink:
    s3:
      bucket: my-bucket
      region: us-east-1
      prefix: infracollect/$JOB_NAME/$JOB_DATE_ISO8601/

Uses the default AWS SDK credential chain (environment variables, shared credentials file, IAM role, etc.).

MinIO

output:
  sink:
    s3:
      bucket: my-bucket
      endpoint: http://localhost:9000
      force_path_style: true
      credentials:
        access_key_id: minioadmin
        secret_access_key: minioadmin
      prefix: $JOB_NAME/

Cloudflare R2

output:
  sink:
    s3:
      bucket: my-bucket
      endpoint: https://<account-id>.r2.cloudflarestorage.com
      credentials:
        access_key_id: <r2-access-key>
        secret_access_key: <r2-secret-key>
      prefix: $JOB_NAME/

---

Stdout

Write output to standard output. Useful for piping to other tools or debugging.

output:
  sink:
    stdout: {}