AWS S3

AWS S3 nodes let you manage objects (files) in Amazon S3 buckets. Each operation is a separate node.

All nodes require an AWS S3 connection configured with your IAM credentials and region.

Nodes

List Objects

List objects in an S3 bucket.

Field	Description	Notes
Bucket	S3 bucket name	Required
Prefix	Filter by key prefix (e.g., uploads/2024/)	Optional. Supports templates
Max Keys	Maximum objects to return	Default: 1000

Supports streaming — each object metadata is emitted individually to downstream nodes.

Read

Read an object from S3 with optional CSV/JSON/JSONL/TSV/XML parsing.

Field	Description	Notes
Source	Bucket + Key below or FileRef from upstream item	Default: settings. The upstream-FileRef mode is for use with S3 List Objects
Bucket	S3 bucket name	Required when Source = settings
Key	Object key (path)	Required when Source = settings. Supports templates
Parse Format	How to parse the body	raw (default), json, jsonl, csv, tsv, or xml

Returns {bucket, key, content_type, size, parse_format, content} where content is the parsed value (string for raw, array of rows for csv/jsonl/tsv, map or array for json/xml).

Write

Write one S3 object from either upstream items or a literal content string. S3 Write is a content-creating sink: it produces the file body and uploads it. (For moving an existing file between buckets without touching its bytes, use S3 Copy or S3 Move instead.)

Field	Description	Notes
Bucket	S3 bucket name	Required
Key	Destination object key	Required. Supports templates
Source	Where the file body comes from	From upstream data (default) or Custom content
Format	How input items are serialized	Visible when Source = data. CSV, TSV, JSON (default), JSON Lines, XML, or Raw
CSV Column Order	Comma-separated columns for CSV/TSV	Visible when Format ∈ {csv, tsv}. Leave empty for union of keys
File Content	Literal body for the upload	Visible when Source = content. Supports templates

The S3 object's Content-Type header is set automatically: from Format when Source = data (e.g. csv → text/csv, jsonl → application/x-ndjson), or application/octet-stream when Source = content.

Source modes:

• From upstream data — serialize every input item into the file using the chosen Format. Use this when you have rows/records from a database query, API response, etc.
• Custom content — write the File Content field verbatim. Use this when you've already composed the bytes yourself (perhaps via templates referencing upstream items) and don't want the engine to serialize anything.

Format choices (Source = data):

• JSON — single JSON array containing every input item. Buffered in memory.
• JSON Lines — one JSON object per line. Streams without buffering.
• CSV / TSV — header is CSV Column Order when set, otherwise the union of keys across all items (sorted alphabetically). Underscore-prefixed metadata fields are skipped. Missing / null cells render empty.
• XML — <items><item>…</item></items> envelope, one element per item.
• Raw — pass-through. Each item must already be a string or []byte; anything else is rejected.

Delete

Delete an object from S3.

Field	Description	Notes
Bucket	S3 bucket name	Required
Key	Object key to delete	Required. Supports templates

Common Patterns

Archive Data to S3 as CSV

1. DataStore Query — fetch the rows (e.g. get_many on tasks)
2. S3 Write — Key: exports/tasks.csv, Format: CSV

The CSV file lands in S3 with one header row + one row per record. No intermediate format node needed.

For very large tables, prefer Format: JSON Lines or Format: CSV — both stream item-by-item and won't buffer the dataset in memory.

Process Uploaded Files

1. Cron — periodic check
2. S3 List Objects — list new files in an input prefix
3. S3 Read — fetch each file
4. Transform — process the data
5. S3 Delete — remove processed files