Skip to main content

​
Overview

Amazon Simple Storage Service (S3) is an object storage service offering industry-leading scalability, availability, and durability. Datazone provides native integration with AWS S3 to read data files directly from your S3 buckets.

​
Connection Parameters

Set on the source when you create it. Names match the API and CLI payload.
​
string
required
Name — a unique identifier for your AWS S3 source.
​
string
required
Bucket Name — the S3 bucket containing your data files.
​
string
required
AWS Access Key ID — the access key ID from your AWS credentials.
​
string
required
AWS Secret Access Key — the secret access key from your AWS credentials. Stored encrypted.
​
string
default:"us-east-1"
AWS Region — the region the bucket lives in, for example eu-west-2.

​
Extract Parameters

Set per extract, in source_parameters. See Shared Extract Parameters for mode, replication_key, and schema_mapping.
​
string
required
Search Pattern — a regular expression matched against object keys in the bucket. Every matching object is read as one dataset, so all matches must share a schema. For example ^exports/orders/.*\.csv$.
​
string
Search Prefix — an S3 key prefix that limits which objects are listed before the pattern is applied, for example exports/orders/. Always set this on a large bucket: without it, every key is listed on every run.
​
string
default:","
Field separator in the files. Use \t for tab-separated exports.
​
string
default:"utf-8"
Character encoding of the files.
​
integer
default:"1000000"
Rows read and written per batch.
​
string
default:"*"
Columns to read, as a comma-separated list.
​
string
Row filter applied to the parsed rows.
In append mode the extract tracks the last-modified timestamp of the objects it has read, so a later run picks up newly written files rather than re-reading the whole prefix.

​
Required Permissions

The AWS IAM user account needs the following permissions on the specified S3 bucket:
  • s3:GetObject - For reading files from the bucket
  • s3:ListBucket - For listing contents of the bucket
  • s3:GetBucketLocation - For determining the bucket’s region

​
Limitations

Be aware of the following limitations when working with AWS S3 CSV sources:
  • CSV, TXT, Parquet, JSON files are supported
  • UTF-8 encoding is recommended
  • Individual file size limits apply based on your AWS S3 configuration
  • The S3 bucket and Datazone instance should ideally be in the same region for optimal performance
  • Cross-region access may incur additional AWS charges

​
Next Steps

After configuring your AWS S3 source:
  1. Create extracts to specify which CSV files to ingest
  2. Configure scheduling for recurring extracts
  3. Integrate the source into your data pipelines
For more information about working with extracts and pipelines, refer to their respective documentation sections.