AWS S3

AWS S3 is an object storage service offering industry-leading scalability, data availability, security, and performance. It is read-only and supports below file formats:

The S3 Wrapper allows you to read data of below formats from S3 within your Postgres database.

1. CSV - with or without header line
2. JSON Lines
3. Parquet

The S3 Wrapper also supports below compression algorithms:

1. gzip
2. bzip2
3. xz
4. zlib

Note for CSV and JSONL files: currently all columns in S3 files must be defined in the foreign table and their types must be text type.

Note for Parquet files: the whole Parquet file will be loaded into local memory if it is compressed, so keep the file size as small as possible.

Supported Data Types For Parquet File#

The S3 Wrapper uses Parquet file data types from arrow_array::types, below are their mappings to Postgres data types.

Preparation#

Before you get started, make sure the wrappers extension is installed on your database:

and then create the foreign data wrapper:

Secure your credentials (optional)#

By default, Postgres stores FDW credentials inide pg_catalog.pg_foreign_server in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with Vault, which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials.

Connecting to S3#

We need to provide Postgres with the credentials to connect to S3, and any additional options. We can do this using the create server command:

The full list of options are below:

• aws_access_key_id (required) - Your access key
• aws_secret_access_key (required) - Your secret key
• aws_region (required) - The region of your bucket (if providing an endpoint URL with a region in it, make sure that they are the same)
• endpoint_url (optional) - An optional URL to allow connection to S3-compliant providers (i.e. Wasabi, Cloudflare R2, Backblaze B2, DigitalOcean Spaces)
• path_style_url (optional) - Whether to use path-style URL access. This is required by some S3-compliant providers. true or false, default is false.

Required S3 permissions#

Below S3 permissions are needed:

• s3:GetObject
• s3:GetObjectAttributes

If the bucket is versioned, we also need:

• s3:GetObjectVersion
• s3:GetObjectVersionAttributes

Connecting to S3-compliant Providers - Supabase Storage#

Connecting to S3-compliant Providers - Wasabi#

Creating Foreign Tables#

The S3 Wrapper supports data reads from S3.

For example:

One file in S3 corresponds a foreign table in Postgres. For CSV and JSONL file, all columns must be present in the foreign table and type must be text. You can do custom transformations, like type conversion, by creating a view on top of the foreign table or using a subquery.

For Parquet file, no need to define all columns in the foreign table but column names must match between Parquet file and its foreign table.

Foreign table options#

The full list of foreign table options are below:

• uri - S3 URI, required. For example, s3://bucket/s3_table.csv
• format - File format, required. csv, jsonl, or parquet
• has_header - If the CSV file has header, optional. true or false, default is false
• compress - Compression algorithm, optional. One of gzip, bzip2, xz, zlib, default is no compression

Query Pushdown Support#

This FDW doesn't support query pushdown.

Examples#

Some examples on how to use S3 foreign tables.

Basic example#

This will create some "foreign table" inside your Postgres database can read data from S3:

Read from Supabase Storage#

This example will read a CSV file stored on Supabase Storage. The access information can be found on Supabase Storage settings page.