S3 and httpfs

中文

DuckLocal can read objects from S3 and S3-compatible storage through DuckDB's httpfs extension. Configure it from the S3 button in the title bar.

Configure credentials

The dialog has four fields:

Field	Default	Notes
Endpoint	s3.amazonaws.com	http:// is honored; otherwise HTTPS is used
Region	us-east-1	Sent as typed
Access Key ID	—	Required
Secret Access Key	—	Required, masked, with a reveal toggle

Confirming runs the equivalent of:

INSTALL httpfs; LOAD httpfs;
CREATE OR REPLACE SECRET ducklocal_s3 (
  TYPE s3, PROVIDER config,
  KEY_ID '…', SECRET '…',
  REGION '…', ENDPOINT '…'
);

Credentials are session-only. They are not written to disk and are cleared when you open or switch a database, so you must reconfigure afterwards. There is no separate way to clear credentials other than reconfiguring.

Browse

Once configured, an S3 root appears in the sidebar. Its contents load only when expanded: the root lists buckets, and each bucket or prefix lists its objects. Buckets and prefixes are sorted, and objects show a human-readable size. Use the refresh button to reload a level or retry after an error.

Listing is capped at 1,000 entries per level, even when the response says more are available. There is no deeper paging, no prefix filter box, and no search — a bucket with more than a thousand objects at one level will appear complete when it is not.

Query objects

Clicking an object whose key has a data-file extension fills the active editor with:

SELECT *
FROM 's3://bucket/key'
LIMIT 100;

That is an ordinary query: the read goes through httpfs, so DuckDB's read_parquet / read_csv_auto behavior applies, including pushdown — filters and column selection are pushed into the file read. The query replaces whatever the active tab held.

Only keys recognized as data files are clickable. The recognized extensions are the same as for local files; see Data sources.

Limitations

• Browsing is read-only. The browser has no upload, download-to-file, delete, copy, or bucket-creation action. Queries still download remote data.
• No session tokens. The X-Amz-Security-Token header is never sent, so temporary STS credentials will not work — use long-lived keys, or an endpoint that accepts key/secret alone.
• Endpoint style is mostly automatic. S3 URLs can address a bucket in two ways: virtual-hosted addressing or path style. Virtual-hosted addressing is used when the host contains amazonaws.com, path style otherwise. For self-hosted gateways this is usually what you want; for one that needs the other style, there is no override.
• A wrong region is retried once, using the region DuckDB reports back. If that fails too, the error is shown as-is.
• Network is required: INSTALL httpfs fetches the extension on first use. Browsing sends HTTP or HTTPS requests to the configured endpoint (http:// selects HTTP; otherwise HTTPS is used). These requests send metadata such as bucket names and key prefixes, plus authentication information in signed headers; browsing does not upload local database files or query results. Queries on remote objects download data to your machine through httpfs.

Troubleshooting

Symptom	Likely cause
Browsing fails immediately after configuring	Region or endpoint wrong; check the message shown in the sidebar
Errors on a self-hosted endpoint	Wrong addressing style, or an endpoint that requires a session token
httpfs installation error	No network, or a proxy that blocks the extension download
Credentials seem forgotten	A database was opened or switched, which clears them by design

When DuckLocal is packaged as a signed app, loading the httpfs extension needs the com.apple.security.cs.disable-library-validation entitlement; the shipped packaging script already includes it.