mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-07-20 23:12:36 +00:00

Files

T

History

Chris Lu 1de741737d test(s3tables): add Apache Doris Iceberg catalog integration test (#9307 )

* test(s3tables): add Apache Doris Iceberg catalog integration test

Adds an end-to-end smoke test that boots the apache/doris all-in-one
container, registers SeaweedFS as an external Iceberg REST catalog
(OAuth2 client_credentials), and validates metadata visibility plus the
parquet read path against tables seeded via the Iceberg REST API and a
PyIceberg writer container, mirroring the existing Trino, Spark, and
Dremio coverage. Wires the test into a new s3-tables-tests workflow job.

* test(s3tables): document weed shell -master flag format and fill in helper docstrings

Restores the explanatory comment on createTableBucket about the
host:port.grpcPort ServerAddress format used by `weed shell -master`
(produced by pb.NewServerAddress) so the dot separator isn't mistaken
for a typo, and adds doc comments for createIcebergNamespace,
createIcebergTable, doIcebergJSONRequest, requireDorisRuntime, and
hasDocker.

2026-05-04 00:28:30 -07:00

append_rows.py

test(s3tables): add Apache Doris Iceberg catalog integration test (#9307 )

2026-05-04 00:28:30 -07:00

Dockerfile.writer

test(s3tables): add Apache Doris Iceberg catalog integration test (#9307 )

2026-05-04 00:28:30 -07:00

doris_catalog_test.go

test(s3tables): add Apache Doris Iceberg catalog integration test (#9307 )

2026-05-04 00:28:30 -07:00

README.md

test(s3tables): add Apache Doris Iceberg catalog integration test (#9307 )

2026-05-04 00:28:30 -07:00

README.md

Apache Doris Iceberg Catalog Integration Test

This directory contains an Apache Doris integration smoke test for SeaweedFS's Iceberg REST Catalog implementation.

What It Tests

TestDorisIcebergCatalog verifies the Doris path end to end:

Starts a local SeaweedFS mini cluster with S3 Tables and Iceberg REST enabled.
Creates a SeaweedFS table bucket.
Creates an Iceberg namespace and an empty table through the SeaweedFS REST catalog OAuth flow.
Creates a second table and populates it with three rows by running a PyIceberg writer container (Dockerfile.writer + append_rows.py) before Doris bootstraps, so the snapshot is part of Doris's first scan.
Starts apache/doris:doris-all-in-one-2.1.0 and waits for the FE MySQL query port and at least one alive BE.
Connects to Doris over the MySQL protocol and registers a Doris EXTERNAL CATALOG of type=iceberg and iceberg.catalog.type=rest that points at the SeaweedFS REST endpoint, using credential = key:secret for the OAuth2 client-credentials flow.
Runs subtests against the SeaweedFS-backed Iceberg tables:
- BasicSelect: Doris is alive and answering SQL.
- CatalogVisible: SHOW CATALOGS lists the SeaweedFS-backed catalog.
- DatabaseVisible: the seeded namespace is exposed as a Doris database.
- TableVisible: the seeded table appears under SHOW TABLES FROM ....
- CountEmptyTable: catalog-to-table resolution and a scan of an empty table.
- ColumnProjection: SELECT id, label succeeds and the response columns are id, label. Failure here means Doris could not parse the schema returned by the SeaweedFS catalog.
- ReadWrittenDataCount and ReadWrittenDataValues: Doris reads back the three PyIceberg-appended rows and the values match. This exercises the actual data path (parquet reads via S3), not just metadata.

The PyIceberg writer image is built on demand via Docker layer caching. The first build pulls python:3.11-slim and pip-installs PyIceberg + PyArrow (~1-2 min in CI); subsequent invocations are cheap.

Running Locally

Build or install weed, then run:

cd test/s3tables/catalog_doris
go test -v -timeout 25m .

The test requires Docker. The GitHub Actions job runs on ubuntu-22.04 and executes the test for pull requests.

Configuration

The test uses these fixed credentials for the local SeaweedFS IAM config:

S3 access key: AKIAIOSFODNN7EXAMPLE
S3 secret key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Region: us-west-2
Warehouse bucket: iceberg-tables

Doris ports:

The all-in-one container exposes 9030 (FE MySQL/SQL) and 8030 (FE HTTP). Only 9030 is mapped to a host port (allocated dynamically) so the Go MySQL client can connect from the test process.
The Iceberg REST endpoint and the S3 endpoint are reached from inside the Doris container via host.docker.internal, matching the Trino and Dremio test paths.

Troubleshooting

Ensure Docker is running: docker version
Ensure weed is built or available on PATH
The Doris all-in-one image is large (~1 GB) and the FE+BE need ~30-60 seconds to register before queries succeed; the test waits up to 4 minutes for the query port and 60 seconds for at least one alive BE.
If queries hang on metadata, run REFRESH CATALOG iceberg_catalog from a Doris MySQL client and retry.
Container logs are printed in the failure message; you can also check docker logs <seaweed-doris-...> while the test is running.