mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-26 09:54:47 +00:00
635f69a821
OLake (github.com/datazip-inc/olake) is a CDC and ingestion engine that writes Apache Iceberg. It covers two paths none of the existing catalog suites reach. It is a strict Java Iceberg client. OLake does not write Iceberg from Go — its Go process spawns a Java sidecar over gRPC and writes through the official Apache Iceberg library, because the Go library has no equality deletes and CDC needs them. That makes it the client class weed/s3api/iceberg's metadata compliance backfill exists to serve: the one that fails with "Cannot parse missing long current-snapshot-id" when spec-required keys are omitted. And it produces equality deletes. Its upsert path commits operation=overwrite with an equality-delete file and a delete manifest. ClickHouse, Doris, Trino, Spark and DuckDB all only append, so nothing else in this directory exercises a delete manifest at all. Six subtests: the destination check reaches SUCCEEDED and actually loads the REST catalog; discover enumerates the source; a full sync commits a snapshot; PyIceberg reads back what the Java writer committed; an update plus a re-sync records an overwrite carrying equality deletes with a delete manifest in the current snapshot; and the catalog does not rewrite manifests the official Java writer produced, which gives the manifest-repair path a negative test to go with the ClickHouse positive one. What it deliberately does not assert is a delete-applied read. PyIceberg refuses to scan a table carrying equality deletes (apache/iceberg#6568) while reading its metadata fine, and an engine that can apply them costs a multi-gigabyte image. Recording the commit correctly is the catalog's contract; applying deletes on read is the engine's. The README says so, says the read half was verified by hand once with StarRocks, and warns against later "upgrading" this to a PyIceberg rows read — which would either fail or, if PyIceberg ever starts skipping deletes instead of raising, pass by not looking. The workflow job asserts the suite actually ran rather than trusting a green exit, for the same reason: at least one top-level PASS and zero SKIP. No product change — the destination config is the generic catalog_type=rest with the standard OAuth2 client-credentials flow, and s3_path_style is not even set, since OLake turns it on itself whenever s3_endpoint is non-empty. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>