Chris LuandGitHub
0dfaa103d0
* lance worker: share the integration tests' scaffolding The recorder that keeps what a handler sent, the config builder and the storage-option fallback all lived inside compaction.rs, so a second test binary would have had to copy them. They move to tests/common. The fallback now reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_ENDPOINT_URL from the environment, defaulting to what it used before. A harness can then point these tests at a gateway that checks what it is given rather than one that accepts anything. * lance worker: maintain one named table, for a harness to drive Compacts and cleans up whatever WEED_LANCE_TABLE names, through the handlers' own detect-then-execute path: a proposal the worker would not have made is not one worth running. The existing tests seed the tables they check. This one deliberately does not, so a harness that has already written a table and knows what is in it can have the real handlers maintain it and then read it back. * test: take a table through its whole life, for Iceberg and Lance Created in the catalog, filled by a real client, maintained by the worker, read again, dropped. The step nothing was checking is the read after maintenance: compaction once rewrote every dictionary-encoded column onto a single value and shipped, because the maintenance tests were thorough about sequence numbers, manifest entries and metadata versions and none of them opened the parquet file the worker had just written. So the assertion is a tally - row count, the cardinality of each dictionary-encoded column, and an md5 over whole rows - taken before maintenance and again after, required to be equal. The cardinalities name the failure that happened; the digest catches a rewrite that keeps every column's cardinality and hands the values to the wrong rows. A compaction that merged nothing fails rather than passes, or the read afterwards is checking a file the worker never wrote. The Iceberg half runs two clients. DuckDB is the one the corruption was reported against and the only one here that writes the deprecated PLAIN_DICTIONARY encoding, which parquet-go normalizes away on write, so a Go writer cannot produce it. PyIceberg writes the modern spelling. Pinning parquet-go back to v0.30.1 fails the DuckDB half and passes the PyIceberg one, which is why both are here. Lance maintenance lives in the Rust worker, so it runs there where cargo is installed and through the two lance calls those handlers wrap where it is not. WEED_LANCE_MAINTENANCE picks one instead of letting the test guess. * ci: run the table lifecycle tests CI maintains the Lance table through the lance library rather than the worker: a cold build of the lance crate costs more than the glue it would be checking, and the worker's own tests cover its handlers. The suite drives the Iceberg maintenance worker, so a change to it now triggers this workflow too. * test: let the lifecycle harness fail instead of skipping Setup failures all exited zero, so a cluster that would not come up, or a port allocation that lost, reported a green run for code nothing had executed. That is the failure mode this whole directory exists to close, and it was in the harness itself. Only a checkout without a weed binary skips now, and it runs the tests so each one says so rather than the package quietly passing. Everything else fails. The filer existence probe gets a deadline while I am here: it ran without one, so an unresponsive filer would hang the suite past every timeout the clients have. * test: make the lifecycle checks check what they claim to Three of them could pass without having looked. The DuckDB skip matched "syntax error", "not implemented" and "Failed to load" anywhere in the output, in any phase. A parse error in the SQL this test generates, or a refusal from our own catalog, would have taken the only coverage of the PLAIN_DICTIONARY encoding out of CI and left it green. It now matches the extension failing to install, and only in the phase that installs it. Everything past LOAD is ours and fails. The digests covered id, category and value. Compaction rewrites the whole row, so a defect confined to ts, or to a Lance vector, changed nothing either side of maintenance. Every persisted column goes in now, ts as microseconds so no timezone sits between the two runs. The Lance drop check caught every exception as proof the dataset was gone. pylance turns credential and transport failures into the same ValueError, so it only accepts the message that means not found. * docs: say up front which maintenance path the Lance half takes The opening summary said the worker maintains both tables. It maintains the Iceberg one always and the Lance one only where cargo is installed, which is not what CI does.
Table Lifecycle Integration Tests
One table, all the way through: created in the catalog, filled by a real client, maintained, read again, dropped. Once for Iceberg and once for Lance. The Iceberg half always maintains through the worker; the Lance half maintains through the Rust worker or through the lance library, depending on what the environment has - see below.
Why this suite exists
#10853 was a compaction that rewrote every dictionary-encoded column onto a single value. It shipped. The maintenance tests we had were thorough about the bookkeeping - sequence numbers, added and deleted manifest entries, metadata versions, the manifest list - and every one of them passed, because not one of them opened the parquet file the worker had just written.
So the assertion here is the dull one nothing else was making: tally the table before maintenance, tally it again after, and require the two to be equal. The tally is a row count, the cardinality of each dictionary-encoded column, and an md5 over whole rows. The cardinalities name the failure that happened; the digest catches a rewrite that keeps every column's cardinality and hands the values to the wrong rows.
The same shape covers Lance, because the exposure is the same: a compaction that merges fragments can hand back a table that reads without complaint and answers wrongly.
What runs
TestIcebergTableLifecycle starts a weed mini cluster, declares an ICEBERG
table bucket, and runs two clients against it:
| Client | Why both |
|---|---|
| DuckDB | the client the bug was reported against, and the only one here that writes the deprecated PLAIN_DICTIONARY encoding - parquet-go normalizes it away on write, so a Go writer cannot produce it |
| PyIceberg | writes RLE_DICTIONARY, the modern spelling, so between the two the merge is checked against both dictionary encodings in the spec |
Between the write and the read, the test runs the worker's whole maintenance cycle in-process against the live filer: compact, expire snapshots, remove orphans, rewrite manifests. A compaction that merged nothing fails the test rather than passing it - otherwise the read afterwards is checking a file the worker never wrote.
TestLanceTableLifecycle does the same against a LANCE bucket: declare
through the namespace, write a fragment per append, maintain, read, drop.
Maintenance goes through the Rust worker's own handlers where cargo is
installed, and through the two lance calls those handlers wrap where it is not.
WEED_LANCE_MAINTENANCE=library|worker picks one instead of letting the test
guess; CI sets library, because a cold build of the lance crate costs more
than the layer it would be checking.
Both tests finish by dropping the table and checking the data actually left the filer, which is the half of a lifecycle a catalog test never reaches.
Running it
cd test/s3tables/lifecycle
(cd ../../../weed && go build .) # the harness runs this binary
go test -v -timeout 40m .
Skipped without Docker and in -short mode. The first run builds the two client
images and pulls duckdb/duckdb:latest; later runs reuse them. The DuckDB half
skips itself, rather than failing, on an image whose iceberg extension cannot
write through a REST catalog.
To watch it catch the bug it was written for, pin parquet-go back to the version that had it:
go mod edit -require=github.com/parquet-go/parquet-go@v0.30.1 && go mod tidy
go test -run TestIcebergTableLifecycle/DuckDB -v .
maintenance collapsed the category column: 7 distinct values -> 1
maintenance collapsed the value column: 13 distinct values -> 1
The PyIceberg half still passes there, which is the reason both clients are in this directory.