test: drive the Lance namespace with Spark (#10864)

* test: drive the Lance namespace with Spark

The counterpart of catalog_spark, which does this for the Iceberg REST
catalog. Spark is the engine most likely to be pointed at a lakehouse,
and it reaches the Lance catalog through the connector's DSV2 catalog -
org.lance.spark.LanceNamespaceSparkCatalog with impl=rest - over the same
routes every other client uses.

    SHOW NAMESPACES -> ['`sparklance-lcephd80`.ml']
    SHOW TABLES -> ['sparklance-lcephd80$ml$embeddings']
    count -> 3
    filtered -> [(2, 'two'), (3, 'three')]
    count after a second commit -> 4

The second insert is there on purpose: a store that cannot order commits
fails on the second one, not the first.

Two things the run settled that were guesses beforehand. CREATE TABLE
works, because the connector declares through the namespace and writes the
data itself rather than pushing Arrow at the server. And SHOW TABLES
returns the namespace's own identifiers - bucket, namespace and name
joined by the delimiter - not bare Spark table names.

Credentials go under the catalog's storage.* prefix, which is handed to
lance as object_store options; a gateway without STS vends none, the same
trap the LanceDB suite documents.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* test: verify the Lance table bucket was actually created

weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame Spark for a bucket that was never made.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* test: bound the Docker probe

An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* test: keep the ivy cache under the user's cache directory

It is mounted into a container running as root, so a shared temp path lets
another local user pre-create it and choose what Spark loads.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* test: assert the vector column's type, not only its name

A column that came back as array<double> or array<string> would still be
called vector and still pass.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* test: read the dataset off its location for real

The catalog being optional is the property that lets duckdb and pandas read
these tables; it was asserted in a comment and printed, never exercised.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* test: do not persist credentials in the Spark Lance checkout

The job only uploads a log on failure; nothing in it pushes.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

* test: say the hosts in the README are placeholders

The suite passes dynamically allocated host.docker.internal ports.

Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm

.github/workflows/s3-tables-tests.yml

@@ -1088,6 +1088,80 @@ jobs:
           path: test/s3tables/catalog_duckdb_lance/test-output.log
           retention-days: 3
 
+  spark-lance-namespace-tests:
+    name: Spark Lance Namespace Integration Tests
+    runs-on: ubuntu-22.04
+    timeout-minutes: 40
+
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v7
+        with:
+          # The job uploads a test log on failure; nothing here needs to push,
+          # so do not leave a token in the checkout for it to pick up.
+          persist-credentials: false
+
+      - name: Set up Go
+        uses: actions/setup-go@v7
+        with:
+          go-version-file: 'go.mod'
+        id: go
+
+      - name: Configure Docker Hub mirror
+        run: |
+          echo '{"registry-mirrors": ["https://mirror.gcr.io"]}' | sudo tee /etc/docker/daemon.json
+          sudo systemctl restart docker
+
+      - name: Pre-pull images
+        run: |
+          pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
+          pull apache/spark:3.5.1
+
+      - name: Run go mod tidy
+        run: go mod tidy
+
+      - name: Build SeaweedFS
+        run: |
+          cd weed && go build -buildvcs=false .
+
+      - name: Run Spark Lance Namespace Integration Tests
+        timeout-minutes: 35
+        working-directory: test/s3tables/catalog_spark_lance
+        run: |
+          set -x
+          set -o pipefail
+          echo "=== System Information ==="
+          uname -a
+          free -h
+          df -h
+          docker info
+          echo "=== Starting Spark Lance Namespace Tests ==="
+
+          go test -v -timeout 30m . 2>&1 | tee test-output.log || {
+            echo "Spark Lance namespace integration tests failed"
+            exit 1
+          }
+
+      - name: Show test output on failure
+        if: failure()
+        working-directory: test/s3tables/catalog_spark_lance
+        run: |
+          echo "=== Test Output ==="
+          if [ -f test-output.log ]; then
+            tail -200 test-output.log
+          fi
+
+          echo "=== Process information ==="
+          ps aux | grep -E "(weed|test|docker|spark)" || true
+
+      - name: Upload test logs on failure
+        if: failure()
+        uses: actions/upload-artifact@v7
+        with:
+          name: spark-lance-namespace-test-logs
+          path: test/s3tables/catalog_spark_lance/test-output.log
+          retention-days: 3
+
   s3-tables-build-verification:
     name: S3 Tables Build Verification
     runs-on: ubuntu-22.04