scylladb

Author SHA1 Message Date

Author	SHA1	Message	Date
copilot-swe-agent[bot]	7ebf5a02d1	Fix variable quoting and profile path in optimized_clang.sh Co-authored-by: mykaul <4655593+mykaul@users.noreply.github.com>	2025-12-14 15:57:50 +00:00
copilot-swe-agent[bot]	03daab52ec	Initial plan	2025-12-14 15:55:30 +00:00
Botond Dénes	7e7e378a4b	Merge 'Revert "Merge 'Add option to use sstable identifier in snapshot' from Benny Halevy"' from null Reverts commit `8192f45e84`. The merge exposed a critical bug where truncate operations during table drop with auto-snapshot fail, causing Raft applier fiber to stop with unhandled exceptions. This leads to schema inconsistencies across nodes and test failures with "Keyspace does not exist" errors. Root Cause Commit `19b6207f` modified `truncate_table_on_all_shards` to set `use_sstable_identifier = true`: ```cpp // Before (working) co_await table::snapshot_on_all_shards(sharded_db, table_shards, name); // After (broken) auto opts = db::snapshot_options{.use_sstable_identifier = true}; co_await table::snapshot_on_all_shards(sharded_db, table_shards, name, opts); ``` This triggers exceptions during snapshot that propagate through Raft state machine, causing: - Raft applier stops: `raft::state_machine_error` at `raft/server.cc:1369` - Schema changes fail to propagate - Nodes report non-existent keyspaces for valid schemas Changes Reverts 15 files (200 deletions, 74 insertions): - Removes `use_sstable_identifier` from truncate/snapshot code paths - Reverts `snapshot_options` struct back to simple `skip_flush` boolean - Removes REST API and nodetool `--use-sstable-identifier` parameter - Removes feature tests from `test/boost/database_test.cc` No backport required - the original feature was merged to master only and never released. <!-- START COPILOT ORIGINAL PROMPT --> <details> <summary>Original prompt</summary> ---- This section details on the original issue you should resolve <issue_title>test_table_drop_with_auto_snapshot failed with InvalidRequest</issue_title> <issue_description>Seen in: https://jenkins.scylladb.com/job/scylla-master/job/next/9968//testReport Logs: [download](https://downloads.scylladb.com/unstable/scylla/master/testLogs/2025-12-08T15:05:42Z/) Error message: ``` cassandra.InvalidRequest: Error from server: code=2200 [Invalid query] message="Keyspace test does not exist" ``` Stacktrace: ``` manager = <test.pylib.manager_client.ManagerClient object at 0xffff734e6c10> @pytest.mark.asyncio async def test_table_drop_with_auto_snapshot(manager: ManagerClient): logger.info("Bootstrapping cluster") cfg = { 'auto_snapshot': True } servers = await manager.servers_add(3, config = cfg) cql = manager.get_cql() # Increases the chance of tablet migration concurrent with schema change await inject_error_on(manager, "tablet_allocator_shuffle", servers) for i in range(3): await cql.run_async("DROP KEYSPACE IF EXISTS test;") await cql.run_async("CREATE KEYSPACE IF NOT EXISTS test WITH replication = {'class': 'NetworkTopologyStrategy', 'replication_factor': 1} AND tablets = {'initial': 8 };") await cql.run_async("CREATE TABLE IF NOT EXISTS test.tbl_sample_kv (id int, value text, PRIMARY KEY (id));") > await cql.run_async("INSERT INTO test.tbl_sample_kv (id, value) VALUES (1, 'ala');") E cassandra.InvalidRequest: Error from server: code=2200 [Invalid query] message="Keyspace test does not exist" test/cluster/test_tablets2.py:173: InvalidRequest ``` </issue_description> <agent_instructions>this issue was exposed by commit `8192f45e84`, please send a pull request reverting that merge commit and mark it as fixing this github issue.</agent_instructions> <comments> <comment_new><author>@yaronkaikov</author><body> @denesb is this something in your team area? if not , please feel free to delegate it or un-assign yourself :-)</body></comment_new> <comment_new><author>@nyh</author><body> This is very strange. Clearly the keyspace `test` does exist at this point, because we created it two lines above and also we ran `CREATE TABLE .. test.tbl_sample_kv` which would have failed if the keyspace `test` didn't exist - so it must exit, no? In the past, we had a bug where the running `CREATE KEYSPACE IF NOT EXISTS` forgot to set the "schema modified" event in the response so it failed to wait for schema agreement, but 1. we fixed this bug (https://github.com/scylladb/scylladb/pull/18819 by @nuivall ) and 2. this bug didn't happen in this case, where CREATE TABLE deed had work to do. But I just realized something... Our fix in https://github.com/scylladb/scylladb/pull/18819 only applies to CREATE KEYSPACE / TABLE / VIEW / TYPE statements. It wasn't applied to `DROP KEYSPACE` - and it should have been.... But I don't have a good theory how a bug like https://github.com/scylladb/scylladb/pull/18819 can explain this specific test failure. Different schema operations are already linearized, so if a `CREATE TABLE test.tbl_sample_kv` succeeded, I don't see how there could possibly be any earlier `DROP KEYSPACE test` that suddenly springs to life. Unless we have a serious bug in our raft-based schema operations.</body></comment_new> <comment_new><author>@nyh</author><body> Another bug we could have in theory is that the Python driver's async `cql.run_async` might have a bug where it is not waiting for the schema agreement despite being told to wait. If it doesn't wait for schema agreement, this can easily explain this bug: 1. the CREATE KEYSPACE, CREATE TABLE both are sent to node A, but 2. the last INSERT INTO is sent to node B which is not yet aware of this new keyspace and table, and fails. Copilot claims that execute_async() does have this bug! > For schema-altering statements, schema agreement (meaning all nodes agree on the new schema) is important before running follow-up operations, but this is enforced only by synchronous helpers like Session.execute(), not the asynchronous version. > If you use execute_async() for schema operations, you are responsible for checking schema agreement yourself, using [Session.check_schema_agreement()](https://docs.datastax.com/en/developer/python-driver/latest/api/cassandra/cluster/#cassandra.cluster.Session.check_schema_agreement) or (in newer code) ResponseFuture.check_schema_agreement. > According to [a discussion on the DataStax support forum](https://support.datastax.com/s/article/Does-the-Python-Driver-for-Cassandra-Wait-for-Schema-Agreement-after-a-Schema-Change?language=en_US) and the [driver’s source code](`7f12a5e1c6/cassandra/cluster.py (L487)`), schema agreement is not ch... </details> <!-- START COPILOT CODING AGENT SUFFIX --> - Fixes scylladb/scylladb#27501 <!-- START COPILOT CODING AGENT TIPS --> --- Closes scylladb/scylladb#27604 * github.com:scylladb/scylladb: Revert "Merge 'Add option to use sstable identifier in snapshot' from Benny Halevy" Initial plan	2025-12-12 13:20:49 +02:00

copilot-swe-agent[bot]

7ebf5a02d1

Fix variable quoting and profile path in optimized_clang.sh

Co-authored-by: mykaul <4655593+mykaul@users.noreply.github.com>

2025-12-14 15:57:50 +00:00

copilot-swe-agent[bot]

03daab52ec

Initial plan

2025-12-14 15:55:30 +00:00

Botond Dénes

7e7e378a4b

Merge 'Revert "Merge 'Add option to use sstable identifier in snapshot' from Benny Halevy"' from null

Reverts commit 8192f45e84.

The merge exposed a critical bug where truncate operations during table drop with auto-snapshot fail, causing Raft applier fiber to stop with unhandled exceptions. This leads to schema inconsistencies across nodes and test failures with "Keyspace does not exist" errors.

**Root Cause**

Commit 19b6207f modified `truncate_table_on_all_shards` to set `use_sstable_identifier = true`:

```cpp
// Before (working)
co_await table::snapshot_on_all_shards(sharded_db, table_shards, name);

// After (broken)
auto opts = db::snapshot_options{.use_sstable_identifier = true};
co_await table::snapshot_on_all_shards(sharded_db, table_shards, name, opts);
```

This triggers exceptions during snapshot that propagate through Raft state machine, causing:
- Raft applier stops: `raft::state_machine_error` at `raft/server.cc:1369`
- Schema changes fail to propagate
- Nodes report non-existent keyspaces for valid schemas

**Changes**

Reverts 15 files (200 deletions, 74 insertions):
- Removes `use_sstable_identifier` from truncate/snapshot code paths
- Reverts `snapshot_options` struct back to simple `skip_flush` boolean
- Removes REST API and nodetool `--use-sstable-identifier` parameter
- Removes feature tests from `test/boost/database_test.cc`

No backport required - the original feature was merged to master only and never released.

<!-- START COPILOT ORIGINAL PROMPT -->

<details>

<summary>Original prompt</summary>

----

*This section details on the original issue you should resolve*

<issue_title>test_table_drop_with_auto_snapshot failed with InvalidRequest</issue_title>
<issue_description>Seen in: https://jenkins.scylladb.com/job/scylla-master/job/next/9968//testReport
Logs: [download](https://downloads.scylladb.com/unstable/scylla/master/testLogs/2025-12-08T15:05:42Z/)

Error message:
```
cassandra.InvalidRequest: Error from server: code=2200 [Invalid query] message="Keyspace test does not exist"
```

Stacktrace:
```
manager = <test.pylib.manager_client.ManagerClient object at 0xffff734e6c10>

    @pytest.mark.asyncio
    async def test_table_drop_with_auto_snapshot(manager: ManagerClient):
        logger.info("Bootstrapping cluster")
        cfg = { 'auto_snapshot': True }
        servers = await manager.servers_add(3, config = cfg)

        cql = manager.get_cql()

        # Increases the chance of tablet migration concurrent with schema change
        await inject_error_on(manager, "tablet_allocator_shuffle", servers)

        for i in range(3):
            await cql.run_async("DROP KEYSPACE IF EXISTS test;")
            await cql.run_async("CREATE KEYSPACE IF NOT EXISTS test WITH replication = {'class': 'NetworkTopologyStrategy', 'replication_factor': 1} AND tablets = {'initial': 8 };")
            await cql.run_async("CREATE TABLE IF NOT EXISTS test.tbl_sample_kv (id int, value text, PRIMARY KEY (id));")
>           await cql.run_async("INSERT INTO test.tbl_sample_kv (id, value) VALUES (1, 'ala');")
E           cassandra.InvalidRequest: Error from server: code=2200 [Invalid query] message="Keyspace test does not exist"

test/cluster/test_tablets2.py:173: InvalidRequest
```
</issue_description>

<agent_instructions>this issue was exposed by commit 8192f45e84, please send a pull request reverting that merge commit and mark it as fixing this github issue.</agent_instructions>

<comments>
<comment_new><author>@yaronkaikov</author><body>
@denesb is this something in your team area? if not , please feel free to delegate it or un-assign yourself :-)</body></comment_new>
<comment_new><author>@nyh</author><body>
This is very strange. Clearly the keyspace `test` does exist at this point, because we created it two lines above and also we ran `CREATE TABLE .. test.tbl_sample_kv` which would have failed if the keyspace `test` didn't exist - so it must exit, no?

In the past, we had a bug where the running `CREATE KEYSPACE IF NOT EXISTS` forgot to set the "schema modified" event in the response so it failed to wait for schema agreement, but 1. we fixed this bug (https://github.com/scylladb/scylladb/pull/18819 by @nuivall ) and 2. this bug didn't happen in this case, where CREATE TABLE deed had work to do.

But I just realized something... Our fix in https://github.com/scylladb/scylladb/pull/18819 only applies to CREATE KEYSPACE / TABLE / VIEW / TYPE statements. It wasn't applied to `DROP KEYSPACE` - and it should have been....

But I don't have a good theory how a bug like https://github.com/scylladb/scylladb/pull/18819 can explain this specific test failure. Different schema operations are already linearized, so if a `CREATE TABLE test.tbl_sample_kv` succeeded, I don't see how there could possibly be any earlier `DROP KEYSPACE test` that suddenly springs to life. Unless we have a serious bug in our raft-based schema operations.</body></comment_new>
<comment_new><author>@nyh</author><body>
Another bug we could have in theory is that the Python driver's async `cql.run_async` might have a bug where it is not waiting for the schema agreement despite being told to wait. If it doesn't wait for schema agreement, this can easily explain this bug:
1. the CREATE KEYSPACE, CREATE TABLE both are sent to node A, but
2. the last INSERT INTO is sent to node B which is not yet aware of this new keyspace and table, and fails.

Copilot claims that **execute_async() does have this bug!**

> For schema-altering statements, schema agreement (meaning all nodes agree on the new schema) is important before running follow-up operations, but this is enforced only by synchronous helpers like Session.execute(), not the asynchronous version.
> If you use execute_async() for schema operations, you are responsible for checking schema agreement yourself, using [Session.check_schema_agreement()](https://docs.datastax.com/en/developer/python-driver/latest/api/cassandra/cluster/#cassandra.cluster.Session.check_schema_agreement) or (in newer code) ResponseFuture.check_schema_agreement.
> According to [a discussion on the DataStax support forum](https://support.datastax.com/s/article/Does-the-Python-Driver-for-Cassandra-Wait-for-Schema-Agreement-after-a-Schema-Change?language=en_US) and the [driver’s source code](7f12a5e1c6/cassandra/cluster.py (L487)), schema agreement is not ch...

</details>

<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes scylladb/scylladb#27501

<!-- START COPILOT CODING AGENT TIPS -->
---

Closes scylladb/scylladb#27604

* github.com:scylladb/scylladb:
  Revert "Merge 'Add option to use sstable identifier in snapshot' from Benny Halevy"
  Initial plan

2025-12-12 13:20:49 +02:00

1 changed files with 2 additions and 2 deletions

									
										4

tools/toolchain/optimized_clang.sh
									
												View File
												
				@@ -123,7 +123,7 @@ _get_distribution_components() {

				                continue

				                ;;

				        esac

				        echo $target

				        echo "$target"

				    done

				}

				@@ -164,7 +164,7 @@ if [[ "${CLANG_BUILD}" = "INSTALL" ]]; then

				    echo "[clang-stage3] build the compiler applied CSPGO profile"

				    cd "${CLANG_BUILD_DIR}"

				    llvm-profdata merge build/csprofiles/default_*.profraw -output=csir.prof

				    llvm-profdata merge build/profiles/csir-*.profraw -output=csir.prof

				    llvm-profdata merge ir.prof csir.prof -output=combined.prof

				    rm -rf build

				    # linker flags are needed for BOLT

Compare commits

3 Commits

copilot/fi ... copilot/fi

4

tools/toolchain/optimized_clang.sh

View File

Compare commits

3 Commits copilot/fi ... copilot/fi

4 tools/toolchain/optimized_clang.sh Unescape Escape View File

3 Commits

copilot/fi ... copilot/fi

4

tools/toolchain/optimized_clang.sh

View File