mirror of
https://github.com/scylladb/scylladb.git
synced 2026-04-22 01:20:39 +00:00
3b20407f25
Currently, applying schema mutations involves flushing all schema tables so that on restart commit log replay is performed on top of latest schema (for correctness). The downside is that schema merge is very sensitive to fdatasync latency. Flushing a single memtable involves many syncs, and we flush several of them. It was observed to take as long as 30 seconds on GCE disks under some conditions. This patch changes the schema merge to rely on a separate commit log to replay the mutations on restart. This way it doesn't have to wait for memtables to be flushed. It has to wait for the commitlog to be synced, but this cost is well amortized. We put the mutations into a separate commit log so that schema can be recovered before replaying user mutations. This is necessary because regular writes have a dependency on schema version, and replaying on top of latest schema satisfies all dependencies. Without this, we could get loss of writes if we replay a write which depends on the latest schema on top of old schema. Also, if we have a separate commit log for schema we can delay schema parsing for after the replay and avoid complexity of recognizing schema transactions in the log and invoking the schema merge logic. I reproduced bad behavior locally on my machine with a tired (high latency) SSD disk, load driver remote. Under high load, I saw table alter (server-side part) taking up to 10 seconds before. After the patch, it takes up to 200 ms (50:1 improvement). Without load, it is 300ms vs 50ms. Fixes #8272 Fixes #8309 Fixes #1459 Closes #10333 * github.com:scylladb/scylla: config: Introduce force_schema_commit_log option config: Introduce unsafe_ignore_truncation_record db: Avoid memtable flush latency on schema merge db: Allow splitting initiatlization of system tables db: Flush system.scylla_local on change migration_manager: Do not drop system.IndexInfo on keyspace drop Introduce SCHEMA_COMMITLOG cluster feature frozen_mutation: Introduce freeze/unfreeze helpers for vectors of mutations db/commitlog: Improve error messages in case of unknown column mapping db/commitlog: Fix error format string to print the version db: Introduce multi-table atomic apply()