Commit Graph

3 Commits

Author SHA1 Message Date
Dawid Pawlik
c2d27d1a50 index: remove Chinese, Japanese, and Korean language analyzers
Remove "chinese", "japanese", and "korean" from the list of accepted
full-text search analyzer options. Exposing these options commits
ScyllaDB to supporting them long-term — if we ever switch from one
backend search engine to another, CJK analyzers are the most likely
to lose out-of-the-box support, unlike the popular European languages
that are broadly available across text analysis libraries.

Restrict the accepted set now, while FTS is still new, to avoid a
future compatibility burden.

Add a test to check if the CJK language analyzer options are rejected.

Fixes: VECTOR-672

Closes scylladb/scylladb#29877
2026-05-18 18:20:47 +03:00
Dawid Pawlik
2076164af9 index: unify custom index description
Move common description logic into a protected helper
`describe_with_target` on `custom_index`, so subclasses can delegate
to it when implementing the `describe()` virtual method.
2026-05-08 11:30:08 +02:00
Dawid Pawlik
fcd15b5cd4 index: add fulltext_index custom index implementation
Introduce `fulltext_index`, a new `custom_index` subclass
for full-text search (FTS).

The index validates that the target column is a text type
(text, varchar, or ascii) and supports two WITH OPTIONS keys:
- 'analyzer': one of standard, english, german, french, spanish,
  italian, portuguese, russian, chinese, japanese, korean, simple,
  whitespace
- 'positions': boolean controlling whether term positions are stored

`view_should_exist()` returns false — no backing materialized view is
created, matching the CDC-backed pattern used by `vector_index`.

Fixes: SCYLLADB-1517
2026-05-08 11:30:08 +02:00