Commit Graph
6 Commits
Author SHA1 Message Date
Parviz MiriyevandGitHub 2212cc8a5f shell: expose retention flags on mq.topic.configure (#9416)
* shell: expose retention flags on mq.topic.configure

The ConfigureTopicRequest proto already carries a TopicRetention message
({retention_seconds, enabled}), and GetTopicConfigurationResponse / the
admin UI both surface it — but the `mq.topic.configure` shell command
sent Retention: nil unconditionally, leaving CLI and IaC users with no
way to set retention without going through the admin UI. Add three
flags so the shell matches the existing surface:

  -retention <duration>     Go duration string (e.g. 168h, 30m)
  -retentionSeconds <int>   raw seconds (mutually exclusive with -retention)
  -retentionEnabled         bool toggle for retention enforcement

Behavior:
- If none of the retention flags are set, the request omits Retention
  (`nil`) — preserving the prior "leave server-side state alone"
  semantics for callers that only care about partition count.
- Setting any retention flag populates TopicRetention with both fields
  so existing operators keep working when only one is provided.
- -retention and -retentionSeconds together is a hard error (ambiguous);
  negative values are rejected.

The new behavior is detected via flag.FlagSet.Visit so default values
of 0 / false are distinguishable from "not provided".

Help text updated with examples for both setting a TTL and disabling
retention on an existing topic. No proto / server changes needed; this
is a CLI-only patch.

* broker, shell: address review for mq.topic.configure retention

Three follow-up fixes for the review comments on the original commit:

1. Server-side: detect retention changes in the early-return path.
   Previously ConfigureTopic returned without persisting when
   partitionCount + schema were unchanged, even if the request supplied
   a different Retention. Now the early-return branch checks both
   schema and retention, persists either or both when they differ, and
   logs which fields actually changed.

2. Server-side: preserve existing retention when request.Retention is
   nil in the fresh-allocation path. Capture the prior resp.Retention
   before overwriting `resp = &mq_pb.ConfigureTopicResponse{}`, and
   only overwrite the new resp.Retention with `request.Retention` when
   the request actually supplies one. Otherwise carry the previous
   retention forward, so partition-count changes (or any path that
   bypasses the early-return branch) don't accidentally clear retention.

3. CLI: switch the `-retention` / `-retentionSeconds` mutual-exclusion
   check from value-based (`!= 0`) to flag.FlagSet.Visit-based, so an
   explicit `-retention=0 -retentionSeconds=N` is also rejected.

4. CLI: when any retention flag is set, fetch the current
   GetTopicConfiguration and fill in the fields the user didn't
   explicitly provide. This means `-retentionEnabled` alone no longer
   zeros the existing duration, and `-retention 24h` alone preserves
   the existing enabled flag. When the topic doesn't exist yet, the
   GetTopicConfiguration error is treated as "no current state" and
   we proceed with just the user-supplied values.

Build / vet / gofmt / `go test ./weed/shell/... ./weed/mq/broker/...`
clean.
2026-05-12 12:37:09 -07:00
9012069bd7 chore: execute goimports to format the code (#7983)
* chore: execute goimports to format the code

Signed-off-by: promalert <promalert@outlook.com>

* goimports -w .

---------

Signed-off-by: promalert <promalert@outlook.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-01-07 13:06:08 -08:00
Chris LuandGitHub cc05874d06 Add message queue agent (#6463)
* scaffold message queue agent

* adjust proto, add mq_agent

* add agent client implementation

* remove unused function

* agent publish server implementation

* adding agent
2025-01-20 22:19:27 -08:00
chrislu ec30a504ba refactor 2024-09-29 10:38:22 -07:00
chrislu 701abbb9df add IsResourceHeavy() to command interface 2024-09-28 20:23:01 -07:00
Chris LuandGitHub 580940bf82 Merge accumulated changes related to message queue (#5098)
* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* tracking topic=>broker

* merge

* comment
2023-12-11 12:05:54 -08:00