Compare commits

...
126 Commits
Author SHA1 Message Date
nelind bfadc936c0 fix(api): dont verify signature or DID during importRepo 2026-06-11 23:23:37 +02:00
nelind 80afd764d3 fix(nix): provide a jemalloc build in the dev shell 2026-06-10 19:56:40 +02:00
LewisandTangled 5bbe2146ff server: serve xrpc over http/3
Lewis: May this revision serve well! <lu5a@proton.me>
2026-06-10 13:18:49 +03:00
LewisandTangled b009ccdaf2 repo: the pg side of MST structural repair
Lewis: May this revision serve well! <lu5a@proton.me>
2026-06-08 16:58:51 +03:00
nelind 39f74b5adf feat(nix): pratice what i preach. expose lib.mkPackages 2026-06-07 22:14:27 +02:00
aveandTangled a231d7da29 feat(auth): accept totp/backup codes in legacy login createSession 2026-06-06 10:42:24 +03:00
aveandTangled 63d84d38fb refactor(auth): unify short-code generation onto util::generate_token_code
Collapse the three ad-hoc short-code generators into one canonical generator
plus a shared normalizer:

- util::generate_token_code now emits the uppercase base32 XXXXX-XXXXX display
  form; new util::normalize_token_code canonicalizes user input (uppercase,
  strip hyphen/whitespace).
- email_token and legacy_2fa now generate via util, store the normalized form,
  and compare normalized input. Their private generate_short_token/generate_code
  (and BASE32_CHARS/CODE_LENGTH) are removed.
- PLC (request/sign) and password reset inline util::generate_token_code,
  persist the normalized form, email the display form, and normalize input
  before lookup. The generate_plc_token/generate_reset_code wrappers are removed.

Behavior changes: legacy login-2FA codes go from 8-digit numeric to XXXXX-XXXXX;
PLC and password-reset codes go from lowercase to uppercase. All four code types
are now accepted case-/hyphen-insensitively. OAuth web-login 2FA, account
deletion, and the long verification_token blobs are intentionally untouched.

Tests: add util normalize tests + email_token/legacy_2fa case/hyphen tests;
update integration tests to expect the canonical stored form and the new
emailed format.
2026-06-06 09:36:08 +03:00
aveandTangled fe9b88141c chore(auth): align at+jwt/refresh+jwt expiry with reference PDS 2026-06-06 01:43:46 +03:00
aveandTangled 72f5dce32b add ave.zone to contributors list in README 2026-06-06 01:42:54 +03:00
nelind cd7e01100e chore(auth): also mention that atproto spec requiers typ be "JWT" for inter-service tokens 2026-06-05 12:49:25 +02:00
aveandTangled 7c248be153 fix(auth): emit uppercase "JWT" typ in service-auth header
RFC 7519 §5.1 recommends the uppercase "JWT" typ for compatibility with
legacy implementations, and it matches the reference @atproto/pds. Parsing
already lowercases, so existing lowercase "jwt" tokens still verify.
2026-06-05 13:34:43 +03:00
aveandTangled 91999819c6 fix(proxy): limit audience of getFeed service-auth to the feed generator 2026-06-04 22:39:14 +03:00
nelindandTangled ffce1d5d05 feat(docs): add some general PDS debugging documentation 2026-06-04 13:45:08 +03:00
TylerandTangled 8e6ace2fe2 fix: derive lexicon DNS authority from all-but-last NSID segment
Permission-set expansion resolved the lexicon's DNS authority using a
fixed `parts[..2]`, which only works for three-segment NSIDs. For a
four-segment NSID such as community.lexicon.bookmarks.authManageBookmarks
this dropped a segment and queried _lexicon.lexicon.community instead of
_lexicon.bookmarks.lexicon.community, failing with "DNS resolution
failed: ... no record found".

The authority is every NSID segment except the last (the name),
reversed. Use parts[..parts.len() - 1] to match the spec and the
existing extract_namespace_authority helper, and update the DNS
authority test with three/four-segment and bookmarks regression cases.
2026-06-04 01:28:13 +03:00
LewisandTangled 3018a20843 fix(plc): allow arbitrary services to sign
Lewis: May this revision serve well! <lu5a@proton.me>
2026-06-03 11:23:51 +03:00
LewisandTangled 37fc06fb39 fix(store): unblock eventlog sync&freeze when writer dies
Lewis: May this revision serve well! <lu5a@proton.me>
2026-06-02 17:29:21 +03:00
LewisandTangled 728a8c4d3b test(store): cross-store, firehose, read-validation coverage w/ faults
Lewis: May this revision serve well! <lu5a@proton.me>
2026-06-02 17:29:21 +03:00
LewisandTangled 3d49e99cc3 test(store): generic consistency checker, gauntlet fault/read
Lewis: May this revision serve well! <lu5a@proton.me>
2026-06-02 17:29:21 +03:00
LewisandTangled 7e823673ca test(store): untested metastore, eventlog, & archival stuff
Lewis: May this revision serve well! <lu5a@proton.me>
2026-06-02 17:29:21 +03:00
Lewis 320933598c fix(auth): error num 401 for oauth
Lewis: May this revision serve well! <lu5a@proton.me>
2026-06-01 19:38:45 +03:00
Lewis 500dc2e0e6 test(store): gauntlet sweep configs for time-travel & fsync repro
Lewis: May this revision serve well! <lu5a@proton.me>
2026-06-01 19:01:04 +03:00
Lewis a220611a8b test(store): D gauntlet faults, crash-loss oracley, recoverable scenarios
Lewis: May this revision serve well! <lu5a@proton.me>
2026-06-01 19:01:04 +03:00
Lewis ca7a4b4b73 fix(store): recover eventlog lastseq from tail not sidecar
Lewis: May this revision serve well! <lu5a@proton.me>
2026-06-01 19:01:04 +03:00
Lewis 8ff02610e4 feat(store): inline-commit mode, committed-extent recovery, failsafe verify&rollback
Lewis: May this revision serve well! <lu5a@proton.me>
2026-06-01 19:01:04 +03:00
LewisandTangled 22f82489d5 test(pds): e2e & durability coverage for MST self-heal
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-31 21:11:36 +03:00
LewisandTangled cee483e358 feat(pds): selfhealing repo writing by detecting corruption & retrying
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-31 21:11:36 +03:00
LewisandTangled 7f8e858137 test(store): gauntlet MST-repairable & misdirected-write scenario
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-31 21:11:36 +03:00
LewisandTangled 44d73dac58 feat(store): rebuild & rewrite missing/corrupt MST blocks
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-31 21:11:36 +03:00
LewisandTangled b8cae15c12 feat(store): detect foreign&corrupt blocks on read & preserve blocks thru recovery
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-31 21:11:36 +03:00
LewisandTangled 9b58961bba feat(repo): missing $type? invent one
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-31 13:02:11 +03:00
Lewis 31ee12ecd3 fix(store): torn hint-file tail should be recoverable on reopen
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-31 11:37:21 +03:00
nelind ea106d5246 chore(nix): update frontend pnpm hash 2026-05-31 00:51:27 +02:00
Lewis 4015217a2e feat(store): Clock trait for DST
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-30 23:46:22 +03:00
Lewis 7e4c9e1dab chore: bump to 0.6.4
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-30 21:23:02 +03:00
LewisandTangled e9dc57d6f4 fix(firehose): lost events if seq commits out of order
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-30 21:10:38 +03:00
Lewis 4e2525b245 fix(migrate): oauth account:* spec, p256 support
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-30 21:07:16 +03:00
nelind 38508c2c6e feat(nix): base the data storage path defaults on the dataDir option 2026-05-30 00:37:55 +02:00
Lewis 7a54ccf6a3 fix(tranquil-config): reject unknown keys in TlsConfig, bump rust 1.96
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-29 09:39:14 +03:00
nelind e85394c314 build: bump version to 0.6.3 2026-05-27 19:28:10 +02:00
isabelandTangled 411c6be108 feat(nix): validate config at build time 2026-05-27 20:14:06 +03:00
isabelandTangled 28aa7ab7fc feat!(tranquil-config): error on unknown keys
an attempt to make it so that fixes like
fc6063dba8 never have to occur again
2026-05-27 20:13:58 +03:00
nelindandTangled 348b69d76b feat(nix): ensure that tranquil-store path is always writable by the service user just like file blob storage 2026-05-27 20:13:07 +03:00
nelindandTangled 4b23ca2c36 fix(docs): some wording on nix config and CI stuffs 2026-05-27 20:13:07 +03:00
LewisandTangled a7052e878c feat: tranquil's own TLS handling
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-27 13:54:33 +03:00
nelind fc6063dba8 fix(nix): override the right frontend package setting in the flake nixos module 2026-05-27 01:04:31 +02:00
LewisandTangled 44cb016762 fix(docs): install docs better
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-26 09:49:59 +03:00
LewisandTangled e6eee18ace fix(docs): migration oauth update
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-25 19:02:13 +03:00
Lewis 96c8375706 fix(test): oauth token eviction should be agnostic
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-25 16:17:10 +03:00
Lewis 86c5995568 fix(config): signal gate is useless since needs admin work to activate anyway
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-24 22:50:14 +03:00
LewisandTangled 036c317fd6 fix(migrate): oauth refreshing, embedded db first invite code
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-24 19:28:57 +03:00
LewisandTangled 4d2c7d4723 feat(auth): verification-gate override, inbound-migration bypass, store deleter improvement
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-23 23:02:43 +03:00
isabelandTangled f24a9f8bc0 build(nix): frontend pin pnpm
some future protections against this acursed toolchain breaking and
people who do .follows on their nixpkgs inputs
2026-05-22 23:24:45 +03:00
Lewis f6ef6ecbd9 feat(docs): first version of 'when PRing' section of contrib
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-21 22:46:59 +03:00
karitham.devandTangled a2567bdb1a fix(locales): french handle should be "identifier" not "username"
My bad translation on that earlier, made it consistent with the rest of the file and usage!
2026-05-21 10:39:29 +03:00
Lewis 56120d252d fix(locale): a little more drift
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-21 09:14:06 +03:00
TrezyandTangled 210b0f463c feat: add About page
Signed-off-by: Trezy <tre@trezy.com>
2026-05-21 09:06:42 +03:00
LewisandTangled c80f504dc0 fix(locale): a couple of drift keys missing in other langs
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-21 09:06:10 +03:00
karitham.devandTangled a3bd7c59ad feat(locales): add french locale, fix missing error keys, remove dead keys
Adds french locale registration and translation file. Fixes 4 missing error keys that were referenced in code but absent from all locale files. Removes 2 unreferenced keys from `register.validation` that duplicated `registerPasskey.errors`.

I *think* that's all that's needed for french locale? Couldn't find anything else required to make it work!
2026-05-20 22:20:20 +03:00
nelind 9ebde27540 fix(docs): markdown is tricky ... 2026-05-19 21:30:35 +02:00
nelindandTangled 28a7834304 docs: nel interjection: be right, be good, be fun 2026-05-19 22:18:38 +03:00
Lewis 191da5b311 chore: slight doc tweak
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-19 21:14:28 +03:00
Lewis a5b4ba7d65 feat(docs): init better docs
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-19 20:45:50 +03:00
TrezyandTangled 90dabd8840 docs: start a contributing file to track how to work with the repo
Signed-off-by: Trezy <tre@trezy.com>
2026-05-19 19:46:58 +03:00
TrezyandTangled 19eaccea74 chore: add dev startup to justfile
Signed-off-by: Trezy <tre@trezy.com>
2026-05-19 19:46:58 +03:00
TrezyandTangled f620a6bc43 chore: add mailpit for local dev
Captures outbound emails, greatly simplifying local dev.

Signed-off-by: Trezy <tre@trezy.com>
2026-05-19 19:46:58 +03:00
TrezyandTangled 04f370aaa1 chore: add frontend service to docker compose
Signed-off-by: Trezy <tre@trezy.com>
2026-05-19 19:46:58 +03:00
TrezyandTangled 3fd8f7ebbf chore: add app service with cargo-watch
Spins up a container with Tranquil running, but automatically uses incremental rebuilds for local dev.

Signed-off-by: Trezy <tre@trezy.com>
2026-05-19 19:46:58 +03:00
TrezyandTangled 0afcb2ee28 chore: add plc directory for local dev
Prevents us from polluting the real PLC directory.

Signed-off-by: Trezy <tre@trezy.com>
2026-05-19 19:46:58 +03:00
TrezyandTangled 8bd556f65b chore: add db health check to docker compose
Safer wafe to depend on database startup.

Signed-off-by: Trezy <tre@trezy.com>
2026-05-19 19:46:58 +03:00
TrezyandTangled cc92594506 chore: add traefik for local dev certs and routing
Signed-off-by: Trezy <tre@trezy.com>
2026-05-19 19:46:58 +03:00
TrezyandTangled 76f22b801b chore: allow native certs during local dev
Allows Traefik certs to be trusted by Tranquil.

Signed-off-by: Trezy <tre@trezy.com>
2026-05-19 19:46:58 +03:00
TrezyandTangled 021b7dbec4 chore: add profiles to docker compose
Allows us to spin up groups from the compose file

Signed-off-by: Trezy <tre@trezy.com>
2026-05-19 19:46:57 +03:00
nelind eb034cb8b3 fix(ci): dont pass cachix cache name as a secret so it doesnt get redacted from CI logs 2026-05-17 17:07:34 +02:00
Lewis bdaf510898 build: bump workspace to 0.6.2
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-17 12:32:05 +03:00
Lewis deb2502112 feat(pds): phantom-file self-heal goes in scheduled compaction + reachability walk
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-17 12:32:05 +03:00
Lewis 1815ddba9f feat(gauntlet): index-backed/hint-backed/readable invariants, ExternalCorruption scenario
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-17 12:32:05 +03:00
Lewis a7517ed5c9 feat(store): consistency check & repair for orphan hints etc
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-17 11:50:32 +03:00
Lewis d07d702dd4 feat(store): try to self-heal phantom index entries on compaction
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-17 11:50:19 +03:00
isabelandTangled 1901b0a630 fix(nix): build against pnpm 11
side note: you may also notice that I have removed the `github:` url for
nixpkgs this is because it save a massive 15mb, is faster and more
reliable than github
See also:
-
<https://nix.dev/manual/nix/stable/protocols/tarball-fetcher#lockable-http-tarball-protocol>
-
<http://web.archive.org/web/20250806225139/https://nix.dev/manual/nix/2.28/protocols/tarball-fetcher#lockable-http-tarball-protocol>
2026-05-14 21:37:28 +03:00
serenityandTangled 58f8d327c1 feat(docs): clarify nix documentation 2026-05-14 16:52:10 +03:00
LewisandTangled a13343e1de fix(oauth): gc tokens in pg in the right order, more exposure of dpop err
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-14 16:51:53 +03:00
Bas van den WollenbergandTangled 60e10af4aa fix(Dockerfile): pnpm v11 build errors
- pnpm doesn't implicitly ignore post-install scripts anymore which means trying to build the docker image now fails. i've explicitly allowed them but idk could also ignore them maybe not sure what they actually do lol

- added `confirmModulesPurge: false` but not sure what it actually does as it's not documented? but either way it's also required for the `pnpm build` step to succeed non-interactively it seems idk weird stuff

- removed msmtp/sendmail while i was at it just like https://tangled.org/tranquil.farm/tranquil-pds/commit/f176f55862bc4c41472f8697cc2ca069c24d5e37
2026-05-14 09:56:07 +03:00
isabelandTangled f176f55862 chore(nix): remove sendmail 2026-05-02 23:32:57 +03:00
LewisandTangled fac9520a16 feat(tranquil-server): email config, tests, fmt
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-02 22:28:59 +03:00
LewisandTangled eee6fb9ff4 feat(comms): EmailSender, permanent/transient routing
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-02 22:28:59 +03:00
LewisandTangled 2462d0ab3b feat(tranquil-comms): smtp and dkim signing
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-02 22:28:59 +03:00
LewisandTangled 85f87f7b28 feat(tranquil-comms): message construction and mx resolution
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-02 22:28:59 +03:00
LewisandTangled b1d86caa78 feat(tranquil-comms): prework for email
Lewis: May this revision serve well! <lu5a@proton.me>
2026-05-02 22:28:59 +03:00
Lewis 9b2cfb3a7e fix(tranquil-store): durable-tail recovery + sync semantics
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-30 11:11:28 +03:00
Lewis efd499bb26 fix(tranquil-store): barrier durability + torn-header recovery
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-29 15:35:42 +03:00
nelindandTangled d4dfe838eb fix(ci): use kaniko to build 2026-04-28 23:06:36 +03:00
LewisandTangled af3821514f test(tranquil-pds): same-rkey batch coverage and inductive inverse for in-batch dups
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-28 22:05:03 +03:00
LewisandTangled 8f7aad3756 fix(tranquil-pds): same-rkey batch semantics and firehose lag recovery
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-28 22:05:03 +03:00
LewisandTangled 75b9e3165f refactor(deploy): container-first cleanup, drop debian malware-style install
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-27 00:13:27 +03:00
LewisandTangled ccc9916109 test(tranquil-pds): websocket firehose end-to-end mst verification
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-26 20:11:27 +03:00
LewisandTangled bc8fd66a45 test(tranquil-pds): mst fuzz + repo integrity properties
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-26 20:11:27 +03:00
LewisandTangled 180de29984 fix(tranquil-pds): firehose car carries inductive proof
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-26 20:11:27 +03:00
max.andTangled 0455dc20bd fix(validation): discord usernames may contain periods at start or end 2026-04-25 09:59:12 +03:00
Lewis 2770b9b14a feat(tranquil-store): cargo-fuzz targets with asan+ubsan
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-24 10:50:08 +03:00
Lewis d436597184 feat(tranquil-store): flaky-device scenario, jemalloc heap-prof
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-24 10:50:08 +03:00
Lewis 4cfca6d956 feat(tranquil-store): soak harness driving leak gate, signal tweaks
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-23 08:39:16 +03:00
Lewis 98b94fb170 feat(tranquil-store): leak gate and metrics sampling for gauntlet
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-22 21:00:17 +03:00
Lewis 4fe01cff72 feat(tranquil-store): sweep subcommand with axis override fan-out
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-22 17:23:15 +03:00
Lewis 00c9eb732f fix(tranquil-store): arc-counted cache handles, reader-eviction race
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-21 22:04:24 +03:00
Lewis 6d2d3b4be4 fix(tranquil-store): commit-marker batch replay, batch-boundary rotation
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-21 14:55:15 +03:00
Lewis 9d81e58803 fix(tranquil-store): no more orphan data files, recover torn-hint tails, header-safe resume
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-21 09:59:37 +03:00
Lewis 2afd075496 fix(tranquil-store): atomic record commits, hint-as-truth recovery
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-20 21:00:54 +03:00
Hayden YoungandTangled c74bf967cf fix(auth): don't require 2fa for app passwords 2026-04-20 18:20:20 +03:00
Lewis 1285d5c675 fix(tranquil-store): bound writer fd usage across rotations
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-20 16:50:19 +03:00
Lewis c30d73cd4d test(tranquil-store): migrate some tests to gauntlet
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-19 23:50:27 +03:00
Lewis 0fab8f2eb9 feat(tranquil-store): tranquil-gauntlet CLI, config overrides, profiles
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-19 23:50:27 +03:00
Lewis 57336fa124 feat(tranquil-store/gauntlet): new invariants & scenarios
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-19 10:25:07 +03:00
Lewis ace105899f feat(tranquil-store/gauntlet): concurrent executor, eventlog, fault recovery
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-19 00:19:08 +03:00
Lewis c80a525e0d feat(tranquil-store/gauntlet): op surface, oracle, workload for eventlog & reads
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-18 10:36:30 +03:00
Lewis 7edb76507b feat(tranquil-store/gauntlet): simulated io fault modes, shrinker, regression dump
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-18 10:36:30 +03:00
LewisandTangled a20e4e05e6 fix(oauth): indigo client send empty assertions
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-17 15:18:26 +03:00
teqandTangled 3e7a199736 docs: nixos and binary cache 2026-04-17 01:32:30 +03:00
teqandTangled 7ed9c8ce59 feat(ci): supply cache by arg 2026-04-17 01:32:30 +03:00
teqandTangled b0c3402a82 feat(ci): add cachix publishing workflows 2026-04-17 01:32:30 +03:00
LewisandTangled b9574f3ec1 fix(lexicons): cache ttl
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-16 19:45:27 +03:00
isabelandTangled baef2be844 feat: allow max prefs to be configured
some people (me) have a large amount of prefs and moving from ref pds to
tranquil ment i lost a few of these. so in this pr i bumped the max
prefs to 1000 and made it configurable via the config
2026-04-16 19:45:17 +03:00
Lewis 1f21b85069 fix(example.toml): lewis forgot about confique
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-16 14:49:57 +03:00
Lewis 09d437b3e3 feat(tranquil-store): gauntlet persistence & restart invariants
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-15 22:28:31 +03:00
Lewis 7f2e83e92f feat(tranquil-store): beginnings of the gauntlet test suite
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-15 20:29:34 +03:00
nelind ec273fa814 feat(nix): cross compiled aarch64 package 2026-04-14 23:26:34 +02:00
Lewis d51bfd59da fix(tranquil-store): exclude 0 refcount blocks from has()
Lewis: May this revision serve well! <lu5a@proton.me>
2026-04-14 08:40:21 +03:00
337 changed files with 39557 additions and 4280 deletions
+58
View File
@@ -25,6 +25,28 @@ fail-fast = false
test-threads = "num-cpus"
slow-timeout = { period = "300s", terminate-after = 2 }
[profile.gauntlet-pr]
retries = 0
fail-fast = true
test-threads = "num-cpus"
slow-timeout = { period = "60s", terminate-after = 5 }
[[profile.gauntlet-pr.overrides]]
filter = "binary(gauntlet_smoke)"
slow-timeout = { period = "300s", terminate-after = 2 }
[profile.gauntlet-nightly]
retries = 0
fail-fast = false
test-threads = "num-cpus"
slow-timeout = { period = "600s", terminate-after = 1 }
[profile.gauntlet-soak]
retries = 0
fail-fast = false
test-threads = 1
slow-timeout = { period = "5m", terminate-after = 1000 }
[test-groups]
serial-env-tests = { max-threads = 1 }
heavy-load-tests = { max-threads = 4 }
@@ -46,6 +68,10 @@ test-group = "serial-env-tests"
filter = "package(tranquil-signal)"
test-group = "serial-env-tests"
[[profile.default.overrides]]
filter = "package(tranquil-config)"
test-group = "serial-env-tests"
[[profile.default.overrides]]
filter = "binary(whole_story)"
test-group = "heavy-load-tests"
@@ -72,6 +98,24 @@ test-group = "io-heavy-sim"
filter = "test(/test_scale_/) | test(/full_backup_and_restore/)"
slow-timeout = { period = "120s", terminate-after = 4 }
[[profile.default.overrides]]
filter = "binary(gauntlet_smoke)"
slow-timeout = { period = "300s", terminate-after = 8 }
[[profile.default.overrides]]
filter = "binary(compaction_restart) | binary(mst_refcount_integrity) | binary(gc_compaction_restart)"
slow-timeout = { period = "120s", terminate-after = 4 }
[[profile.default.overrides]]
filter = "test(/retention_time_travel_survives_many_seeds/)"
slow-timeout = { period = "300s", terminate-after = 24 }
test-group = "io-heavy-sim"
[[profile.default.overrides]]
filter = "binary(fd_lifecycle)"
slow-timeout = { period = "300s", terminate-after = 4 }
test-group = "io-heavy-sim"
[[profile.ci.overrides]]
filter = "test(/import_with_verification/) | test(/plc_migration/)"
test-group = "serial-env-tests"
@@ -88,6 +132,10 @@ test-group = "serial-env-tests"
filter = "package(tranquil-signal)"
test-group = "serial-env-tests"
[[profile.ci.overrides]]
filter = "package(tranquil-config)"
test-group = "serial-env-tests"
[[profile.ci.overrides]]
filter = "binary(whole_story)"
test-group = "heavy-load-tests"
@@ -99,3 +147,13 @@ test-group = "heavy-load-tests"
[[profile.ci.overrides]]
filter = "binary(repo_lifecycle)"
test-group = "heavy-load-tests"
[[profile.ci.overrides]]
filter = "test(/retention_time_travel_survives_many_seeds/)"
slow-timeout = { period = "300s", terminate-after = 24 }
test-group = "io-heavy-sim"
[[profile.ci.overrides]]
filter = "binary(fd_lifecycle)"
slow-timeout = { period = "300s", terminate-after = 4 }
test-group = "io-heavy-sim"
+12
View File
@@ -0,0 +1,12 @@
target/
.git/
.jj/
**/node_modules/
frontend/dist/
frontend/coverage/
frontend/.pnpm-store/
.direnv/
result
.env
*.output
reference-pds-bsky/
+4 -1
View File
@@ -1,6 +1,9 @@
/target
target/
.env
.direnv
result
frontend/node_modules/
frontend/dist/
frontend/coverage/
frontend/.pnpm-store
frontend/.npmrc
@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "SELECT nextval('firehose_seq') as \"seq!\"",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "seq!",
"type_info": "Int8"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "09142b5a6b91fe4fd795fb6c85af5461e2621f6c3545b6d26e4618d813904b75"
}
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE repo_seq r\n SET seq = $1::bigint + (o.ord - 1)\n FROM unnest($2::bigint[]) WITH ORDINALITY AS o(id, ord)\n WHERE r.id = o.id",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Int8",
"Int8Array"
]
},
"nullable": []
},
"hash": "0b895d105363a60040323c5c47bc897905af3da980d0dc1b3b60bbdcdd90ea7a"
}
@@ -0,0 +1,14 @@
{
"db_name": "PostgreSQL",
"query": "\n DELETE FROM repo_seq\n WHERE did = $1\n AND id <> (SELECT id FROM repo_seq WHERE did = $1 ORDER BY id DESC LIMIT 1)\n ",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text"
]
},
"nullable": []
},
"hash": "122d79e006c0e11214e3cf11cfb98bdff6fbf156893b482572532ebcd8979e61"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT id, did, handle, email, created_at, deactivated_at, takedown_ref, is_admin\n FROM users WHERE did = $1",
"query": "SELECT id, did, handle, email, created_at, deactivated_at, takedown_ref, is_admin, inbound_migration\n FROM users WHERE handle = $1",
"describe": {
"columns": [
{
@@ -42,6 +42,11 @@
"ordinal": 7,
"name": "is_admin",
"type_info": "Bool"
},
{
"ordinal": 8,
"name": "inbound_migration",
"type_info": "Bool"
}
],
"parameters": {
@@ -57,8 +62,9 @@
false,
true,
true,
false,
false
]
},
"hash": "6b51995c40519a63f85c70f29ca8bd6ec1963c8562d78215d980785dc46a6384"
"hash": "18bbda5582db1b32d02ab8a3eee970c9508b9bd67239c2f936639a9f863b30ff"
}
@@ -1,14 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE users SET deactivated_at = NULL WHERE did = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text"
]
},
"nullable": []
},
"hash": "23201d4e26bc650939e30f69fb0bca00d351d057098afebc1017f70a84b4bd22"
}
@@ -1,11 +1,11 @@
{
"db_name": "PostgreSQL",
"query": "\n INSERT INTO repo_seq (did, event_type, active, status)\n VALUES ($1, 'account', false, 'deleted')\n RETURNING seq\n ",
"query": "\n INSERT INTO repo_seq (did, event_type, active, status)\n VALUES ($1, 'account', false, 'deleted')\n RETURNING id\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "seq",
"name": "id",
"type_info": "Int8"
}
],
@@ -18,5 +18,5 @@
false
]
},
"hash": "a0ea3630adb9e411d94d88084df6b05d7fc3ba8c306146fc932b736fe342eceb"
"hash": "2c3c0740793f29b9bd9e0f81c2372d427dee091e20da8f71ae1e5f2769aa985d"
}
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT pg_try_advisory_xact_lock($1) as \"locked!\"",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "locked!",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Int8"
]
},
"nullable": [
null
]
},
"hash": "3cc7247d2cd7e85a6f21d783dd956f8fecc6f63c994014bb4e91504d498b5fd5"
}
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "\n INSERT INTO repo_seq (did, event_type, handle)\n VALUES ($1, 'identity', $2)\n ",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": []
},
"hash": "4f3ef746cb5fc0916d3df5f92494c538497d3b9c1cfcc6e44c9b80f2193094f6"
}
@@ -1,23 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "\n INSERT INTO repo_seq (did, event_type, handle)\n VALUES ($1, 'identity', $2)\n RETURNING seq\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "seq",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false
]
},
"hash": "50b2178cd174af1dc132a22048e52a6af492f4d8025ec512dd9c0c9ba62eb202"
}
@@ -0,0 +1,14 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE users SET deactivated_at = NULL, inbound_migration = FALSE WHERE did = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text"
]
},
"nullable": []
},
"hash": "51e029e572777e6a103fd7fd5550494de9d4cac7e3ff84e27ddec1a6aaefc047"
}
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE comms_queue\n SET status = 'failed'::comms_status,\n attempts = max_attempts,\n last_error = $2,\n updated_at = NOW()\n WHERE id = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"Text"
]
},
"nullable": []
},
"hash": "5bee4ed5296667e4ca7e1a97aec28d30a470b8aee7b378ec9ca4e34de4faf349"
}
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT id as \"id!\" FROM repo_seq WHERE seq IS NULL ORDER BY id LIMIT $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id!",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Int8"
]
},
"nullable": [
false
]
},
"hash": "714a1c9a73e2fed8182d4ab2fe4a2c33bc10a52e27ed95d3a6641477ae42395e"
}
@@ -1,11 +1,11 @@
{
"db_name": "PostgreSQL",
"query": "SELECT seq, did, created_at, event_type as \"event_type: RepoEventType\", commit_cid, prev_cid, prev_data_cid,\n ops, blobs, block_cids, block_data, blocks_cids, handle, active, status, rev\n FROM repo_seq\n WHERE seq = $1",
"query": "SELECT seq as \"seq!\", did, created_at, event_type as \"event_type: RepoEventType\", commit_cid, prev_cid, prev_data_cid,\n ops, blobs, block_cids, block_data, blocks_cids, handle, active, status, rev\n FROM repo_seq\n WHERE seq = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "seq",
"name": "seq!",
"type_info": "Int8"
},
{
@@ -90,7 +90,7 @@
]
},
"nullable": [
false,
true,
false,
false,
false,
@@ -108,5 +108,5 @@
true
]
},
"hash": "4702a94384ce938d3f5384bc2106a48500f8749134e6bfd1c619a9151f5d3772"
"hash": "7cdf3b8136782a5b2e4aff91907a680566b4610e6cbf9d9396d31f9bbd99c3b8"
}
@@ -1,14 +1,8 @@
{
"db_name": "PostgreSQL",
"query": "\n INSERT INTO repo_seq (did, event_type, commit_cid, prev_cid, ops, blobs, block_cids, block_data, rev)\n VALUES ($1, 'commit', $2, $3::TEXT, $4, $5, $6, $7, $8)\n RETURNING seq\n ",
"query": "\n INSERT INTO repo_seq (did, event_type, commit_cid, prev_cid, ops, blobs, block_cids, block_data, rev)\n VALUES ($1, 'commit', $2, $3::TEXT, $4, $5, $6, $7, $8)\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "seq",
"type_info": "Int8"
}
],
"columns": [],
"parameters": {
"Left": [
"Text",
@@ -21,9 +15,7 @@
"Text"
]
},
"nullable": [
false
]
"nullable": []
},
"hash": "a325862f897484bb1a8863fd7a1bc3fd17ab20b6e30498f9900daf80609c413d"
"hash": "800b93a38eef70bafdf0a1b221994f3c19ab562f84d18265376296a592b4c767"
}
@@ -1,14 +1,8 @@
{
"db_name": "PostgreSQL",
"query": "\n INSERT INTO repo_seq (did, event_type, commit_cid, prev_cid, ops, blobs, block_cids, block_data, prev_data_cid, rev)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)\n RETURNING seq\n ",
"query": "\n INSERT INTO repo_seq (did, event_type, commit_cid, prev_cid, ops, blobs, block_cids, block_data, prev_data_cid, rev)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "seq",
"type_info": "Int8"
}
],
"columns": [],
"parameters": {
"Left": [
"Text",
@@ -23,9 +17,7 @@
"Text"
]
},
"nullable": [
false
]
"nullable": []
},
"hash": "97dd3cfe72b74b4dac686c030a07f6eb56dabb2df72815f3248760b4e3f7950e"
"hash": "82276a34f473ff8b9790b78915ae3f72873d4b748cdb47bd2b81edc2f7a6e5a8"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM repo_seq WHERE did = $1 AND seq != $2",
"query": "DELETE FROM repo_seq WHERE did = $1 AND id <> $2",
"describe": {
"columns": [],
"parameters": {
@@ -11,5 +11,5 @@
},
"nullable": []
},
"hash": "b6d6548acb89d6384cd226f6ed0d66de27fde3af24b4a7a3fce7e098812e38a5"
"hash": "856e458c1c78f030545dc147f76d1b91fa4cc6e178eb51133c52358855ba3f02"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at",
"query": "UPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE attempts < max_attempts\n AND scheduled_for <= $1\n AND (\n status = 'pending'\n OR (status = 'processing'\n AND updated_at < $1 - INTERVAL '10 minutes')\n )\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at",
"describe": {
"columns": [
{
@@ -154,5 +154,5 @@
true
]
},
"hash": "8047fda41bd94f819213decb8b3e0aba49a8dbdb10217eefd77e3567f8c9694a"
"hash": "890aa92acdcb0fe2a3bf04d87e1f16a801d271da7cedc32fc42c2ef5b100faae"
}
@@ -1,14 +1,8 @@
{
"db_name": "PostgreSQL",
"query": "\n INSERT INTO repo_seq (did, event_type, active, status)\n VALUES ($1, 'account', $2, $3)\n RETURNING seq\n ",
"query": "\n INSERT INTO repo_seq (did, event_type, active, status)\n VALUES ($1, 'account', $2, $3)\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "seq",
"type_info": "Int8"
}
],
"columns": [],
"parameters": {
"Left": [
"Text",
@@ -16,9 +10,7 @@
"Text"
]
},
"nullable": [
false
]
"nullable": []
},
"hash": "623042c0b901e17ff37c5bcc1a4eefe56fc8b996ca8326e75796897d13c9e9ed"
"hash": "8ccd42b5b1b75f45e094889ee26af8517de28ca687efa81fd9c4fce598338fe9"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "\n DELETE FROM oauth_token\n WHERE id IN (\n SELECT id FROM oauth_token\n WHERE did = $1\n ORDER BY updated_at ASC\n OFFSET $2\n )\n ",
"query": "\n DELETE FROM oauth_token\n WHERE id IN (\n SELECT id FROM oauth_token\n WHERE did = $1\n ORDER BY created_at DESC\n OFFSET $2\n )\n ",
"describe": {
"columns": [],
"parameters": {
@@ -11,5 +11,5 @@
},
"nullable": []
},
"hash": "56cd24903171eddc2ededd9079ffe10937c34e99b0305f25c980ca754da44625"
"hash": "8f4357f7a18ddcf6b686a4555f244d37c35917364b8f917ca6ee2d4030ace742"
}
@@ -1,11 +1,11 @@
{
"db_name": "PostgreSQL",
"query": "SELECT seq, did, created_at, event_type as \"event_type: RepoEventType\", commit_cid, prev_cid, prev_data_cid,\n ops, blobs, block_cids, block_data, blocks_cids, handle, active, status, rev\n FROM repo_seq\n WHERE seq > $1\n ORDER BY seq ASC",
"query": "SELECT seq as \"seq!\", did, created_at, event_type as \"event_type: RepoEventType\", commit_cid, prev_cid, prev_data_cid,\n ops, blobs, block_cids, block_data, blocks_cids, handle, active, status, rev\n FROM repo_seq\n WHERE seq > $1\n ORDER BY seq ASC",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "seq",
"name": "seq!",
"type_info": "Int8"
},
{
@@ -90,7 +90,7 @@
]
},
"nullable": [
false,
true,
false,
false,
false,
@@ -108,5 +108,5 @@
true
]
},
"hash": "c47746c256fd8c9d386551f3bd270e9a96fecdb713642d07d60fb4aeb02c8426"
"hash": "957bc81c63510ab502d2f8cea75dafcbdf2c8ebb448c86990d5e404ee5280438"
}
@@ -1,14 +1,8 @@
{
"db_name": "PostgreSQL",
"query": "\n INSERT INTO repo_seq (did, event_type, commit_cid, rev, block_cids, block_data)\n VALUES ($1, 'sync', $2, $3, $4, $5)\n RETURNING seq\n ",
"query": "\n INSERT INTO repo_seq (did, event_type, commit_cid, rev, block_cids, block_data)\n VALUES ($1, 'sync', $2, $3, $4, $5)\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "seq",
"type_info": "Int8"
}
],
"columns": [],
"parameters": {
"Left": [
"Text",
@@ -18,9 +12,7 @@
"ByteaArray"
]
},
"nullable": [
false
]
"nullable": []
},
"hash": "5996774d097a484f81353d49e4a99b07c30a003377c924add576bfb48ddde29b"
"hash": "9ba7869f29286cfd6b811ffa1e26bda1d637b98b44be66cdc265b1af4cec2d4f"
}
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT pg_advisory_xact_lock($1)",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "pg_advisory_xact_lock",
"type_info": "Void"
}
],
"parameters": {
"Left": [
"Int8"
]
},
"nullable": [
null
]
},
"hash": "a06e1d9f6f95e4c4c2b98310ebddcc9d963cc033582bf2e945e8bf3a301b4247"
}
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT setval('firehose_seq', $1)",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "setval",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Int8"
]
},
"nullable": [
null
]
},
"hash": "bd0407818c8bdae1f1024777951b08cb9c386fe751722e1caf6ad243e271697a"
}
@@ -0,0 +1,12 @@
{
"db_name": "PostgreSQL",
"query": "NOTIFY repo_updates",
"describe": {
"columns": [],
"parameters": {
"Left": []
},
"nullable": []
},
"hash": "d4eacb9bca838dd18dcfd8ca39b83ef50968bfaaabc213536c049cc222ffe9c8"
}
@@ -1,11 +1,11 @@
{
"db_name": "PostgreSQL",
"query": "SELECT seq, did, created_at, event_type as \"event_type: RepoEventType\", commit_cid, prev_cid, prev_data_cid,\n ops, blobs, block_cids, block_data, blocks_cids, handle, active, status, rev\n FROM repo_seq\n WHERE seq > $1\n ORDER BY seq ASC\n LIMIT $2",
"query": "SELECT seq as \"seq!\", did, created_at, event_type as \"event_type: RepoEventType\", commit_cid, prev_cid, prev_data_cid,\n ops, blobs, block_cids, block_data, blocks_cids, handle, active, status, rev\n FROM repo_seq\n WHERE seq > $1\n ORDER BY seq ASC\n LIMIT $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "seq",
"name": "seq!",
"type_info": "Int8"
},
{
@@ -91,7 +91,7 @@
]
},
"nullable": [
false,
true,
false,
false,
false,
@@ -109,5 +109,5 @@
true
]
},
"hash": "89c8ce80d8c52b4668c34f63e48b51cacc7e8ff300958a192378f1d72de04f9a"
"hash": "da2770e44c0ec21756ce4b6b0c57abff4f38cd8c680c5ced7d3b35f3c1f58da7"
}
@@ -1,11 +1,11 @@
{
"db_name": "PostgreSQL",
"query": "SELECT seq, did, created_at, event_type as \"event_type: RepoEventType\", commit_cid, prev_cid, prev_data_cid,\n ops, blobs, block_cids, block_data, blocks_cids, handle, active, status, rev\n FROM repo_seq\n WHERE seq > $1\n ORDER BY seq ASC\n LIMIT $2",
"query": "SELECT seq as \"seq!\", did, created_at, event_type as \"event_type: RepoEventType\", commit_cid, prev_cid, prev_data_cid,\n ops, blobs, block_cids, block_data, blocks_cids, handle, active, status, rev\n FROM repo_seq\n WHERE seq > $1 AND seq < $2\n ORDER BY seq ASC",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "seq",
"name": "seq!",
"type_info": "Int8"
},
{
@@ -91,7 +91,7 @@
]
},
"nullable": [
false,
true,
false,
false,
false,
@@ -109,5 +109,5 @@
true
]
},
"hash": "0349b2af4d21105029ecc2bf81ad7be9987ec17ac088730596849d7a45558e57"
"hash": "dfe011b2d2bafbbbbea508a4977fc34bb1be4cf2612296df23763e6e5841c33e"
}
@@ -1,11 +1,11 @@
{
"db_name": "PostgreSQL",
"query": "SELECT seq, did, created_at, event_type as \"event_type: RepoEventType\", commit_cid, prev_cid, prev_data_cid,\n ops, blobs, block_cids, block_data, blocks_cids, handle, active, status, rev\n FROM repo_seq\n WHERE seq > $1 AND seq < $2\n ORDER BY seq ASC",
"query": "SELECT seq as \"seq!\", did, created_at, event_type as \"event_type: RepoEventType\", commit_cid, prev_cid, prev_data_cid,\n ops, blobs, block_cids, block_data, blocks_cids, handle, active, status, rev\n FROM repo_seq\n WHERE seq > $1\n ORDER BY seq ASC\n LIMIT $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "seq",
"name": "seq!",
"type_info": "Int8"
},
{
@@ -91,7 +91,7 @@
]
},
"nullable": [
false,
true,
false,
false,
false,
@@ -109,5 +109,5 @@
true
]
},
"hash": "ad784aff9bb5c1f19f90a3b1c3b3694a3e1aa889b0fe3e55cfebb7af8c620256"
"hash": "f16d41b41a5388997e6ed408da55056173340ed12b60b1a956de87f73f9d0824"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT id, did, handle, email, created_at, deactivated_at, takedown_ref, is_admin\n FROM users WHERE handle = $1",
"query": "SELECT id, did, handle, email, created_at, deactivated_at, takedown_ref, is_admin, inbound_migration\n FROM users WHERE did = $1",
"describe": {
"columns": [
{
@@ -42,6 +42,11 @@
"ordinal": 7,
"name": "is_admin",
"type_info": "Bool"
},
{
"ordinal": 8,
"name": "inbound_migration",
"type_info": "Bool"
}
],
"parameters": {
@@ -57,8 +62,9 @@
false,
true,
true,
false,
false
]
},
"hash": "584bceda60d8b6a02e7dc44d833e3fba13151f36ba9f75c64e33d6cb534cc939"
"hash": "f1c4ec28b02d09ffce35aa8249c1747a70c12a3ecfc1ff6ca1847840f770db2f"
}
@@ -0,0 +1,21 @@
when:
- event: ["push", "manual"]
branch: ["main"]
engine: nixery
dependencies:
nixpkgs:
- nix
- cachix
- gnugrep
steps:
- name: Authenticate
command: |
cachix authtoken "$CACHIX_AUTH_TOKEN"
- name: Build and push aarch64
command: |
SUBS="--option extra-substituters https://tranquil.cachix.org --option extra-trusted-public-keys tranquil.cachix.org-1:PoO+mGL6a6LcJiPakMDHN4E218/ei/7v2sxeDtNkSRg="
nix-store -qR --include-outputs $(nix-store -qd $(nix build .#packages.x86_64-linux.tranquil-pds-aarch64 $SUBS --print-out-paths --no-link)) | grep -v '\.drv$' | cachix push tranquil
@@ -0,0 +1,28 @@
when:
- event: ["push", "manual"]
branch: ["main"]
engine: nixery
dependencies:
nixpkgs:
- nix
- cachix
- gnugrep
steps:
- name: Authenticate
command: |
cachix authtoken "$CACHIX_AUTH_TOKEN"
- name: Build and push x86_64
command: |
SUBS="--option extra-substituters https://tranquil.cachix.org --option extra-trusted-public-keys tranquil.cachix.org-1:PoO+mGL6a6LcJiPakMDHN4E218/ei/7v2sxeDtNkSRg="
nix-store -qR --include-outputs $(nix-store -qd $(nix build .#packages.x86_64-linux.tranquil-pds $SUBS --print-out-paths --no-link)) | grep -v '\.drv$' | cachix push tranquil
nix-store -qR --include-outputs $(nix-store -qd $(nix build .#packages.x86_64-linux.tranquil-frontend $SUBS --print-out-paths --no-link)) | grep -v '\.drv$' | cachix push tranquil
- name: Build and push devShell
command: |
SUBS="--option extra-substituters https://tranquil.cachix.org --option extra-trusted-public-keys tranquil.cachix.org-1:PoO+mGL6a6LcJiPakMDHN4E218/ei/7v2sxeDtNkSRg="
nix develop $SUBS --profile dev-profile -c true
cachix push tranquil dev-profile
+24 -12
View File
@@ -1,24 +1,36 @@
when:
- event: []
branch: []
- event: [ "manual" ]
- event: [ "push" ]
branch: [ "main" ]
engine: nixery
dependencies:
nixpkgs:
- podman
- kaniko
environment:
DOCKER_CONFIG: "/kaniko/.docker"
steps:
- name: Create podman config
- name: Configure Kaniko
command: |
mkdir -p ~/.config/containers
echo "unqualified-search-registries = [\"docker.io\"]" >> ~/.config/containers/registries.conf
mkdir -p /kaniko/.docker/
echo "{
\"auths\": {
\"https://atcr.io/v1\":{
\"auth\": \"$ATCR_CREDENTIALS\"
}
}
}" > /kaniko/.docker/config.json
- name: Build image
command: |
podman build . -t tranquil-pds:latest -t "tranquil-pds:$TANGLED_COMMIT_SHA"
- name: Publish image
command: |
podman push --creds "$ATCR_USERNAME:$ATCR_PASSWORD" tranquil-pds:latest "atcr.io/tranquil.farm/tranquil-pds:latest"
podman push --creds "$ATCR_USERNAME:$ATCR_PASSWORD" "tranquil-pds:$TANGLED_COMMIT_SHA" "atcr.io/tranquil.farm/tranquil-pds:$TANGLED_COMMIT_SHA"
executor \
--context=$(pwd) \
--ignore-path=$(pwd) \
--dockerfile=$(pwd)/Dockerfile \
--destination="atcr.io/tranquil.farm/tranquil-pds:latest" \
--destination="atcr.io/tranquil.farm/tranquil-pds:$TANGLED_COMMIT_SHA" \
--push-retry=3 \
--skip-push-permission-check
+135
View File
@@ -0,0 +1,135 @@
# Contributing to Tranquil PDS
## When PRing
In order of importance:
- **You must run your change! Every contribution that says "here's xyz. untested." does not help the project.**
- Relevant tests to your PR must pass. The whole suite doesn't have to be proven to have run, because there are a *ton* of tests and they're quite heavy, but hopefully there are existing tests for whatever you're PRing, and if there aren't, please add those too.
- Run cargo fmt :P
> 🦪 Lewis
>
> Good CI fixes some of these. We should really get around to that.
Things that would also be nice but aren't like, a pain in our side:
- Big changes should be stacked PRs that are broken up into digestible pieces. Those stacked PRs should hopefully be able to be merged individually if necessary.
## Local Development
### Prerequisites
- [Docker](https://docs.docker.com/get-docker/) and Docker Compose
- Add `pds.test` to your hosts file (one-time setup):
```
127.0.0.1 pds.test
```
- **macOS / Linux:** `/etc/hosts`
- **Windows:** `C:\Windows\System32\drivers\etc\hosts`
### Starting the dev environment
```bash
just run-dev
```
This starts the following services via `docker-compose`:
- **Traefik** — HTTPS reverse proxy at `https://pds.test`
- **Backend** — Rust server with `cargo-watch` (auto-rebuilds on file changes)
- **Frontend** — Vite dev server with hot module replacement
- **Postgres** — Database on port 5432
- **PLC Directory** — Local [did-method-plc](https://github.com/did-method-plc/did-method-plc) server for DID registration
- **Mailpit** — Local email server with web UI at [http://localhost:8025](http://localhost:8025)
Once all services are running, open **https://pds.test** in your browser.
### Trusting the self-signed certificate
Traefik generates a self-signed TLS certificate. Your browser will show a security warning on first visit. You can either click through it, or add the certificate to your system trust store for a seamless experience:
**macOS:**
```bash
# Extract the cert from traefik and add it to the system keychain
echo | openssl s_client -connect localhost:443 -servername pds.test 2>/dev/null | openssl x509 > /tmp/pds-test.pem
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain /tmp/pds-test.pem
```
**Linux (Debian/Ubuntu):**
```bash
echo | openssl s_client -connect localhost:443 -servername pds.test 2>/dev/null | openssl x509 | sudo tee /usr/local/share/ca-certificates/pds-test.crt
sudo update-ca-certificates
```
**Linux (Fedora/RHEL):**
```bash
echo | openssl s_client -connect localhost:443 -servername pds.test 2>/dev/null | openssl x509 | sudo tee /etc/pki/ca-trust/source/anchors/pds-test.pem
sudo update-ca-trust
```
**Windows (PowerShell as Administrator):**
```powershell
$cert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2
$cert.Import([System.Text.Encoding]::UTF8.GetBytes((echo | openssl s_client -connect localhost:443 -servername pds.test 2>$null | openssl x509)))
$store = New-Object System.Security.Cryptography.X509Certificates.X509Store("Root", "LocalMachine")
$store.Open("ReadWrite")
$store.Add($cert)
$store.Close()
```
Restart your browser after adding the certificate.
### Stopping the dev environment
```bash
# Stop containers (preserves database + build cache)
docker compose --profile dev down
# Stop and wipe all data (fresh start)
docker compose --profile dev down -v
```
### Direct database access
Postgres is exposed on port 5432:
```bash
psql postgres://postgres:postgres@localhost:5432/pds
```
### How it works
- **Source code** is bind-mounted into the containers so that changes made on the host will be immediately reflected in the application
- **Backend** uses `cargo-watch` to recompile and restart when Rust files change
- **Frontend** uses Vite's HMR for instant browser updates when frontend files change
- **Build cache** (`target/` directory and cargo registry) are stored in Docker volumes, so incremental compilation persists across container restarts
- **Traefik** routes `/`, `/xrpc`, `/oauth`, `/.well-known`, `/u`, and `/health` to the backend; everything else goes to the Vite dev server
- **Mailpit** captures all outgoing email — open [http://localhost:8025](http://localhost:8025) to view verification emails during registration
- **PLC Directory** runs locally so DID registration doesn't hit the real `plc.directory`
### Running the backend natively
If you prefer running the Rust backend outside Docker (faster incremental builds on host), you need:
- Rust toolchain (see `rust-toolchain.toml`)
- `protoc` (`brew install protobuf` on macOS)
- PostgreSQL (start with `docker compose up db`)
Then run:
```bash
cargo run -p tranquil-server -- --config config.toml
```
And start the frontend separately:
```bash
cd frontend && pnpm install && pnpm dev
```
Generated
+327 -43
View File
@@ -9,7 +9,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "087113bd50d9adce24850eed5d0476c7d199d532fce8fab5173650331e09033a"
dependencies = [
"abnf-core",
"nom",
"nom 7.1.3",
]
[[package]]
@@ -18,7 +18,7 @@ version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c44e09c43ae1c368fb91a03a566472d0087c26cf7e1b9e8e289c14ede681dd7d"
dependencies = [
"nom",
"nom 7.1.3",
]
[[package]]
@@ -210,16 +210,32 @@ version = "0.6.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5493c3bedbacf7fd7382c6346bbd66687d12bbaad3a89a2d2c303ee6cf20b048"
dependencies = [
"asn1-rs-derive",
"asn1-rs-derive 0.5.1",
"asn1-rs-impl",
"displaydoc",
"nom",
"nom 7.1.3",
"num-traits",
"rusticata-macros",
"thiserror 1.0.69",
"time",
]
[[package]]
name = "asn1-rs"
version = "0.7.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b7f43a50ac4fdca5df8e885c21b835997f0a1cdee65494a6847694a98652d9d8"
dependencies = [
"asn1-rs-derive 0.6.0",
"asn1-rs-impl",
"displaydoc",
"nom 7.1.3",
"num-traits",
"rusticata-macros",
"thiserror 2.0.18",
"time",
]
[[package]]
name = "asn1-rs-derive"
version = "0.5.1"
@@ -232,6 +248,18 @@ dependencies = [
"synstructure",
]
[[package]]
name = "asn1-rs-derive"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3109e49b1e4909e9db6515a30c633684d68cdeaa252f215214cb4fa1a5bfee2c"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.117",
"synstructure",
]
[[package]]
name = "asn1-rs-impl"
version = "0.2.0"
@@ -1046,7 +1074,7 @@ version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "08807e080ed7f9d5433fa9b275196cfc35414f66a0c79d864dc51a0d825231a3"
dependencies = [
"bit-vec",
"bit-vec 0.8.0",
]
[[package]]
@@ -1055,6 +1083,15 @@ version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5e764a1d40d510daf35e07be9eb06e75770908c27d411ee6c92109c9840eaaf7"
[[package]]
name = "bit-vec"
version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b71798fca2c1fe1086445a7258a4bc81e6e49dcd24c8d0dd9a1e57395b603f51"
dependencies = [
"serde",
]
[[package]]
name = "bitflags"
version = "2.11.0"
@@ -1558,7 +1595,7 @@ checksum = "06b4f5ec222421e22bb0a8cbaa36b1d2b50fd45cdd30c915ded34108da78b29f"
dependencies = [
"confique-macro",
"serde",
"toml",
"toml 0.9.12+spec-1.1.0",
]
[[package]]
@@ -1970,9 +2007,23 @@ version = "9.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5cd0a5c643689626bec213c4d8bd4d96acc8ffdb4ad4bb6bc16abf27d5f4b553"
dependencies = [
"asn1-rs",
"asn1-rs 0.6.2",
"displaydoc",
"nom",
"nom 7.1.3",
"num-bigint",
"num-traits",
"rusticata-macros",
]
[[package]]
name = "der-parser"
version = "10.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "07da5016415d5a3c4dd39b11ed26f915f52fc4e0dc197d87908bc916e51bc1a6"
dependencies = [
"asn1-rs 0.7.2",
"displaydoc",
"nom 7.1.3",
"num-bigint",
"num-traits",
"rusticata-macros",
@@ -2216,6 +2267,22 @@ dependencies = [
"zeroize",
]
[[package]]
name = "email-encoding"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9298e6504d9b9e780ed3f7dfd43a61be8cd0e09eb07f7706a945b0072b6670b6"
dependencies = [
"base64 0.22.1",
"memchr",
]
[[package]]
name = "email_address"
version = "0.2.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e079f19b08ca6239f47f8ba8509c11cf3ea30095831f7fed61441475edd8c449"
[[package]]
name = "embedded-io"
version = "0.4.0"
@@ -2818,6 +2885,34 @@ dependencies = [
"tracing",
]
[[package]]
name = "h3"
version = "0.0.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "10872b55cfb02a821b69dc7cf8dc6a71d6af25eb9a79662bec4a9d016056b3be"
dependencies = [
"bytes",
"fastrand",
"futures-util",
"http 1.4.0",
"pin-project-lite",
"tokio",
]
[[package]]
name = "h3-quinn"
version = "0.0.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b2e732c8d91a74731663ac8479ab505042fbf547b9a207213ab7fbcbfc4f8b4"
dependencies = [
"bytes",
"futures",
"h3",
"quinn",
"tokio",
"tokio-util",
]
[[package]]
name = "half"
version = "2.7.1"
@@ -3780,6 +3875,37 @@ version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2"
[[package]]
name = "lettre"
version = "0.11.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dabda5859ee7c06b995b9d1165aa52c39110e079ef609db97178d86aeb051fa7"
dependencies = [
"async-trait",
"base64 0.22.1",
"ed25519-dalek",
"email-encoding",
"email_address",
"fastrand",
"futures-io",
"futures-util",
"httpdate",
"idna",
"mime",
"nom 8.0.0",
"percent-encoding",
"quoted_printable",
"rsa",
"rustls 0.23.37",
"sha2",
"socket2 0.6.3",
"tokio",
"tokio-rustls 0.26.4",
"tracing",
"url",
"webpki-roots 1.0.6",
]
[[package]]
name = "libc"
version = "0.2.183"
@@ -4409,6 +4535,15 @@ dependencies = [
"minimal-lexical",
]
[[package]]
name = "nom"
version = "8.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "df9761775871bdef83bee530e60050f7e54b1105350d6884eb0fb4f46c2f9405"
dependencies = [
"memchr",
]
[[package]]
name = "nonzero_ext"
version = "0.3.0"
@@ -4558,7 +4693,16 @@ version = "0.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a8d8034d9489cdaf79228eb9f6a3b8d7bb32ba00d6645ebd48eef4077ceb5bd9"
dependencies = [
"asn1-rs",
"asn1-rs 0.6.2",
]
[[package]]
name = "oid-registry"
version = "0.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "12f40cff3dde1b6087cc5d5f5d4d65712f34016a03ed60e9c08dcc392736b5b7"
dependencies = [
"asn1-rs 0.7.2",
]
[[package]]
@@ -4839,7 +4983,7 @@ checksum = "9114f9c1683dd09c5f4fa024c89fdad783eaae21d3d52dd23ddaaffa29ffb168"
dependencies = [
"either",
"fnv",
"nom",
"nom 7.1.3",
"once_cell",
"postcard",
"quick-xml",
@@ -5050,7 +5194,7 @@ version = "3.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e67ba7e9b2b56446f1d419b1d807906278ffa1a658a8a5d8a39dcb1f5a78614f"
dependencies = [
"toml_edit",
"toml_edit 0.25.5+spec-1.1.0",
]
[[package]]
@@ -5128,7 +5272,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "37566cb3fdacef14c0737f9546df7cfeadbfbc9fef10991038bf5015d0c80532"
dependencies = [
"bit-set",
"bit-vec",
"bit-vec 0.8.0",
"bitflags",
"num-traits",
"rand 0.9.2",
@@ -5371,6 +5515,7 @@ checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20"
dependencies = [
"bytes",
"cfg_aliases",
"futures-io",
"pin-project-lite",
"quinn-proto",
"quinn-udp",
@@ -5427,6 +5572,12 @@ dependencies = [
"proc-macro2",
]
[[package]]
name = "quoted_printable"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "478e0585659a122aa407eb7e3c0e1fa51b1d8a870038bd29f0cf4a8551eea972"
[[package]]
name = "r-efi"
version = "5.3.0"
@@ -5568,6 +5719,20 @@ dependencies = [
"crossbeam-utils",
]
[[package]]
name = "rcgen"
version = "0.14.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "57f6d249aad744e274e682777a50283a225a32705394ee6d5fcc01efa25e4055"
dependencies = [
"pem",
"ring",
"rustls-pki-types",
"time",
"x509-parser 0.18.1",
"yasna",
]
[[package]]
name = "redis"
version = "1.1.0"
@@ -5833,7 +5998,7 @@ version = "4.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "faf0c4a6ece9950b9abdb62b1cfcf2a68b3b67a10ba445b3bb85be2a293d0632"
dependencies = [
"nom",
"nom 7.1.3",
]
[[package]]
@@ -6050,6 +6215,16 @@ dependencies = [
"zeroize",
]
[[package]]
name = "secrecy"
version = "0.10.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e891af845473308773346dc847b2c23ee78fe442e0472ac50e22a18a93d3ae5a"
dependencies = [
"serde",
"zeroize",
]
[[package]]
name = "security-framework"
version = "3.7.0"
@@ -6201,6 +6376,15 @@ dependencies = [
"syn 2.0.117",
]
[[package]]
name = "serde_spanned"
version = "0.6.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bf41e0cfaf7226dca15e8197172c295a782857fcb97fad1808a166870dee75a3"
dependencies = [
"serde",
]
[[package]]
name = "serde_spanned"
version = "1.0.4"
@@ -7144,6 +7328,18 @@ dependencies = [
"tokio",
]
[[package]]
name = "toml"
version = "0.8.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dc1beb996b9d83529a9e75c17a1686767d148d70663143c7854d8b4a09ced362"
dependencies = [
"serde",
"serde_spanned 0.6.9",
"toml_datetime 0.6.11",
"toml_edit 0.22.27",
]
[[package]]
name = "toml"
version = "0.9.12+spec-1.1.0"
@@ -7152,13 +7348,22 @@ checksum = "cf92845e79fc2e2def6a5d828f0801e29a2f8acc037becc5ab08595c7d5e9863"
dependencies = [
"indexmap 2.13.0",
"serde_core",
"serde_spanned",
"serde_spanned 1.0.4",
"toml_datetime 0.7.5+spec-1.1.0",
"toml_parser",
"toml_writer",
"winnow 0.7.15",
]
[[package]]
name = "toml_datetime"
version = "0.6.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "22cddaf88f4fbc13c51aebbf5f8eceb5c7c5a9da2ac40a13519eb5b0a0e8f11c"
dependencies = [
"serde",
]
[[package]]
name = "toml_datetime"
version = "0.7.5+spec-1.1.0"
@@ -7177,6 +7382,20 @@ dependencies = [
"serde_core",
]
[[package]]
name = "toml_edit"
version = "0.22.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a"
dependencies = [
"indexmap 2.13.0",
"serde",
"serde_spanned 0.6.9",
"toml_datetime 0.6.11",
"toml_write",
"winnow 0.7.15",
]
[[package]]
name = "toml_edit"
version = "0.25.5+spec-1.1.0"
@@ -7198,6 +7417,12 @@ dependencies = [
"winnow 1.0.0",
]
[[package]]
name = "toml_write"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801"
[[package]]
name = "toml_writer"
version = "1.0.7+spec-1.1.0"
@@ -7405,7 +7630,7 @@ dependencies = [
[[package]]
name = "tranquil-api"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"anyhow",
"axum",
@@ -7456,7 +7681,7 @@ dependencies = [
[[package]]
name = "tranquil-auth"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"anyhow",
"base32",
@@ -7479,7 +7704,7 @@ dependencies = [
[[package]]
name = "tranquil-cache"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"async-trait",
"base64 0.22.1",
@@ -7493,11 +7718,19 @@ dependencies = [
[[package]]
name = "tranquil-comms"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"async-trait",
"base64 0.22.1",
"chrono",
"ed25519-dalek",
"futures",
"hickory-resolver",
"lettre",
"rand 0.8.5",
"reqwest",
"rsa",
"secrecy",
"serde_json",
"sqlx",
"thiserror 2.0.18",
@@ -7511,7 +7744,7 @@ dependencies = [
[[package]]
name = "tranquil-config"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"confique",
"serde",
@@ -7519,7 +7752,7 @@ dependencies = [
[[package]]
name = "tranquil-crypto"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"aes-gcm",
"base64 0.22.1",
@@ -7535,7 +7768,7 @@ dependencies = [
[[package]]
name = "tranquil-db"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"async-trait",
"chrono",
@@ -7552,7 +7785,7 @@ dependencies = [
[[package]]
name = "tranquil-db-traits"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"async-trait",
"base64 0.22.1",
@@ -7568,7 +7801,7 @@ dependencies = [
[[package]]
name = "tranquil-infra"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"async-trait",
"bytes",
@@ -7579,9 +7812,10 @@ dependencies = [
[[package]]
name = "tranquil-lexicon"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"chrono",
"futures",
"hickory-resolver",
"parking_lot",
"reqwest",
@@ -7597,7 +7831,7 @@ dependencies = [
[[package]]
name = "tranquil-oauth"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"anyhow",
"axum",
@@ -7620,7 +7854,7 @@ dependencies = [
[[package]]
name = "tranquil-oauth-server"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"axum",
"base64 0.22.1",
@@ -7653,7 +7887,7 @@ dependencies = [
[[package]]
name = "tranquil-pds"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"aes-gcm",
"anyhow",
@@ -7693,6 +7927,7 @@ dependencies = [
"multibase",
"multihash",
"p256 0.13.2",
"parking_lot",
"rand 0.8.5",
"redis",
"regex",
@@ -7705,6 +7940,7 @@ dependencies = [
"sha2",
"sqlx",
"subtle",
"tempfile",
"testcontainers",
"testcontainers-modules",
"thiserror 2.0.18",
@@ -7745,7 +7981,7 @@ dependencies = [
[[package]]
name = "tranquil-repo"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"bytes",
"cid",
@@ -7757,7 +7993,7 @@ dependencies = [
[[package]]
name = "tranquil-ripple"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"async-trait",
"backon",
@@ -7782,7 +8018,7 @@ dependencies = [
[[package]]
name = "tranquil-scopes"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"axum",
"futures",
@@ -7798,15 +8034,30 @@ dependencies = [
[[package]]
name = "tranquil-server"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"arc-swap",
"axum",
"bytes",
"clap",
"dotenvy",
"ed25519-dalek",
"futures-util",
"h3",
"h3-quinn",
"hex",
"http 1.4.0",
"hyper 1.8.1",
"hyper-util",
"quinn",
"rcgen",
"rustls 0.23.37",
"rustls-pemfile",
"thiserror 2.0.18",
"tokio",
"tokio-rustls 0.26.4",
"tokio-util",
"tower",
"tracing",
"tracing-subscriber",
"tranquil-api",
@@ -7819,7 +8070,7 @@ dependencies = [
[[package]]
name = "tranquil-signal"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"async-trait",
"chrono",
@@ -7842,7 +8093,7 @@ dependencies = [
[[package]]
name = "tranquil-storage"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"async-trait",
"aws-config",
@@ -7859,12 +8110,13 @@ dependencies = [
[[package]]
name = "tranquil-store"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"async-trait",
"bytes",
"chrono",
"cid",
"clap",
"dashmap",
"fjall",
"flume 0.11.1",
@@ -7892,6 +8144,7 @@ dependencies = [
"thiserror 2.0.18",
"tikv-jemallocator",
"tokio",
"toml 0.8.23",
"tracing",
"tracing-subscriber",
"tranquil-db",
@@ -7906,7 +8159,7 @@ dependencies = [
[[package]]
name = "tranquil-sync"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"anyhow",
"axum",
@@ -7928,7 +8181,7 @@ dependencies = [
[[package]]
name = "tranquil-types"
version = "0.5.3"
version = "0.6.5"
dependencies = [
"chrono",
"cid",
@@ -8441,9 +8694,9 @@ checksum = "15784340a24c170ce60567282fb956a0938742dbfbf9eff5df793a686a009b8b"
dependencies = [
"base64 0.21.7",
"base64urlsafedata",
"der-parser",
"der-parser 9.0.0",
"hex",
"nom",
"nom 7.1.3",
"openssl",
"openssl-sys",
"rand 0.9.2",
@@ -8457,7 +8710,7 @@ dependencies = [
"uuid",
"webauthn-attestation-ca",
"webauthn-rs-proto",
"x509-parser",
"x509-parser 0.16.0",
]
[[package]]
@@ -8853,6 +9106,9 @@ name = "winnow"
version = "0.7.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "df79d97927682d2fd8adb29682d1140b343be4ac0f08fd68b7765d9c059d3945"
dependencies = [
"memchr",
]
[[package]]
name = "winnow"
@@ -9008,17 +9264,35 @@ version = "0.16.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fcbc162f30700d6f3f82a24bf7cc62ffe7caea42c0b2cba8bf7f3ae50cf51f69"
dependencies = [
"asn1-rs",
"asn1-rs 0.6.2",
"data-encoding",
"der-parser",
"der-parser 9.0.0",
"lazy_static",
"nom",
"oid-registry",
"nom 7.1.3",
"oid-registry 0.7.1",
"rusticata-macros",
"thiserror 1.0.69",
"time",
]
[[package]]
name = "x509-parser"
version = "0.18.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d43b0f71ce057da06bc0851b23ee24f3f86190b07203dd8f567d0b706a185202"
dependencies = [
"asn1-rs 0.7.2",
"data-encoding",
"der-parser 10.0.0",
"lazy_static",
"nom 7.1.3",
"oid-registry 0.8.1",
"ring",
"rusticata-macros",
"thiserror 2.0.18",
"time",
]
[[package]]
name = "xattr"
version = "1.6.1"
@@ -9047,6 +9321,16 @@ version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cfe53a6657fd280eaa890a3bc59152892ffa3e30101319d168b781ed6529b049"
[[package]]
name = "yasna"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b5f6765e852b9b4dc8e2a76843e4d64d1cea8e79bcde0b6901aea8e7c7f08282"
dependencies = [
"bit-vec 0.9.1",
"time",
]
[[package]]
name = "yoke"
version = "0.8.1"
+15 -2
View File
@@ -26,7 +26,7 @@ members = [
]
[workspace.package]
version = "0.5.3"
version = "0.6.5"
edition = "2024"
license = "AGPL-3.0-or-later"
@@ -59,6 +59,7 @@ presage = { git = "https://github.com/whisperfish/presage", rev = "fe3ed54c4844a
unicode-segmentation = "1"
aes-gcm = "0.10"
arc-swap = "1"
backon = "1"
bincode = { version = "2", features = ["serde"] }
anyhow = "1.0"
@@ -81,11 +82,15 @@ foca = { version = "1", features = ["bincode-codec", "tracing"] }
futures = "0.3"
futures-util = "0.3"
governor = "0.10"
h3 = "0.0.8"
h3-quinn = "0.0.10"
hex = "0.4"
hickory-resolver = { version = "0.24", features = ["tokio-runtime"] }
hkdf = "0.12"
hmac = "0.12"
http = "1.4"
hyper = { version = "1", features = ["server", "http1", "http2"] }
hyper-util = { version = "0.1", features = ["server", "server-auto", "server-graceful", "service", "tokio"] }
image = { version = "0.25", default-features = false, features = ["jpeg", "png", "gif", "webp"] }
qrcodegen = "1.8"
infer = "0.19"
@@ -93,6 +98,7 @@ ipld-core = "0.4"
iroh-car = "0.5"
jacquard-common = { version = "0.9", features = ["crypto-k256"] }
jacquard-repo = "0.9"
lettre = { version = "0.11", default-features = false, features = ["builder", "smtp-transport", "tokio1", "tokio1-rustls-tls", "pool", "dkim", "tracing"] }
jsonwebtoken = { version = "10.2", features = ["rust_crypto"] }
k256 = { version = "0.13", features = ["ecdsa", "pem", "pkcs8"] }
metrics = "0.24"
@@ -102,9 +108,15 @@ parking_lot = "0.12"
multihash = "0.19"
p256 = { version = "0.13", features = ["ecdsa"] }
p384 = { version = "0.13", features = ["ecdsa"] }
quinn = { version = "0.11", default-features = false, features = ["runtime-tokio", "rustls-ring", "log"] }
rand = "0.8"
rcgen = { version = "0.14", default-features = false, features = ["ring", "pem"] }
redis = { version = "1.0", features = ["tokio-comp", "connection-manager"] }
regex = "1"
rsa = "0.9"
rustls = { version = "0.23", default-features = false, features = ["ring", "std", "tls12", "logging"] }
rustls-pemfile = "2"
secrecy = { version = "0.10", features = ["serde"] }
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls-webpki-roots", "http2", "charset", "macos-system-configuration"] }
serde = { version = "1.0", features = ["derive"] }
serde_bytes = "0.11"
@@ -116,8 +128,9 @@ sqlx = { version = "0.8", features = ["runtime-tokio-rustls", "postgres", "uuid"
subtle = "2.5"
thiserror = "2.0"
tokio = { version = "1.48", features = ["macros", "rt-multi-thread", "time", "signal", "process", "io-util", "fs"] }
tokio-util = "0.7.18"
tokio-util = { version = "0.7.18", features = ["rt"] }
tokio-tungstenite = { version = "0.28", features = ["rustls-tls-webpki-roots"] }
tokio-rustls = { version = "0.26", default-features = false, features = ["ring", "tls12", "logging"] }
totp-rs = { version = "5", features = ["qr"] }
tower = "0.5"
tower-http = { version = "0.6", features = ["fs", "cors"] }
+5 -7
View File
@@ -1,12 +1,12 @@
FROM node:24-alpine AS frontend
RUN corepack enable && corepack prepare pnpm@latest --activate
WORKDIR /app
COPY frontend/package.json frontend/pnpm-lock.yaml ./
COPY frontend/package.json frontend/pnpm-lock.yaml frontend/pnpm-workspace.yaml ./
RUN pnpm install --frozen-lockfile
COPY frontend/ ./
RUN pnpm build
FROM rust:1.92-alpine AS builder
FROM rust:1.96-alpine AS builder
RUN apk add --no-cache ca-certificates musl-dev pkgconfig openssl-dev openssl-libs-static mold clang protoc
ENV RUSTFLAGS="-C linker=clang -C link-arg=-fuse-ld=mold"
WORKDIR /app
@@ -35,7 +35,7 @@ COPY crates/tranquil-oauth-server ./crates/tranquil-oauth-server
COPY crates/tranquil-store ./crates/tranquil-store
COPY crates/tranquil-signal ./crates/tranquil-signal
COPY crates/tranquil-server ./crates/tranquil-server
COPY migrations ./crates/tranquil-pds/migrations
COPY migrations ./migrations
RUN --mount=type=cache,target=/usr/local/cargo/registry \
--mount=type=cache,target=/app/target \
if [ "$SLIM" = "true" ]; then \
@@ -46,13 +46,11 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry \
cp target/release/tranquil-server /tmp/tranquil-pds
FROM alpine:3.23
RUN apk add --no-cache msmtp ca-certificates \
&& ln -sf /usr/bin/msmtp /usr/sbin/sendmail
RUN apk add --no-cache ca-certificates
COPY --from=builder /tmp/tranquil-pds /usr/local/bin/tranquil-pds
COPY --from=frontend /app/dist /var/lib/tranquil-pds/frontend
COPY migrations /app/migrations
WORKDIR /app
ENV SERVER_HOST=0.0.0.0
ENV SERVER_HOST=[::]
ENV SERVER_PORT=3000
EXPOSE 3000
CMD ["tranquil-pds"]
+20
View File
@@ -0,0 +1,20 @@
FROM rust:1.96-alpine
RUN apk add --no-cache \
ca-certificates \
musl-dev \
pkgconfig \
openssl-dev \
openssl-libs-static \
mold \
clang \
protoc
RUN cargo install cargo-watch
ENV RUSTFLAGS="-C linker=clang -C link-arg=-fuse-ld=mold"
ENV SQLX_OFFLINE=true
WORKDIR /app
CMD ["cargo", "watch", "-x", "run -p tranquil-server -- --config /app/config.dev.toml"]
+15 -8
View File
@@ -6,7 +6,7 @@ A Personal Data Server for the AT Protocol.
We came together to make this PDS to enable and empower our users to better host their data on this shared protocol. All of our decisions as a project are guided by their usefulness to the community: PDS hosters and end-users both.
Comparatively: Bluesky the company created a "reference PDS" that we can self-host quite easily, and that's great, but Bluesky has an incentive to make software for themselvess first & foremost, then secondly their software can be useful for us self-hosters. In contrast, Tranquil is not from a company, and will never be.
Comparatively: Bluesky the company created a "reference PDS" that we can self-host quite easily, and that's great, but Bluesky has an incentive to make software for themselves first & foremost, then secondly their software can be useful for us self-hosters. In contrast, Tranquil is not from a company, and will never be.
## What's different about Tranquil PDS
@@ -20,7 +20,7 @@ It is a superset of the reference PDS, including:
- account delegation: letting others manage an account with configurable permission levels
- a built-in web UI for account management, repo browsing, and admin
Unlike the ref PDS, Tranquil itself is compiled to a single binary with no nodeJS runtime. However, at time of writing, Tranquil requires postgres running separately.
Unlike the ref PDS, Tranquil is a single binary with no nodejs runtime. That said, at time of writing, Tranquil does require postgres running separately.
## Quick Start
@@ -46,22 +46,28 @@ just test
just lint
```
Nix users can enter a devshell with `nix develop`, or `direnv allow` to auto-enter via the bundled `.envrc`. Pre-built artifacts including the devshell are available from our [binary cache](docs/2_INSTALL_NIX.md#binary-cache).
## Production Deployment
### Quick Deploy (Docker/Podman Compose)
Edit `config.toml` with your values. Generate secrets with `openssl rand -base64 48`.
`docker-compose.prod.yaml` pulls the prebuilt image `atcr.io/tranquil.farm/tranquil-pds:latest`. Sign in to the registry first with `podman login atcr.io`. The Containers guide covers building from source.
```bash
cp example.toml config.toml
```
Edit `config.toml` with your values and generate secrets with `openssl rand -base64 48`. Set the postgres password to match `docker-compose.prod.yaml`. nginx needs a TLS certificate before it starts, so follow the wildcard cert steps in the [Containers guide](docs/2_INSTALL_CONTAINERS.md).
```bash
podman-compose -f docker-compose.prod.yaml up -d
```
### Installation Guides
- [Debian](docs/install-debian.md)
- [Containers](docs/install-containers.md)
- [Kubernetes](docs/install-kubernetes.md)
- [Nix](docs/2_INSTALL_NIX.md)
- [Containers](docs/2_INSTALL_CONTAINERS.md)
## Community
@@ -74,7 +80,7 @@ We currently don't have a shared space to chat and organize Tranquil things, but
- [@oyster.cafe](https://tangled.org/did:plc:3fwecdnvtcscjnrx2p4n7alz)
- [@nel.pet](https://tangled.org/did:plc:h5wsnqetncv6lu2weom35lg2)
### Amazing contributers
### Amazing contributors
- [@isabelroses.com](https://tangled.org/did:plc:qxichs7jsycphrsmbujwqbfb)
- [@quilling.dev](https://tangled.org/did:plc:jrtgsidnmxaen4offglr5lsh)
@@ -85,6 +91,8 @@ We currently don't have a shared space to chat and organize Tranquil things, but
- [@a.starrysky.fyi](https://tangled.org/did:plc:uuyqs6y3pwtbteet4swt5i5y)
- [@sans-self.org](https://tangled.org/did:plc:wydyrngmxbcsqdvhmd7whmye)
- [@tachyonism.tngl.sh](https://tangled.org/did:plc:w6qiwij62bmdugsd3gemhpy2)
- [@trezy.codes](https://tangled.org/did:plc:4jrld6fwpnwqehtce56qshzv)
- [@ave.zone](https://tangled.org/did:plc:mchrltkrhuzpxleiwpmmvpar)
- Could be your name here too!
### Tranquil PDS instances in the wild!
@@ -99,4 +107,3 @@ This project is very grateful to [@nonbinary.computer](https://tangled.org/did:p
## License
AGPL-3.0-or-later. Documentation is CC BY-SA 4.0. See [LICENSE](LICENSE) for details.
+30
View File
@@ -0,0 +1,30 @@
[server]
hostname = "pds.test"
allow_http_proxy = true
invite_code_required = false
disable_rate_limiting = true
[frontend]
enabled = true
dir = "/app/frontend/public"
[database]
url = "postgres://postgres:postgres@db:5432/pds"
[storage]
path = "/var/lib/tranquil-pds/blobs"
[plc]
directory_url = "http://plc:2582"
[email]
from_address = "noreply@pds.test"
from_name = "Tranquil PDS (Dev)"
[email.smarthost]
host = "mailpit"
port = 1025
tls = "none"
[secrets]
allow_insecure = true
+4 -3
View File
@@ -12,7 +12,6 @@ use tranquil_pds::auth::{Auth, NotTakendown, Permissive};
use tranquil_pds::state::AppState;
const APP_BSKY_NAMESPACE: &str = "app.bsky";
const MAX_PREFERENCES_COUNT: usize = 100;
const MAX_PREFERENCE_SIZE: usize = 10_000;
const PERSONAL_DETAILS_PREF: &str = "app.bsky.actor.defs#personalDetailsPref";
const DECLARED_AGE_PREF: &str = "app.bsky.actor.defs#declaredAgePref";
@@ -92,6 +91,8 @@ pub async fn put_preferences(
auth: Auth<NotTakendown>,
Json(input): Json<PutPreferencesInput>,
) -> Response {
let max_preferences_count: usize = tranquil_config::get().server.max_preferences_count;
let has_full_access = auth.permissions().has_full_access();
let user_id: uuid::Uuid = match state.repos.user.get_id_by_did(&auth.did).await {
Ok(Some(id)) => id,
@@ -99,11 +100,11 @@ pub async fn put_preferences(
return ApiError::InternalError(Some("User not found".into())).into_response();
}
};
if input.preferences.len() > MAX_PREFERENCES_COUNT {
if input.preferences.len() > max_preferences_count {
return ApiError::InvalidRequest(format!(
"Too many preferences: {} exceeds limit of {}",
input.preferences.len(),
MAX_PREFERENCES_COUNT
max_preferences_count
))
.into_response();
}
+4 -6
View File
@@ -9,7 +9,6 @@ use tranquil_pds::state::AppState;
#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
pub struct SignalStatusOutput {
pub enabled: bool,
pub linked: bool,
}
@@ -23,13 +22,12 @@ pub async fn get_signal_status(
State(state): State<AppState>,
_auth: Auth<Admin>,
) -> Result<Json<SignalStatusOutput>, ApiError> {
let enabled = tranquil_config::get().signal.enabled;
let linked = match &state.signal_sender {
Some(slot) => slot.is_linked().await,
None => false,
};
Ok(Json(SignalStatusOutput { enabled, linked }))
Ok(Json(SignalStatusOutput { linked }))
}
pub async fn link_signal_device(
@@ -76,7 +74,7 @@ pub async fn link_signal_device(
let result = tokio::select! {
biased;
_ = shutdown.cancelled() => {
tracing::info!("Signal linking aborted due to server shutdown");
tracing::info!("server shutting down, aborting signal linking");
return;
}
r = link_result.completion => r,
@@ -84,10 +82,10 @@ pub async fn link_signal_device(
match result {
Ok(Ok(client)) => {
if slot_for_task.complete_link(generation, client).await {
tracing::info!("Signal device linked successfully");
tracing::info!("signal device linked");
} else {
tracing::warn!(
"Signal link completed but generation mismatch or already linked; discarding"
"discarding completed signal link, generation mismatch or already linked"
);
}
}
@@ -510,6 +510,7 @@ pub async fn create_account(
telegram_username: comms.telegram,
signal_username: comms.signal,
deactivated_at,
inbound_migration: is_migration || is_did_web_byod,
encrypted_key_bytes: repo.encrypted_key_bytes,
encryption_version: tranquil_pds::config::ENCRYPTION_VERSION,
reserved_key_id,
+1 -1
View File
@@ -3,5 +3,5 @@ mod sign;
mod submit;
pub use request::request_plc_operation_signature;
pub use sign::{ServiceInput, SignPlcOperationInput, SignPlcOperationOutput, sign_plc_operation};
pub use sign::{SignPlcOperationInput, SignPlcOperationOutput, sign_plc_operation};
pub use submit::{SubmitPlcOperationInput, submit_plc_operation};
@@ -6,10 +6,6 @@ use tranquil_pds::api::error::{ApiError, DbResultExt};
use tranquil_pds::auth::{Auth, Permissive};
use tranquil_pds::state::AppState;
fn generate_plc_token() -> String {
tranquil_pds::util::generate_token_code()
}
pub async fn request_plc_operation_signature(
State(state): State<AppState>,
auth: Auth<Permissive>,
@@ -28,12 +24,13 @@ pub async fn request_plc_operation_signature(
.ok_or(ApiError::AccountNotFound)?;
let _ = state.repos.infra.delete_plc_tokens_for_user(user_id).await;
let plc_token = generate_plc_token();
let display_token = tranquil_pds::util::generate_token_code();
let stored_token = tranquil_pds::util::normalize_token_code(&display_token);
let expires_at = Utc::now() + Duration::minutes(10);
state
.repos
.infra
.insert_plc_token(user_id, &plc_token, expires_at)
.insert_plc_token(user_id, &stored_token, expires_at)
.await
.log_db_err("creating PLC token")?;
@@ -42,7 +39,7 @@ pub async fn request_plc_operation_signature(
state.repos.user.as_ref(),
state.repos.infra.as_ref(),
user_id,
&plc_token,
&display_token,
hostname,
)
.await
+5 -24
View File
@@ -9,7 +9,7 @@ use tranquil_pds::api::ApiError;
use tranquil_pds::api::error::DbResultExt;
use tranquil_pds::auth::{Auth, Permissive};
use tranquil_pds::circuit_breaker::with_circuit_breaker;
use tranquil_pds::plc::{PlcError, PlcService, ServiceType, create_update_op, sign_operation};
use tranquil_pds::plc::{PlcError, PlcService, create_update_op, sign_operation};
use tranquil_pds::state::AppState;
#[derive(Debug, Deserialize)]
@@ -19,14 +19,7 @@ pub struct SignPlcOperationInput {
pub rotation_keys: Option<Vec<String>>,
pub also_known_as: Option<Vec<String>>,
pub verification_methods: Option<HashMap<String, String>>,
pub services: Option<HashMap<String, ServiceInput>>,
}
#[derive(Debug, Deserialize, Clone)]
pub struct ServiceInput {
#[serde(rename = "type")]
pub service_type: ServiceType,
pub endpoint: String,
pub services: Option<HashMap<String, PlcService>>,
}
#[derive(Debug, Serialize)]
@@ -50,9 +43,10 @@ pub async fn sign_plc_operation(
"PLC operations are only valid for did:plc identities".into(),
));
}
let token = input.token.as_ref().ok_or_else(|| {
let raw_token = input.token.as_ref().ok_or_else(|| {
ApiError::InvalidRequest("Email confirmation token required to sign PLC operations".into())
})?;
let token = &tranquil_pds::util::normalize_token_code(raw_token);
let user_id = state
.repos
@@ -106,25 +100,12 @@ pub async fn sign_plc_operation(
if last_op.is_tombstone() {
return Err(ApiError::from(PlcError::Tombstoned));
}
let services = input.services.map(|s| {
s.into_iter()
.map(|(k, v)| {
(
k,
PlcService {
service_type: v.service_type,
endpoint: v.endpoint,
},
)
})
.collect()
});
let unsigned_op = create_update_op(
&last_op,
input.rotation_keys,
input.verification_methods,
input.also_known_as,
services,
input.services,
)
.map_err(|e| match e {
PlcError::Tombstoned => ApiError::InvalidRequest("Cannot update tombstoned DID".into()),
@@ -129,20 +129,13 @@ pub async fn submit_plc_operation(
.await
.map_err(ApiError::from)?;
match state
if let Err(e) = state
.repos
.repo
.insert_identity_event(did, Some(&user.handle))
.await
{
Ok(seq) => {
if let Err(e) = state.repos.repo.notify_update(seq).await {
warn!("Failed to notify identity event: {:?}", e);
}
}
Err(e) => {
warn!("Failed to sequence identity event: {:?}", e);
}
warn!("Failed to sequence identity event: {:?}", e);
}
let _ = state
.cache
+12 -2
View File
@@ -15,8 +15,12 @@ pub mod temp;
use tranquil_pds::state::AppState;
pub fn api_routes() -> axum::Router<AppState> {
use axum::extract::DefaultBodyLimit;
use axum::routing::{get, post};
let blob_body_limit =
DefaultBodyLimit::max(tranquil_config::get().server.max_blob_size as usize);
axum::Router::new()
.route("/_health", get(server::health))
.route(
@@ -68,7 +72,10 @@ pub fn api_routes() -> axum::Router<AppState> {
.route("/com.atproto.repo.deleteRecord", post(repo::delete_record))
.route("/com.atproto.repo.listRecords", get(repo::list_records))
.route("/com.atproto.repo.describeRepo", get(repo::describe_repo))
.route("/com.atproto.repo.uploadBlob", post(repo::upload_blob))
.route(
"/com.atproto.repo.uploadBlob",
post(repo::upload_blob).layer(blob_body_limit),
)
.route("/com.atproto.repo.applyWrites", post(repo::apply_writes))
.route(
"/com.atproto.server.checkAccountStatus",
@@ -247,7 +254,10 @@ pub fn api_routes() -> axum::Router<AppState> {
"/_identity.verifyHandleOwnership",
post(identity::verify_handle_ownership),
)
.route("/com.atproto.repo.importRepo", post(repo::import_repo))
.route(
"/com.atproto.repo.importRepo",
post(repo::import_repo).layer(blob_body_limit),
)
.route(
"/com.atproto.admin.deleteAccount",
post(admin::delete_account),
@@ -282,7 +282,7 @@ async fn process_messaging_channel_update(
"Invalid Telegram username. Must be 5-32 characters, alphanumeric or underscore".into(),
),
CommsChannel::Signal => ApiError::InvalidRequest(
"Invalid Signal username. Must be 3-32 characters followed by .XX (e.g. username.01)".into(),
"Invalid Signal username. Must be a 3-32 character nickname, a dot, then a 2-20 digit discriminator".into(),
),
CommsChannel::Email => ApiError::InvalidEmail,
});
+8 -78
View File
@@ -77,27 +77,6 @@ pub async fn import_repo(
blocks.len(),
root
);
let Some(root_block) = blocks.get(&root) else {
return Err(ApiError::InvalidRequest(
"Root block not found in CAR file".into(),
));
};
let commit_did: Did = match jacquard_repo::commit::Commit::from_cbor(root_block) {
Ok(commit) => commit
.did()
.as_str()
.parse()
.map_err(|_| ApiError::InvalidRequest("Commit contains invalid DID".into()))?,
Err(e) => {
return Err(ApiError::InvalidRequest(format!("Invalid commit: {}", e)));
}
};
if commit_did != *did {
return Err(ApiError::InvalidRepo(format!(
"CAR file is for DID {} but you are authenticated as {}",
commit_did, did
)));
}
let skip_verification = std::env::var("SKIP_IMPORT_VERIFICATION")
.ok()
.map(|v| v == "true" || v == "1")
@@ -106,13 +85,15 @@ pub async fn import_repo(
.map(|c| c.import.skip_verification)
.unwrap_or(false)
});
let is_migration = user.deactivated_at.is_some();
let is_migration = user.inbound_migration && user.deactivated_at.is_some();
if skip_verification {
warn!("Skipping all CAR verification for import (SKIP_IMPORT_VERIFICATION=true)");
} else if is_migration {
debug!("Verifying CAR file structure for migration (skipping signature verification)");
warn!("Skipping all CAR verification for repo import (SKIP_IMPORT_VERIFICATION=true)");
} else {
debug!(
"Verifying CAR file structure for repo import (skipping signature and DID verification)"
);
let verifier = CarVerifier::new();
match verifier.verify_car_structure_only(did, &root, &blocks) {
match verifier.verify_car_structure_only(&root, &blocks) {
Ok(verified) => {
debug!(
"CAR structure verification successful: rev={}, data_cid={}",
@@ -142,56 +123,6 @@ pub async fn import_repo(
)));
}
}
} else {
debug!("Verifying CAR file signature and structure for DID {}", did);
let verifier = CarVerifier::new();
match verifier.verify_car(did, &root, &blocks).await {
Ok(verified) => {
debug!(
"CAR verification successful: rev={}, data_cid={}",
verified.rev, verified.data_cid
);
}
Err(tranquil_pds::sync::verify::VerifyError::DidMismatch {
commit_did,
expected_did,
}) => {
return Err(ApiError::InvalidRepo(format!(
"CAR file is for DID {} but you are authenticated as {}",
commit_did, expected_did
)));
}
Err(tranquil_pds::sync::verify::VerifyError::InvalidSignature) => {
return Err(ApiError::InvalidRequest(
"CAR file commit signature verification failed".into(),
));
}
Err(tranquil_pds::sync::verify::VerifyError::DidResolutionFailed(msg)) => {
warn!("DID resolution failed during import verification: {}", msg);
return Err(ApiError::InvalidRequest(format!(
"Failed to verify DID: {}",
msg
)));
}
Err(tranquil_pds::sync::verify::VerifyError::NoSigningKey) => {
return Err(ApiError::InvalidRequest(
"DID document does not contain a signing key".into(),
));
}
Err(tranquil_pds::sync::verify::VerifyError::MstValidationFailed(msg)) => {
return Err(ApiError::InvalidRequest(format!(
"MST validation failed: {}",
msg
)));
}
Err(e) => {
error!("CAR verification error: {:?}", e);
return Err(ApiError::InvalidRequest(format!(
"CAR verification failed: {}",
e
)));
}
}
}
let max_blocks = tranquil_config::get().import.max_blocks as usize;
let _write_lock = state.repo_write_locks.lock(user_id).await;
@@ -417,7 +348,6 @@ async fn sequence_import_event(
rev: None,
};
let seq = state.repos.repo.insert_commit_event(&data).await?;
state.repos.repo.notify_update(seq).await?;
state.repos.repo.insert_commit_event(&data).await?;
Ok(())
}
+79 -64
View File
@@ -1,6 +1,6 @@
use super::validation::validate_record_with_status;
use super::validation_mode::{ValidationMode, deserialize_validation_mode};
use crate::repo::record::write::CommitInfo;
use crate::repo::record::write::{CommitInfo, ensure_record_type};
use axum::{Json, extract::State};
use jacquard_repo::{mst::Mst, storage::BlockStore};
use serde::{Deserialize, Serialize};
@@ -14,8 +14,8 @@ use tranquil_pds::auth::{
};
use tranquil_pds::repo::TrackingBlockStore;
use tranquil_pds::repo_ops::{
FinalizeParams, RecordOp, begin_repo_write, extract_backlinks, extract_blob_cids,
finalize_repo_write,
CommitResult, FinalizeParams, RecordOp, begin_repo_write, extract_backlinks, extract_blob_cids,
finalize_repo_write, with_repair_retry,
};
use tranquil_pds::state::AppState;
use tranquil_pds::types::{AtIdentifier, AtUri, Did, Nsid, Rkey};
@@ -27,7 +27,6 @@ struct WriteAccumulator {
mst: Mst<TrackingBlockStore>,
results: Vec<WriteResult>,
ops: Vec<RecordOp>,
modified_keys: Vec<String>,
all_blob_cids: Vec<String>,
backlinks_to_add: Vec<Backlink>,
backlinks_to_remove: Vec<AtUri>,
@@ -44,7 +43,6 @@ async fn process_single_write(
mst,
mut results,
mut ops,
mut modified_keys,
mut all_blob_cids,
mut backlinks_to_add,
mut backlinks_to_remove,
@@ -56,6 +54,8 @@ async fn process_single_write(
rkey,
value,
} => {
let value = ensure_record_type(value, collection);
let value = &*value;
let validation_status = if validate.should_skip() {
None
} else {
@@ -69,8 +69,19 @@ async fn process_single_write(
.await?,
)
};
all_blob_cids.extend(extract_blob_cids(value));
let rkey = rkey.clone().unwrap_or_else(Rkey::generate);
let key = format!("{}/{}", collection, rkey);
if mst
.get(&key)
.await
.map_err(|e| ApiError::from_mst_error("read MST for applyWrites create", &e))?
.is_some()
{
return Err(ApiError::InvalidRequest(format!(
"Record already exists at {key}"
)));
}
all_blob_cids.extend(extract_blob_cids(value));
let record_ipld = tranquil_pds::util::json_to_ipld(value);
let record_bytes = serde_ipld_dagcbor::to_vec(&record_ipld)
.map_err(|_| ApiError::InvalidRecord("Failed to serialize record".into()))?;
@@ -78,12 +89,10 @@ async fn process_single_write(
.put(&record_bytes)
.await
.map_err(|_| ApiError::InternalError(Some("Failed to store record".into())))?;
let key = format!("{}/{}", collection, rkey);
modified_keys.push(key.clone());
let new_mst = mst
.add(&key, record_cid)
.await
.map_err(|_| ApiError::InternalError(Some("Failed to add to MST".into())))?;
.map_err(|e| ApiError::from_mst_error("add record to MST", &e))?;
let uri = AtUri::from_parts(did, collection, &rkey);
backlinks_to_add.extend(extract_backlinks(&uri, value));
results.push(WriteResult::CreateResult {
@@ -100,7 +109,6 @@ async fn process_single_write(
mst: new_mst,
results,
ops,
modified_keys,
all_blob_cids,
backlinks_to_add,
backlinks_to_remove,
@@ -111,6 +119,8 @@ async fn process_single_write(
rkey,
value,
} => {
let value = ensure_record_type(value, collection);
let value = &*value;
let validation_status = if validate.should_skip() {
None
} else {
@@ -124,6 +134,14 @@ async fn process_single_write(
.await?,
)
};
let key = format!("{}/{}", collection, rkey);
let prev_record_cid = mst
.get(&key)
.await
.map_err(|e| ApiError::from_mst_error("read update target from MST", &e))?
.ok_or_else(|| {
ApiError::InvalidRequest("Update target record does not exist".into())
})?;
all_blob_cids.extend(extract_blob_cids(value));
let record_ipld = tranquil_pds::util::json_to_ipld(value);
let record_bytes = serde_ipld_dagcbor::to_vec(&record_ipld)
@@ -132,21 +150,10 @@ async fn process_single_write(
.put(&record_bytes)
.await
.map_err(|_| ApiError::InternalError(Some("Failed to store record".into())))?;
let key = format!("{}/{}", collection, rkey);
modified_keys.push(key.clone());
let prev_record_cid = mst
.get(&key)
.await
.map_err(|e| {
ApiError::InternalError(Some(format!("Failed to read prev record: {}", e)))
})?
.ok_or_else(|| {
ApiError::InvalidRequest("Update target record does not exist".into())
})?;
let new_mst = mst
.update(&key, record_cid)
.await
.map_err(|_| ApiError::InternalError(Some("Failed to update MST".into())))?;
.map_err(|e| ApiError::from_mst_error("update record in MST", &e))?;
let uri = AtUri::from_parts(did, collection, rkey);
backlinks_to_remove.push(uri.clone());
backlinks_to_add.extend(extract_backlinks(&uri, value));
@@ -165,7 +172,6 @@ async fn process_single_write(
mst: new_mst,
results,
ops,
modified_keys,
all_blob_cids,
backlinks_to_add,
backlinks_to_remove,
@@ -173,20 +179,17 @@ async fn process_single_write(
}
WriteOp::Delete { collection, rkey } => {
let key = format!("{}/{}", collection, rkey);
modified_keys.push(key.clone());
let prev_record_cid = mst
.get(&key)
.await
.map_err(|e| {
ApiError::InternalError(Some(format!("Failed to read prev record: {}", e)))
})?
.map_err(|e| ApiError::from_mst_error("read delete target from MST", &e))?
.ok_or_else(|| {
ApiError::InvalidRequest("Delete target record does not exist".into())
})?;
let new_mst = mst
.delete(&key)
.await
.map_err(|_| ApiError::InternalError(Some("Failed to delete from MST".into())))?;
.map_err(|e| ApiError::from_mst_error("delete record from MST", &e))?;
backlinks_to_remove.push(AtUri::from_parts(did, collection, rkey));
results.push(WriteResult::DeleteResult {});
ops.push(RecordOp::Delete {
@@ -198,7 +201,6 @@ async fn process_single_write(
mst: new_mst,
results,
ops,
modified_keys,
all_blob_cids,
backlinks_to_add,
backlinks_to_remove,
@@ -219,7 +221,6 @@ async fn process_writes(
mst: initial_mst,
results: Vec::new(),
ops: Vec::new(),
modified_keys: Vec::new(),
all_blob_cids: Vec::new(),
backlinks_to_add: Vec::new(),
backlinks_to_remove: Vec::new(),
@@ -231,6 +232,45 @@ async fn process_writes(
.await
}
async fn execute_apply_writes(
state: &AppState,
user_id: uuid::Uuid,
did: &Did,
input: &ApplyWritesInput,
controller_did: Option<&Did>,
write_summary: Option<serde_json::Value>,
) -> Result<(CommitResult, Vec<WriteResult>), ApiError> {
let (ctx, mst) = begin_repo_write(state, user_id, input.swap_commit.as_deref()).await?;
let WriteAccumulator {
mst: final_mst,
results,
ops,
all_blob_cids,
backlinks_to_add,
backlinks_to_remove,
} = process_writes(&input.writes, mst, did, input.validate, &ctx.tracking_store).await?;
let commit_result = finalize_repo_write(
state,
ctx,
final_mst,
FinalizeParams {
did,
user_id,
controller_did,
delegation_detail: write_summary,
ops,
blob_cids: &all_blob_cids,
backlinks_to_add,
backlinks_to_remove,
},
)
.await?;
Ok((commit_result, results))
}
#[derive(Deserialize)]
#[serde(tag = "$type")]
pub enum WriteOp {
@@ -345,25 +385,6 @@ pub async fn apply_writes(
.log_db_err("fetching user for batch write")?
.ok_or(ApiError::InternalError(Some("User not found".into())))?;
let (ctx, mst) = begin_repo_write(&state, user_id, input.swap_commit.as_deref()).await?;
let WriteAccumulator {
mst: final_mst,
results,
ops,
modified_keys,
all_blob_cids,
backlinks_to_add,
backlinks_to_remove,
} = process_writes(
&input.writes,
mst,
&did,
input.validate,
&ctx.tracking_store,
)
.await?;
let write_summary: Option<serde_json::Value> = controller_did.as_ref().map(|_| {
let writes: Vec<serde_json::Value> = input
.writes
@@ -397,22 +418,16 @@ pub async fn apply_writes(
})
});
let commit_result = finalize_repo_write(
&state,
ctx,
final_mst,
FinalizeParams {
did: &did,
let (commit_result, results) = with_repair_retry(&state, user_id, || {
execute_apply_writes(
&state,
user_id,
controller_did: controller_did.as_ref(),
delegation_detail: write_summary,
ops,
modified_keys: &modified_keys,
blob_cids: &all_blob_cids,
backlinks_to_add,
backlinks_to_remove,
},
)
&did,
&input,
controller_did.as_ref(),
write_summary.clone(),
)
})
.await?;
Ok(Json(ApplyWritesOutput {
+40 -23
View File
@@ -4,13 +4,15 @@ use cid::Cid;
use serde::{Deserialize, Serialize};
use serde_json::json;
use std::str::FromStr;
use tracing::error;
use tranquil_pds::api::error::ApiError;
use tranquil_pds::auth::{Active, Auth, VerifyScope};
use tranquil_pds::cid_types::RecordCid;
use tranquil_pds::repo_ops::{FinalizeParams, RecordOp, begin_repo_write, finalize_repo_write};
use tranquil_pds::repo_ops::{
FinalizeParams, RecordOp, begin_repo_write, finalize_repo_write, with_repair_retry,
};
use tranquil_pds::state::AppState;
use tranquil_pds::types::{AtIdentifier, AtUri, Nsid, Rkey};
use tranquil_pds::types::{AtIdentifier, AtUri, Did, Nsid, Rkey};
use uuid::Uuid;
#[derive(Deserialize)]
pub struct DeleteRecordInput {
@@ -41,13 +43,30 @@ pub async fn delete_record(
let user_id = repo_auth.user_id;
let controller_did = repo_auth.controller_did;
let (ctx, mst) = begin_repo_write(&state, user_id, input.swap_commit.as_deref()).await?;
let out = with_repair_retry(&state, user_id, || {
delete_record_inner(&state, &did, user_id, controller_did.as_ref(), &input)
})
.await?;
Ok(Json(out))
}
async fn delete_record_inner(
state: &AppState,
did: &Did,
user_id: Uuid,
controller_did: Option<&Did>,
input: &DeleteRecordInput,
) -> Result<DeleteRecordOutput, ApiError> {
let (ctx, mst) = begin_repo_write(state, user_id, input.swap_commit.as_deref()).await?;
let key = format!("{}/{}", input.collection, input.rkey);
if let Some(swap_record_str) = &input.swap_record {
let expected_cid = Cid::from_str(swap_record_str).ok();
let actual_cid = mst.get(&key).await.ok().flatten();
let actual_cid = mst
.get(&key)
.await
.map_err(|e| ApiError::from_mst_error("read swap target from MST", &e))?;
if expected_cid != actual_cid {
return Err(ApiError::InvalidSwap(Some(
"Record has been modified or does not exist".into(),
@@ -55,18 +74,18 @@ pub async fn delete_record(
}
}
let prev_record_cid = mst.get(&key).await.map_err(|e| {
error!("Failed to read prev record from MST: {}", e);
ApiError::InternalError(Some("Failed to read MST".into()))
})?;
let prev_record_cid = mst
.get(&key)
.await
.map_err(|e| ApiError::from_mst_error("read prev record from MST", &e))?;
let Some(prev_record_cid) = prev_record_cid else {
return Ok(Json(DeleteRecordOutput { commit: None }));
return Ok(DeleteRecordOutput { commit: None });
};
let new_mst = mst.delete(&key).await.map_err(|e| {
error!("Failed to delete from MST: {}", e);
ApiError::InternalError(Some("Failed to delete from MST".into()))
})?;
let new_mst = mst
.delete(&key)
.await
.map_err(|e| ApiError::from_mst_error("delete record from MST", &e))?;
let op = RecordOp::Delete {
collection: input.collection.clone(),
@@ -74,18 +93,17 @@ pub async fn delete_record(
prev: RecordCid::from(prev_record_cid),
};
let modified_keys = [key];
let deleted_uri = AtUri::from_parts(&did, &input.collection, &input.rkey);
let deleted_uri = AtUri::from_parts(did, &input.collection, &input.rkey);
let commit_result = finalize_repo_write(
&state,
state,
ctx,
new_mst,
FinalizeParams {
did: &did,
did,
user_id,
controller_did: controller_did.as_ref(),
delegation_detail: controller_did.as_ref().map(|_| {
controller_did,
delegation_detail: controller_did.map(|_| {
json!({
"action": "delete",
"collection": input.collection,
@@ -93,7 +111,6 @@ pub async fn delete_record(
})
}),
ops: vec![op],
modified_keys: &modified_keys,
blob_cids: &[],
backlinks_to_add: vec![],
backlinks_to_remove: vec![deleted_uri],
@@ -101,10 +118,10 @@ pub async fn delete_record(
)
.await?;
Ok(Json(DeleteRecordOutput {
Ok(DeleteRecordOutput {
commit: Some(CommitInfo {
cid: commit_result.commit_cid.to_string(),
rev: commit_result.rev,
}),
}))
})
}
+111 -63
View File
@@ -5,8 +5,8 @@ use cid::Cid;
use jacquard_repo::storage::BlockStore;
use serde::{Deserialize, Serialize};
use serde_json::json;
use std::borrow::Cow;
use std::str::FromStr;
use tracing::error;
use tranquil_pds::api::error::{ApiError, DbResultExt};
use tranquil_pds::auth::{
Active, Auth, AuthSource, RepoScopeAction, ScopeVerified, VerifyScope, require_not_migrated,
@@ -14,7 +14,7 @@ use tranquil_pds::auth::{
};
use tranquil_pds::repo_ops::{
FinalizeParams, RecordOp, begin_repo_write, extract_backlinks, extract_blob_cids,
finalize_repo_write,
finalize_repo_write, with_repair_retry,
};
use tranquil_pds::state::AppState;
use tranquil_pds::types::{AtIdentifier, AtUri, Did, Nsid, Rkey};
@@ -62,6 +62,29 @@ pub async fn prepare_repo_write<A: RepoScopeAction>(
})
}
pub(crate) fn ensure_record_type<'a>(
record: &'a serde_json::Value,
collection: &Nsid,
) -> Cow<'a, serde_json::Value> {
let serde_json::Value::Object(map) = record else {
return Cow::Borrowed(record);
};
let needs_fill = match map.get("$type") {
None | Some(serde_json::Value::Null) => true,
Some(serde_json::Value::String(existing)) => existing.is_empty(),
Some(_) => false,
};
if !needs_fill {
return Cow::Borrowed(record);
}
let mut map = map.clone();
map.insert(
"$type".to_string(),
serde_json::Value::String(collection.to_string()),
);
Cow::Owned(serde_json::Value::Object(map))
}
#[derive(Deserialize)]
#[allow(dead_code)]
pub struct CreateRecordInput {
@@ -95,15 +118,32 @@ pub struct CreateRecordOutput {
pub async fn create_record(
State(state): State<AppState>,
auth: Auth<Active>,
Json(input): Json<CreateRecordInput>,
Json(mut input): Json<CreateRecordInput>,
) -> Result<Json<CreateRecordOutput>, ApiError> {
if let Cow::Owned(record) = ensure_record_type(&input.record, &input.collection) {
input.record = record;
}
let scope_proof = auth.verify_repo_create(&input.collection)?;
let repo_auth = prepare_repo_write(&state, &scope_proof, &input.repo).await?;
let did = repo_auth.did;
let user_id = repo_auth.user_id;
let controller_did = repo_auth.controller_did;
let (ctx, mut mst) = begin_repo_write(&state, user_id, input.swap_commit.as_deref()).await?;
let out = with_repair_retry(&state, user_id, || {
create_record_inner(&state, &did, user_id, controller_did.as_ref(), &input)
})
.await?;
Ok(Json(out))
}
async fn create_record_inner(
state: &AppState,
did: &Did,
user_id: Uuid,
controller_did: Option<&Did>,
input: &CreateRecordInput,
) -> Result<CreateRecordOutput, ApiError> {
let (ctx, mut mst) = begin_repo_write(state, user_id, input.swap_commit.as_deref()).await?;
let validation_status = if input.validate.should_skip() {
None
@@ -119,12 +159,12 @@ pub async fn create_record(
)
};
let rkey = input.rkey.unwrap_or_else(Rkey::generate);
let rkey = input.rkey.clone().unwrap_or_else(Rkey::generate);
let mut ops: Vec<RecordOp> = Vec::new();
let mut conflict_uris_to_cleanup: Vec<AtUri> = Vec::new();
if !input.validate.should_skip() {
let record_uri = AtUri::from_parts(&did, &input.collection, &rkey);
let record_uri = AtUri::from_parts(did, &input.collection, &rkey);
let backlinks = extract_backlinks(&record_uri, &input.record);
if !backlinks.is_empty() {
@@ -149,24 +189,18 @@ pub async fn create_record(
Ok(Some(cid)) => cid,
Ok(None) => continue,
Err(e) => {
error!(
"Failed to read conflict record from MST {}: {:?}",
conflict_uri, e
);
return Err(ApiError::InternalError(Some(
"Failed to read conflicting record from MST".into(),
)));
return Err(ApiError::from_mst_error(
&format!("read conflict record from MST {conflict_uri}"),
&e,
));
}
};
mst = mst.delete(&conflict_key).await.map_err(|e| {
error!(
"Failed to delete conflict from MST {}: {:?}",
conflict_uri, e
);
ApiError::InternalError(Some(
"Failed to delete conflicting record from MST".into(),
))
ApiError::from_mst_error(
&format!("delete conflict from MST {conflict_uri}"),
&e,
)
})?;
ops.push(RecordOp::Delete {
@@ -179,6 +213,18 @@ pub async fn create_record(
}
}
let key = format!("{}/{}", input.collection, rkey);
if mst
.get(&key)
.await
.map_err(|e| ApiError::from_mst_error("read MST for create existence check", &e))?
.is_some()
{
return Err(ApiError::InvalidRequest(format!(
"Record already exists at {key}"
)));
}
let record_ipld = tranquil_pds::util::json_to_ipld(&input.record);
let record_bytes = serde_ipld_dagcbor::to_vec(&record_ipld)
.map_err(|_| ApiError::InvalidRecord("Failed to serialize record".into()))?;
@@ -187,12 +233,10 @@ pub async fn create_record(
.put(&record_bytes)
.await
.map_err(|_| ApiError::InternalError(Some("Failed to save record block".into())))?;
let key = format!("{}/{}", input.collection, rkey);
mst = mst
.add(&key, record_cid)
.await
.map_err(|_| ApiError::InternalError(Some("Failed to add to MST".into())))?;
.map_err(|e| ApiError::from_mst_error("add record to MST", &e))?;
ops.push(RecordOp::Create {
collection: input.collection.clone(),
@@ -200,34 +244,20 @@ pub async fn create_record(
cid: tranquil_pds::cid_types::RecordCid::from(record_cid),
});
let modified_keys: Vec<String> = ops
.iter()
.map(|op| match op {
RecordOp::Create {
collection, rkey, ..
}
| RecordOp::Update {
collection, rkey, ..
}
| RecordOp::Delete {
collection, rkey, ..
} => format!("{}/{}", collection, rkey),
})
.collect();
let blob_cids = extract_blob_cids(&input.record);
let created_uri = AtUri::from_parts(&did, &input.collection, &rkey);
let created_uri = AtUri::from_parts(did, &input.collection, &rkey);
let backlinks_to_add = extract_backlinks(&created_uri, &input.record);
let commit_result = finalize_repo_write(
&state,
state,
ctx,
mst,
FinalizeParams {
did: &did,
did,
user_id,
controller_did: controller_did.as_ref(),
delegation_detail: controller_did.as_ref().map(|_| {
controller_did,
delegation_detail: controller_did.map(|_| {
json!({
"action": "create",
"collection": input.collection,
@@ -235,7 +265,6 @@ pub async fn create_record(
})
}),
ops,
modified_keys: &modified_keys,
blob_cids: &blob_cids,
backlinks_to_add,
backlinks_to_remove: conflict_uris_to_cleanup,
@@ -243,7 +272,7 @@ pub async fn create_record(
)
.await?;
Ok(Json(CreateRecordOutput {
Ok(CreateRecordOutput {
uri: created_uri,
cid: record_cid.to_string(),
commit: CommitInfo {
@@ -251,7 +280,7 @@ pub async fn create_record(
rev: commit_result.rev,
},
validation_status,
}))
})
}
#[derive(Deserialize)]
@@ -283,15 +312,32 @@ pub struct PutRecordOutput {
pub async fn put_record(
State(state): State<AppState>,
auth: Auth<Active>,
Json(input): Json<PutRecordInput>,
Json(mut input): Json<PutRecordInput>,
) -> Result<Json<PutRecordOutput>, ApiError> {
if let Cow::Owned(record) = ensure_record_type(&input.record, &input.collection) {
input.record = record;
}
let upsert_proof = auth.verify_repo_upsert(&input.collection)?;
let repo_auth = prepare_repo_write(&state, &upsert_proof, &input.repo).await?;
let did = repo_auth.did;
let user_id = repo_auth.user_id;
let controller_did = repo_auth.controller_did;
let (ctx, mst) = begin_repo_write(&state, user_id, input.swap_commit.as_deref()).await?;
let out = with_repair_retry(&state, user_id, || {
put_record_inner(&state, &did, user_id, controller_did.as_ref(), &input)
})
.await?;
Ok(Json(out))
}
async fn put_record_inner(
state: &AppState,
did: &Did,
user_id: Uuid,
controller_did: Option<&Did>,
input: &PutRecordInput,
) -> Result<PutRecordOutput, ApiError> {
let (ctx, mst) = begin_repo_write(state, user_id, input.swap_commit.as_deref()).await?;
let validation_status = if input.validate.should_skip() {
None
@@ -309,9 +355,13 @@ pub async fn put_record(
let key = format!("{}/{}", input.collection, input.rkey);
let read_cid = |r: Result<Option<Cid>, jacquard_repo::error::RepoError>| {
r.map_err(|e| ApiError::from_mst_error("read MST for put", &e))
};
if let Some(swap_record_str) = &input.swap_record {
let expected_cid = Cid::from_str(swap_record_str).ok();
let actual_cid = mst.get(&key).await.ok().flatten();
let actual_cid = read_cid(mst.get(&key).await)?;
if expected_cid != actual_cid {
return Err(ApiError::InvalidSwap(Some(
"Record has been modified or does not exist".into(),
@@ -319,7 +369,7 @@ pub async fn put_record(
}
}
let existing_cid = mst.get(&key).await.ok().flatten();
let existing_cid = read_cid(mst.get(&key).await)?;
let record_ipld = tranquil_pds::util::json_to_ipld(&input.record);
let record_bytes = serde_ipld_dagcbor::to_vec(&record_ipld)
.map_err(|_| ApiError::InvalidRecord("Failed to serialize record".into()))?;
@@ -330,21 +380,21 @@ pub async fn put_record(
.map_err(|_| ApiError::InternalError(Some("Failed to save record block".into())))?;
if existing_cid == Some(record_cid) {
return Ok(Json(PutRecordOutput {
uri: AtUri::from_parts(&did, &input.collection, &input.rkey),
return Ok(PutRecordOutput {
uri: AtUri::from_parts(did, &input.collection, &input.rkey),
cid: record_cid.to_string(),
commit: None,
validation_status,
}));
});
}
let record_uri = AtUri::from_parts(&did, &input.collection, &input.rkey);
let record_uri = AtUri::from_parts(did, &input.collection, &input.rkey);
let (new_mst, op, is_update, backlinks_to_remove) = match existing_cid {
Some(prev_cid) => {
let new_mst = mst
.update(&key, record_cid)
.await
.map_err(|_| ApiError::InternalError(Some("Failed to update MST".into())))?;
.map_err(|e| ApiError::from_mst_error("update record in MST", &e))?;
let op = RecordOp::Update {
collection: input.collection.clone(),
rkey: input.rkey.clone(),
@@ -357,7 +407,7 @@ pub async fn put_record(
let new_mst = mst
.add(&key, record_cid)
.await
.map_err(|_| ApiError::InternalError(Some("Failed to add to MST".into())))?;
.map_err(|e| ApiError::from_mst_error("add record to MST", &e))?;
let op = RecordOp::Create {
collection: input.collection.clone(),
rkey: input.rkey.clone(),
@@ -367,19 +417,18 @@ pub async fn put_record(
}
};
let modified_keys = [key];
let blob_cids = extract_blob_cids(&input.record);
let backlinks_to_add = extract_backlinks(&record_uri, &input.record);
let commit_result = finalize_repo_write(
&state,
state,
ctx,
new_mst,
FinalizeParams {
did: &did,
did,
user_id,
controller_did: controller_did.as_ref(),
delegation_detail: controller_did.as_ref().map(|_| {
controller_did,
delegation_detail: controller_did.map(|_| {
json!({
"action": if is_update { "update" } else { "create" },
"collection": input.collection,
@@ -387,7 +436,6 @@ pub async fn put_record(
})
}),
ops: vec![op],
modified_keys: &modified_keys,
blob_cids: &blob_cids,
backlinks_to_add,
backlinks_to_remove,
@@ -395,7 +443,7 @@ pub async fn put_record(
)
.await?;
Ok(Json(PutRecordOutput {
Ok(PutRecordOutput {
uri: record_uri,
cid: record_cid.to_string(),
commit: Some(CommitInfo {
@@ -403,5 +451,5 @@ pub async fn put_record(
rev: commit_result.rev,
}),
validation_status,
}))
})
}
@@ -665,27 +665,22 @@ pub async fn delete_account(
error!("DB error deleting account: {:?}", e);
ApiError::InternalError(None)
})?;
let account_seq = tranquil_pds::repo_ops::sequence_account_event(
if let Err(e) = tranquil_pds::repo_ops::sequence_account_event(
&state,
did,
tranquil_db_traits::AccountStatus::Deleted,
)
.await;
match account_seq {
Ok(seq) => {
if let Err(e) = state.repos.repo.delete_sequences_except(did, seq).await {
warn!(
"Failed to cleanup sequences for deleted account {}: {}",
did, e
);
}
}
Err(e) => {
warn!(
"Failed to sequence account deletion event for {}: {}",
did, e
);
}
.await
{
warn!(
"Failed to sequence account deletion event for {}: {}",
did, e
);
} else if let Err(e) = state.repos.repo.purge_did_events_keeping_latest(did).await {
warn!(
"Failed to cleanup sequences for deleted account {}: {}",
did, e
);
}
let _ = state
.cache
+6 -4
View File
@@ -5,7 +5,7 @@ use tranquil_pds::BUILD_VERSION;
use tranquil_pds::state::AppState;
use tranquil_pds::util::{discord_app_id, discord_bot_username, telegram_bot_username};
fn get_available_comms_channels() -> Vec<CommsChannel> {
async fn get_available_comms_channels(state: &AppState) -> Vec<CommsChannel> {
let cfg = tranquil_config::get();
let mut channels = vec![CommsChannel::Email];
if cfg.discord.bot_token.is_some() {
@@ -14,7 +14,9 @@ fn get_available_comms_channels() -> Vec<CommsChannel> {
if cfg.telegram.bot_token.is_some() {
channels.push(CommsChannel::Telegram);
}
if cfg.signal.enabled {
if let Some(slot) = &state.signal_sender
&& slot.is_linked().await
{
channels.push(CommsChannel::Signal);
}
channels
@@ -66,7 +68,7 @@ pub struct DescribeServerOutput {
pub telegram_bot_username: Option<String>,
}
pub async fn describe_server() -> Json<DescribeServerOutput> {
pub async fn describe_server(State(state): State<AppState>) -> Json<DescribeServerOutput> {
let cfg = tranquil_config::get();
let pds_hostname = &cfg.server.hostname;
@@ -82,7 +84,7 @@ pub async fn describe_server() -> Json<DescribeServerOutput> {
email: cfg.server.contact_email.clone(),
},
version: BUILD_VERSION,
available_comms_channels: get_available_comms_channels(),
available_comms_channels: get_available_comms_channels(&state).await,
self_hosted_did_web_enabled: is_self_hosted_did_web_enabled(),
discord_bot_username: discord_bot_username().map(String::from),
discord_app_id: discord_app_id().map(String::from),
+1
View File
@@ -51,6 +51,7 @@ pub use session::{
auto_resend_verification, confirm_signup, create_session, delete_session,
get_legacy_login_preference, get_session, list_sessions, refresh_session, resend_verification,
revoke_all_sessions, revoke_session, update_legacy_login_preference, update_locale,
verification_blocks_login,
};
pub use signing_key::reserve_signing_key;
pub use totp::{
+11 -8
View File
@@ -13,10 +13,6 @@ use tranquil_pds::state::AppState;
use tranquil_pds::types::PlainPassword;
use tranquil_pds::validation::validate_password;
fn generate_reset_code() -> String {
tranquil_pds::util::generate_token_code()
}
#[derive(Deserialize)]
pub struct RequestPasswordResetInput {
#[serde(alias = "identifier")]
@@ -70,12 +66,13 @@ pub async fn request_password_reset(
return Err(ApiError::InternalError(None));
}
};
let code = generate_reset_code();
let display_code = tranquil_pds::util::generate_token_code();
let stored_code = tranquil_pds::util::normalize_token_code(&display_code);
let expires_at = Utc::now() + Duration::minutes(10);
if let Err(e) = state
.repos
.user
.set_password_reset_code(user_id, &code, expires_at)
.set_password_reset_code(user_id, &stored_code, expires_at)
.await
{
error!("DB error setting reset code: {:?}", e);
@@ -86,7 +83,7 @@ pub async fn request_password_reset(
state.repos.user.as_ref(),
state.repos.infra.as_ref(),
user_id,
&code,
&display_code,
hostname,
)
.await
@@ -133,7 +130,13 @@ pub async fn reset_password(
if let Err(e) = validate_password(password) {
return Err(ApiError::InvalidRequest(e.to_string()));
}
let user = match state.repos.user.get_user_by_reset_code(token).await {
let normalized_token = tranquil_pds::util::normalize_token_code(token);
let user = match state
.repos
.user
.get_user_by_reset_code(&normalized_token)
.await
{
Ok(Some(u)) => u,
Ok(None) => {
return Err(ApiError::InvalidToken(None));
+33 -5
View File
@@ -8,18 +8,28 @@ use bcrypt::verify;
use serde::{Deserialize, Serialize};
use serde_json::json;
use tracing::{error, info, warn};
use tranquil_db_traits::{SessionId, TokenFamilyId};
use tranquil_db_traits::{ChannelVerificationStatus, SessionId, TokenFamilyId};
use tranquil_pds::api::error::{ApiError, DbResultExt};
use tranquil_pds::api::{EmptyResponse, PreferredLocaleOutput, SuccessResponse};
use tranquil_pds::auth::{
Active, Auth, NormalizedLoginIdentifier, Permissive, require_legacy_session_mfa,
require_reauth_window,
};
use tranquil_pds::rate_limit::{LoginLimit, RateLimited, RefreshSessionLimit};
use tranquil_pds::rate_limit::{
LoginLimit, RateLimited, RefreshSessionLimit, TotpVerifyLimit,
check_user_rate_limit_with_message,
};
use tranquil_pds::state::AppState;
use tranquil_pds::types::{AccountState, Did, Handle, PlainPassword};
use tranquil_types::TokenId;
pub fn verification_blocks_login(channel_verification: &ChannelVerificationStatus) -> bool {
!tranquil_config::get()
.server
.disable_account_verification_gate
&& !channel_verification.has_any_verified()
}
#[derive(Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct CreateSessionInput {
@@ -129,14 +139,13 @@ pub async fn create_session(
warn!("Login attempt for takendown account: {}", row.did);
return Err(ApiError::AccountTakedown);
}
let is_verified = row.channel_verification.has_any_verified();
let is_delegated = state
.repos
.delegation
.is_delegated_account(&row.did)
.await
.unwrap_or(false);
if !is_verified && !is_delegated {
if verification_blocks_login(&row.channel_verification) && !is_delegated {
warn!("Login attempt for unverified account: {}", row.did);
let resend_info = auto_resend_verification(&state, &row.did).await;
let handle = resend_info
@@ -162,16 +171,35 @@ pub async fn create_session(
let has_totp = row.totp_enabled;
let email_2fa_enabled = row.email_2fa_enabled;
let is_legacy_login = has_totp || email_2fa_enabled;
let used_totp_factor = tranquil_pds::auth::legacy_2fa::used_totp_factor(
has_totp,
input.auth_factor_token.as_deref(),
);
let twofa_ctx = tranquil_pds::auth::legacy_2fa::Legacy2faContext {
is_app_password: app_password_name.is_some(),
email_2fa_enabled,
has_totp,
allow_legacy_login: row.allow_legacy_login,
};
if let Some(token) = input.auth_factor_token.as_deref()
&& has_totp
&& tranquil_pds::auth::legacy_2fa::looks_like_totp_token(token)
{
check_user_rate_limit_with_message::<TotpVerifyLimit>(
&state,
row.did.as_str(),
"Too many verification attempts. Please try again in a few minutes.",
)
.await?;
}
match tranquil_pds::auth::legacy_2fa::process_legacy_2fa(
state.cache.as_ref(),
&row.did,
&twofa_ctx,
input.auth_factor_token.as_deref(),
async |t: &str| {
crate::server::totp::verify_totp_or_backup_for_user(&state, &row.did, t).await
},
)
.await
{
@@ -278,7 +306,7 @@ pub async fn create_session(
error!("Failed to insert session: {:?}", e);
return Err(ApiError::InternalError(None));
}
if is_legacy_login {
if is_legacy_login && !used_totp_factor {
warn!(
did = %row.did,
ip = %client_ip,
+5 -5
View File
@@ -35,7 +35,7 @@ pub fn create_access_token_with_scope_metadata(
scope,
TokenType::Access,
key_bytes,
Duration::minutes(15),
Duration::minutes(120),
hostname,
)
}
@@ -54,7 +54,7 @@ pub fn create_access_token_with_delegation(
scope,
TokenType::Access,
key_bytes,
Duration::minutes(15),
Duration::minutes(120),
act,
hostname,
)
@@ -69,7 +69,7 @@ pub fn create_refresh_token_with_metadata(
TokenScope::Refresh.as_str(),
TokenType::Refresh,
key_bytes,
Duration::days(14),
Duration::days(90),
None,
)
}
@@ -198,7 +198,7 @@ pub fn create_access_token_hs256_with_metadata(
TokenScope::Access.as_str(),
TokenType::Access,
secret,
Duration::minutes(15),
Duration::minutes(120),
)
}
@@ -211,7 +211,7 @@ pub fn create_refresh_token_hs256_with_metadata(
TokenScope::Refresh.as_str(),
TokenType::Refresh,
secret,
Duration::days(14),
Duration::days(90),
)
}
+14 -1
View File
@@ -15,7 +15,9 @@ impl TokenType {
match self {
Self::Access => "at+jwt",
Self::Refresh => "refresh+jwt",
Self::Service => "jwt",
// RFC 7519 §5.1 recommends the uppercase "JWT".
// and for atproto inter-service auth its a requirement.
Self::Service => "JWT",
}
}
}
@@ -290,6 +292,17 @@ mod tests {
assert!(TokenType::from_str("bearer").is_err());
}
#[test]
fn service_token_header_serializes_typ_as_uppercase_jwt() {
// RFC 7519 §5.1 recommends the JWT `typ` header value be uppercase "JWT".
let header = Header {
alg: SigningAlgorithm::ES256K,
typ: TokenType::Service,
};
let json = serde_json::to_string(&header).expect("serialize header");
assert!(json.contains(r#""typ":"JWT""#), "got {json}");
}
#[test]
fn signing_algorithm_case_insensitive() {
assert_eq!(
+11
View File
@@ -10,7 +10,14 @@ tranquil-signal = { workspace = true }
async-trait = { workspace = true }
base64 = { workspace = true }
ed25519-dalek = { workspace = true }
futures = { workspace = true }
hickory-resolver = { workspace = true }
lettre = { workspace = true }
rand = { workspace = true }
reqwest = { workspace = true }
rsa = { workspace = true }
secrecy = { workspace = true }
serde_json = { workspace = true }
sqlx = { workspace = true }
thiserror = { workspace = true }
@@ -18,3 +25,7 @@ tokio = { workspace = true }
tracing = { workspace = true }
tranquil-db-traits = { workspace = true }
uuid = { workspace = true }
[dev-dependencies]
chrono = { workspace = true }
tokio = { workspace = true, features = ["macros", "rt-multi-thread", "time", "io-util", "net"] }
+227
View File
@@ -0,0 +1,227 @@
use std::fs;
use base64::Engine;
use base64::engine::general_purpose::STANDARD as BASE64_STANDARD;
use ed25519_dalek::pkcs8::DecodePrivateKey as _;
use lettre::Message;
use lettre::message::dkim::{
DkimCanonicalization, DkimCanonicalizationType, DkimConfig as LettreDkimConfig,
DkimSigningAlgorithm, DkimSigningKey,
};
use lettre::message::header::HeaderName;
use rsa::pkcs1::EncodeRsaPrivateKey;
use rsa::pkcs8::LineEnding;
use super::types::{DkimKeyPath, DkimSelector, EmailDomain};
use crate::sender::SendError;
const SIGNED_HEADERS: &[&str] = &[
"From",
"Sender",
"Reply-To",
"To",
"Cc",
"Subject",
"Date",
"In-Reply-To",
"References",
"MIME-Version",
"Content-Type",
"Content-Transfer-Encoding",
];
pub struct DkimSigner {
config: LettreDkimConfig,
}
impl DkimSigner {
pub fn load(
selector: DkimSelector,
domain: EmailDomain,
path: DkimKeyPath,
) -> Result<Self, SendError> {
let pem = fs::read_to_string(path.as_path()).map_err(|e| {
SendError::DkimSign(format!("read DKIM key {}: {e}", path.as_path().display()))
})?;
Self::from_pem(selector, domain, &pem)
}
pub fn from_pem(
selector: DkimSelector,
domain: EmailDomain,
pem: &str,
) -> Result<Self, SendError> {
let key = parse_key(pem)?;
let canonicalization = DkimCanonicalization {
header: DkimCanonicalizationType::Relaxed,
body: DkimCanonicalizationType::Relaxed,
};
let headers = SIGNED_HEADERS
.iter()
.copied()
.map(HeaderName::new_from_ascii_str)
.collect();
let config = LettreDkimConfig::new(
selector.into_inner(),
domain.into_inner(),
key,
headers,
canonicalization,
);
Ok(Self { config })
}
pub fn sign(&self, message: &mut Message) {
message.sign(&self.config);
}
}
impl std::fmt::Debug for DkimSigner {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("DkimSigner")
}
}
fn parse_key(input: &str) -> Result<DkimSigningKey, SendError> {
let trimmed = input.trim_start();
match trimmed {
s if s.starts_with("-----BEGIN RSA PRIVATE KEY-----") => {
DkimSigningKey::new(input, DkimSigningAlgorithm::Rsa)
.map_err(|e| SendError::DkimSign(format!("RSA PKCS#1 PEM rejected: {e}")))
}
s if s.starts_with("-----BEGIN PRIVATE KEY-----") => parse_pkcs8(input),
s if s.starts_with("-----BEGIN") => Err(SendError::DkimSign(
"unrecognized PEM type; expected an RSA or Ed25519 private key".to_string(),
)),
_ => DkimSigningKey::new(input.trim(), DkimSigningAlgorithm::Ed25519).map_err(|e| {
SendError::DkimSign(format!(
"expected base64-encoded 32-byte Ed25519 seed or a PEM-wrapped key: {e}"
))
}),
}
}
fn parse_pkcs8(pem: &str) -> Result<DkimSigningKey, SendError> {
let ed25519_err = match ed25519_dalek::SigningKey::from_pkcs8_pem(pem) {
Ok(key) => {
let seed = BASE64_STANDARD.encode(key.to_bytes());
return DkimSigningKey::new(&seed, DkimSigningAlgorithm::Ed25519)
.map_err(|e| SendError::DkimSign(format!("re-import Ed25519 seed: {e}")));
}
Err(e) => e,
};
let rsa_err = match rsa::RsaPrivateKey::from_pkcs8_pem(pem) {
Ok(key) => {
let pkcs1 = key
.to_pkcs1_pem(LineEnding::LF)
.map_err(|e| SendError::DkimSign(format!("re-encode RSA PKCS#8 as PKCS#1: {e}")))?;
return DkimSigningKey::new(pkcs1.as_str(), DkimSigningAlgorithm::Rsa)
.map_err(|e| SendError::DkimSign(format!("re-import RSA PKCS#1: {e}")));
}
Err(e) => e,
};
Err(SendError::DkimSign(format!(
"PKCS#8 PEM rejected by both parsers; ed25519: {ed25519_err}; rsa: {rsa_err}"
)))
}
#[cfg(test)]
mod tests {
use super::*;
use ed25519_dalek::pkcs8::EncodePrivateKey as _;
use lettre::message::Mailbox;
use lettre::message::header::ContentType;
use rsa::pkcs1::DecodeRsaPrivateKey as _;
const ED25519_RAW_SEED_B64: &str = "QkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkI=";
const RSA_PKCS1_PEM: &str = include_str!("test_fixtures/rsa2048-priv-pkcs1.pem");
fn ed25519_pkcs8_pem() -> String {
let key = ed25519_dalek::SigningKey::from_bytes(&[7u8; 32]);
key.to_pkcs8_pem(LineEnding::LF).unwrap().to_string()
}
fn rsa_pkcs8_pem() -> String {
let key = rsa::RsaPrivateKey::from_pkcs1_pem(RSA_PKCS1_PEM).unwrap();
key.to_pkcs8_pem(LineEnding::LF).unwrap().to_string()
}
fn signer(pem: &str) -> DkimSigner {
DkimSigner::from_pem(
DkimSelector::parse("default").unwrap(),
EmailDomain::parse("nel.pet").unwrap(),
pem,
)
.expect("key should load")
}
fn signed_headers(signer: &DkimSigner) -> String {
let from: Mailbox = "sender@nel.pet".parse().unwrap();
let to: Mailbox = "recipient@nel.pet".parse().unwrap();
let mut message = Message::builder()
.from(from)
.to(to)
.subject("Roundtrip")
.header(ContentType::TEXT_PLAIN)
.body("Body".to_string())
.unwrap();
signer.sign(&mut message);
String::from_utf8(message.formatted()).unwrap()
}
#[test]
fn rejects_garbage() {
assert!(matches!(
parse_key("not a key"),
Err(SendError::DkimSign(_))
));
}
#[test]
fn rejects_unknown_pem_type() {
let pem = "-----BEGIN OPENSSH PRIVATE KEY-----\nx\n-----END OPENSSH PRIVATE KEY-----\n";
match parse_key(pem) {
Err(SendError::DkimSign(msg)) => assert!(msg.contains("unrecognized"), "msg: {msg}"),
other => panic!("expected unrecognized PEM error, got {other:?}"),
}
}
#[test]
fn ed25519_raw_seed_signs() {
let raw = signed_headers(&signer(ED25519_RAW_SEED_B64));
assert_signed_with(&raw, "a=ed25519-sha256");
}
#[test]
fn ed25519_pkcs8_pem_signs() {
let raw = signed_headers(&signer(&ed25519_pkcs8_pem()));
assert_signed_with(&raw, "a=ed25519-sha256");
}
#[test]
fn rsa_pkcs1_pem_signs() {
let raw = signed_headers(&signer(RSA_PKCS1_PEM));
assert_signed_with(&raw, "a=rsa-sha256");
}
#[test]
fn rsa_pkcs8_pem_signs() {
let raw = signed_headers(&signer(&rsa_pkcs8_pem()));
assert_signed_with(&raw, "a=rsa-sha256");
}
fn assert_signed_with(raw: &str, algorithm: &str) {
assert!(
raw.contains("DKIM-Signature:"),
"no signature header: {raw}"
);
assert!(raw.contains(algorithm), "missing {algorithm}: {raw}");
assert!(
raw.contains("c=relaxed/relaxed"),
"expected relaxed/relaxed canonicalization: {raw}"
);
}
}
+151
View File
@@ -0,0 +1,151 @@
use lettre::Message;
use lettre::message::Mailbox;
use lettre::message::header::ContentType;
use uuid::Uuid;
use super::types::EmailDomain;
use crate::sender::SendError;
use crate::types::QueuedComms;
pub(super) fn build(from: &Mailbox, qc: &QueuedComms) -> Result<Message, SendError> {
let to: Mailbox = qc
.recipient
.parse()
.map_err(|e: lettre::address::AddressError| SendError::InvalidRecipient(e.to_string()))?;
let subject = qc.subject.as_deref().unwrap_or("Notification");
let message_id = format!("<{}@{}>", Uuid::new_v4(), from.email.domain());
Message::builder()
.from(from.clone())
.to(to)
.subject(subject)
.message_id(Some(message_id))
.header(ContentType::TEXT_PLAIN)
.body(qc.body.clone())
.map_err(|e| SendError::MessageBuild(e.to_string()))
}
pub(super) fn recipient_domain(message: &Message) -> Result<EmailDomain, SendError> {
let envelope = message.envelope();
let first = envelope
.to()
.first()
.ok_or_else(|| SendError::MessageBuild("envelope has no recipients".to_string()))?;
EmailDomain::parse(first.domain())
.map_err(|e| SendError::InvalidRecipient(format!("invalid recipient domain: {e}")))
}
#[cfg(test)]
mod tests {
use super::*;
use crate::types::{CommsChannel, CommsStatus, CommsType};
use chrono::Utc;
use uuid::Uuid;
fn from_mailbox() -> Mailbox {
"Test Sender <noreply@nel.pet>".parse().unwrap()
}
fn fixture(recipient: &str, subject: Option<&str>, body: &str) -> QueuedComms {
QueuedComms {
id: Uuid::new_v4(),
user_id: None,
channel: CommsChannel::Email,
comms_type: CommsType::Welcome,
status: CommsStatus::Pending,
recipient: recipient.to_string(),
subject: subject.map(String::from),
body: body.to_string(),
metadata: None,
attempts: 0,
max_attempts: 3,
last_error: None,
created_at: Utc::now(),
updated_at: Utc::now(),
scheduled_for: Utc::now(),
processed_at: None,
}
}
#[test]
fn build_basic_message() {
let msg = build(
&from_mailbox(),
&fixture("user@nel.pet", Some("Welcome"), "Hello world."),
)
.unwrap();
let raw = String::from_utf8(msg.formatted()).unwrap();
let lower = raw.to_lowercase();
assert!(raw.contains("From: \"Test Sender\" <noreply@nel.pet>"));
assert!(raw.contains("To: user@nel.pet"));
assert!(raw.contains("Subject: Welcome"));
assert!(lower.contains("content-type: text/plain"));
assert!(raw.contains("Hello world."));
}
#[test]
fn utf8_subject_is_encoded() {
let msg = build(
&from_mailbox(),
&fixture("user@nel.pet", Some("héllo wörld"), "Body"),
)
.unwrap();
let raw = String::from_utf8(msg.formatted()).unwrap();
assert!(raw.contains("=?utf-8?"));
assert!(!raw.contains("héllo"));
}
#[test]
fn header_injection_rejected() {
let result = build(
&from_mailbox(),
&fixture("x@nel.pet\r\nBcc: evil@x", Some("s"), "b"),
);
assert!(matches!(result, Err(SendError::InvalidRecipient(_))));
}
#[test]
fn subject_crlf_does_not_inject_headers() {
let msg = build(
&from_mailbox(),
&fixture("user@nel.pet", Some("hi\r\nBcc: evil@nel.pet"), "body"),
)
.expect("subject CRLF should be encoded, not rejected");
let raw = String::from_utf8(msg.formatted()).unwrap();
assert!(
!raw.contains("Bcc:"),
"CRLF in subject must not produce a Bcc header: {raw}"
);
assert!(
raw.contains("Subject: ="),
"subject with non-printable chars should be RFC 2047 encoded: {raw}"
);
}
#[test]
fn message_id_uses_from_domain() {
let msg = build(&from_mailbox(), &fixture("user@nel.pet", Some("s"), "b")).unwrap();
let raw = String::from_utf8(msg.formatted()).unwrap();
let line = raw
.lines()
.find(|l| l.starts_with("Message-ID:") || l.starts_with("Message-Id:"))
.expect("message-id header present");
assert!(
line.contains("@nel.pet>"),
"message-id should use From domain: {line}"
);
}
#[test]
fn missing_subject_uses_default() {
let msg = build(&from_mailbox(), &fixture("user@nel.pet", None, "Body")).unwrap();
let raw = String::from_utf8(msg.formatted()).unwrap();
assert!(raw.contains("Subject: Notification"));
}
#[test]
fn recipient_domain_extracted() {
let msg = build(&from_mailbox(), &fixture("user@Nel.PET", Some("s"), "b")).unwrap();
let d = recipient_domain(&msg).unwrap();
assert_eq!(d.as_str(), "nel.pet");
}
}
+194
View File
@@ -0,0 +1,194 @@
pub mod dkim;
pub mod message;
mod mx;
pub mod transport;
pub mod types;
use std::sync::Arc;
use std::time::Duration;
use async_trait::async_trait;
use hickory_resolver::TokioAsyncResolver;
use lettre::message::Mailbox;
use lettre::transport::smtp::AsyncSmtpTransport;
use lettre::transport::smtp::PoolConfig;
use lettre::transport::smtp::authentication::Credentials;
use lettre::transport::smtp::extension::ClientId;
use tokio::sync::Semaphore;
use tracing::{info, warn};
pub use self::dkim::DkimSigner;
pub use self::transport::SendMode;
use self::types::{
DkimKeyPath, DkimSelector, EmailDomain, HeloName, SmtpHost, SmtpPassword, SmtpPort,
SmtpUsername, TlsMode,
};
use crate::sender::{CommsSender, SendError};
use crate::types::{CommsChannel, QueuedComms};
pub struct EmailSender {
from: Mailbox,
mode: SendMode,
dkim: Option<DkimSigner>,
}
impl EmailSender {
pub fn new(from: Mailbox, mode: SendMode, dkim: Option<DkimSigner>) -> Self {
Self { from, mode, dkim }
}
pub fn from_config(cfg: &tranquil_config::TranquilConfig) -> Result<Option<Self>, SendError> {
let Some(from_address) = cfg.email.from_address.as_deref().filter(|s| !s.is_empty()) else {
info!("Email sender disabled: MAIL_FROM_ADDRESS unset");
return Ok(None);
};
let from = build_from(&cfg.email.from_name, from_address)?;
let dkim = build_dkim(&cfg.email.dkim)?;
let mode = match cfg
.email
.smarthost
.host
.as_deref()
.filter(|h| !h.is_empty())
{
Some(host) => build_smarthost(cfg, host)?,
None => build_direct_mx(cfg)?,
};
info!(?mode, dkim = dkim.is_some(), "Email sender initialized");
Ok(Some(Self { from, mode, dkim }))
}
}
fn config_invalid(field: &str, error: impl std::fmt::Display) -> SendError {
SendError::ConfigInvalid(format!("{field}: {error}"))
}
fn build_from(from_name: &str, from_address: &str) -> Result<Mailbox, SendError> {
let raw = match from_name.is_empty() {
true => from_address.to_string(),
false => format!("\"{}\" <{}>", from_name.replace('"', "'"), from_address),
};
raw.parse::<Mailbox>()
.map_err(|e| config_invalid("MAIL_FROM_ADDRESS / MAIL_FROM_NAME", e))
}
fn build_smarthost(
cfg: &tranquil_config::TranquilConfig,
host_raw: &str,
) -> Result<SendMode, SendError> {
let host = SmtpHost::parse(host_raw).map_err(|e| config_invalid("MAIL_SMARTHOST_HOST", e))?;
let port = SmtpPort::parse(cfg.email.smarthost.port)
.map_err(|e| config_invalid("MAIL_SMARTHOST_PORT", e))?;
let tls = TlsMode::parse(&cfg.email.smarthost.tls)
.map_err(|e| config_invalid("MAIL_SMARTHOST_TLS", e))?;
let helo = resolve_helo(cfg)?;
let pool = PoolConfig::new()
.max_size(cfg.email.smarthost.pool_size)
.idle_timeout(Duration::from_secs(60));
let command_timeout = Duration::from_secs(cfg.email.smarthost.command_timeout_secs);
let total_timeout = Duration::from_secs(cfg.email.smarthost.total_timeout_secs);
let builder = match tls {
TlsMode::Implicit => AsyncSmtpTransport::<lettre::Tokio1Executor>::relay(host.as_str())
.map_err(|e| config_invalid("smarthost TLS setup", e))?,
TlsMode::Starttls => {
AsyncSmtpTransport::<lettre::Tokio1Executor>::starttls_relay(host.as_str())
.map_err(|e| config_invalid("smarthost TLS setup", e))?
}
TlsMode::None => {
AsyncSmtpTransport::<lettre::Tokio1Executor>::builder_dangerous(host.as_str())
}
};
let builder = builder
.port(port.as_u16())
.hello_name(ClientId::Domain(helo.into_inner()))
.timeout(Some(command_timeout))
.pool_config(pool);
let builder = match (
cfg.email.smarthost.username.as_deref(),
cfg.email.smarthost.password.as_deref(),
) {
(Some(u), Some(p)) => {
let username =
SmtpUsername::parse(u).map_err(|e| config_invalid("MAIL_SMARTHOST_USERNAME", e))?;
let password =
SmtpPassword::parse(p).map_err(|e| config_invalid("MAIL_SMARTHOST_PASSWORD", e))?;
builder.credentials(Credentials::new(
username.into_inner(),
password.expose().to_string(),
))
}
_ => builder,
};
Ok(SendMode::Smarthost {
transport: Box::new(builder.build()),
total_timeout,
})
}
fn build_direct_mx(cfg: &tranquil_config::TranquilConfig) -> Result<SendMode, SendError> {
let helo = resolve_helo(cfg)?;
let resolver = TokioAsyncResolver::tokio_from_system_conf()
.map(Arc::new)
.map_err(|e| config_invalid("system DNS configuration", e))?;
let max_concurrent = cfg.email.direct_mx.max_concurrent_sends.max(1);
Ok(SendMode::DirectMx {
resolver,
helo,
command_timeout: Duration::from_secs(cfg.email.direct_mx.command_timeout_secs),
total_timeout: Duration::from_secs(cfg.email.direct_mx.total_timeout_secs),
require_tls: cfg.email.direct_mx.require_tls,
inflight: Arc::new(Semaphore::new(max_concurrent)),
})
}
fn resolve_helo(cfg: &tranquil_config::TranquilConfig) -> Result<HeloName, SendError> {
let raw = cfg
.email
.helo_name
.clone()
.unwrap_or_else(|| cfg.server.hostname_without_port().to_string());
HeloName::parse(&raw).map_err(|e| config_invalid(&format!("HELO name {raw:?}"), e))
}
fn build_dkim(cfg: &tranquil_config::DkimConfig) -> Result<Option<DkimSigner>, SendError> {
let selector = match cfg.selector.as_deref() {
Some(s) => s,
None => return Ok(None),
};
let domain = cfg
.domain
.as_deref()
.ok_or_else(|| SendError::DkimSign("MAIL_DKIM_DOMAIN required when selector set".into()))?;
let key_path = cfg.private_key_path.as_deref().ok_or_else(|| {
SendError::DkimSign("MAIL_DKIM_KEY_PATH required when selector set".into())
})?;
let selector = DkimSelector::parse(selector)
.map_err(|e| SendError::DkimSign(format!("invalid DKIM selector: {e}")))?;
let domain = EmailDomain::parse(domain)
.map_err(|e| SendError::DkimSign(format!("invalid DKIM domain: {e}")))?;
let path = DkimKeyPath::parse(key_path)
.map_err(|e| SendError::DkimSign(format!("DKIM key path invalid: {e}")))?;
DkimSigner::load(selector, domain, path).map(Some)
}
#[async_trait]
impl CommsSender for EmailSender {
fn channel(&self) -> CommsChannel {
CommsChannel::Email
}
async fn send(&self, notification: &QueuedComms) -> Result<(), SendError> {
let mut message = message::build(&self.from, notification)?;
if let Some(signer) = &self.dkim {
signer.sign(&mut message);
}
match transport::dispatch(&self.mode, message).await {
Ok(()) => Ok(()),
Err(e) => {
warn!(comms_id = %notification.id, error = %e, "SMTP send failed");
Err(e)
}
}
}
}
+250
View File
@@ -0,0 +1,250 @@
use hickory_resolver::TokioAsyncResolver;
use hickory_resolver::error::{ResolveError, ResolveErrorKind};
use hickory_resolver::proto::op::ResponseCode;
use rand::seq::SliceRandom;
use super::types::{EmailDomain, MxHost, MxPriority, MxRecord};
use crate::sender::SendError;
pub async fn resolve(
resolver: &TokioAsyncResolver,
domain: &EmailDomain,
) -> Result<Vec<MxRecord>, SendError> {
match resolver.mx_lookup(domain.as_str()).await {
Ok(lookup) => interpret_lookup(
lookup
.iter()
.map(|mx| (mx.preference(), mx.exchange().clone())),
domain,
),
Err(e) => classify_lookup_error(e, domain),
}
}
fn interpret_lookup(
items: impl IntoIterator<Item = (u16, hickory_resolver::Name)>,
domain: &EmailDomain,
) -> Result<Vec<MxRecord>, SendError> {
let entries: Vec<_> = items.into_iter().collect();
match entries.iter().any(|(_, name)| name.is_root()) {
true => Err(SendError::DnsPermanent(format!(
"null MX record at {}: domain refuses mail",
domain.as_str()
))),
false => {
let records: Vec<MxRecord> = entries
.into_iter()
.filter_map(|(prio, name)| {
MxHost::parse(&name.to_utf8()).ok().map(|host| MxRecord {
priority: MxPriority::new(prio),
host,
})
})
.collect();
match records.is_empty() {
true => implicit_mx(domain),
false => Ok(prioritize(records)),
}
}
}
}
fn prioritize(mut records: Vec<MxRecord>) -> Vec<MxRecord> {
records.shuffle(&mut rand::thread_rng());
records.sort_by_key(|r| r.priority);
records
}
fn classify_lookup_error(
e: ResolveError,
domain: &EmailDomain,
) -> Result<Vec<MxRecord>, SendError> {
match e.kind() {
ResolveErrorKind::NoRecordsFound { response_code, .. } => match *response_code {
ResponseCode::NoError => implicit_mx(domain),
ResponseCode::NXDomain => Err(SendError::DnsPermanent(format!(
"domain {} does not exist",
domain.as_str()
))),
other => Err(SendError::DnsTransient(format!(
"MX lookup for {} failed with {other}",
domain.as_str()
))),
},
_ => Err(SendError::DnsTransient(e.to_string())),
}
}
fn implicit_mx(domain: &EmailDomain) -> Result<Vec<MxRecord>, SendError> {
MxHost::parse(domain.as_str())
.map(|host| {
vec![MxRecord {
priority: MxPriority::new(0),
host,
}]
})
.map_err(|e| SendError::DnsPermanent(format!("invalid recipient domain: {e}")))
}
#[cfg(test)]
mod tests {
use super::*;
fn record(prio: u16, host: &str) -> MxRecord {
MxRecord {
priority: MxPriority::new(prio),
host: MxHost::parse(host).unwrap(),
}
}
#[test]
fn prioritize_sorts_by_priority_ascending() {
let result = prioritize(vec![
record(20, "mx2.nel.pet"),
record(10, "mx1.nel.pet"),
record(10, "mx1b.nel.pet"),
]);
assert_eq!(result[0].priority.as_u16(), 10);
assert_eq!(result[1].priority.as_u16(), 10);
assert_eq!(result[2].priority.as_u16(), 20);
}
#[test]
fn prioritize_randomizes_equal_priority_order() {
let attempts: Vec<Vec<String>> = (0..200)
.map(|_| {
prioritize(vec![
record(10, "a.nel.pet"),
record(10, "b.nel.pet"),
record(10, "c.nel.pet"),
record(10, "d.nel.pet"),
])
.into_iter()
.map(|r| r.host.as_str().to_string())
.collect()
})
.collect();
let distinct: std::collections::HashSet<_> = attempts.iter().cloned().collect();
assert!(
distinct.len() > 1,
"equal-priority MX order should vary across calls; got only {}",
distinct.len()
);
}
#[test]
fn implicit_mx_uses_domain_as_host() {
let d = EmailDomain::parse("nel.pet").unwrap();
let result = implicit_mx(&d).unwrap();
assert_eq!(result.len(), 1);
assert_eq!(result[0].priority.as_u16(), 0);
assert_eq!(result[0].host.as_str(), "nel.pet");
}
#[test]
fn no_error_response_yields_implicit_mx() {
let d = EmailDomain::parse("nel.pet").unwrap();
let err = ResolveError::from(ResolveErrorKind::NoRecordsFound {
query: Box::new(hickory_resolver::proto::op::Query::default()),
soa: None,
negative_ttl: None,
response_code: ResponseCode::NoError,
trusted: false,
});
let result = classify_lookup_error(err, &d).unwrap();
assert_eq!(result.len(), 1);
assert_eq!(result[0].host.as_str(), "nel.pet");
}
#[test]
fn nxdomain_response_is_permanent() {
let d = EmailDomain::parse("does-not-exist.invalid").unwrap();
let err = ResolveError::from(ResolveErrorKind::NoRecordsFound {
query: Box::new(hickory_resolver::proto::op::Query::default()),
soa: None,
negative_ttl: None,
response_code: ResponseCode::NXDomain,
trusted: true,
});
match classify_lookup_error(err, &d) {
Err(SendError::DnsPermanent(_)) => {}
other => panic!("expected DnsPermanent, got {other:?}"),
}
}
#[test]
fn servfail_response_is_transient() {
let d = EmailDomain::parse("nel.pet").unwrap();
let err = ResolveError::from(ResolveErrorKind::NoRecordsFound {
query: Box::new(hickory_resolver::proto::op::Query::default()),
soa: None,
negative_ttl: None,
response_code: ResponseCode::ServFail,
trusted: false,
});
match classify_lookup_error(err, &d) {
Err(SendError::DnsTransient(_)) => {}
other => panic!("expected DnsTransient, got {other:?}"),
}
}
#[test]
fn timeout_is_transient() {
let d = EmailDomain::parse("nel.pet").unwrap();
let err = ResolveError::from(ResolveErrorKind::Timeout);
match classify_lookup_error(err, &d) {
Err(SendError::DnsTransient(_)) => {}
other => panic!("expected DnsTransient, got {other:?}"),
}
}
#[test]
fn message_variant_is_transient() {
let d = EmailDomain::parse("nel.pet").unwrap();
let err = ResolveError::from(ResolveErrorKind::Message("transient resolver glitch"));
match classify_lookup_error(err, &d) {
Err(SendError::DnsTransient(_)) => {}
other => panic!("expected DnsTransient default, got {other:?}"),
}
}
#[test]
fn null_mx_is_permanent() {
let d = EmailDomain::parse("nomail.nel.pet").unwrap();
let result = interpret_lookup(vec![(0, hickory_resolver::Name::root())], &d);
match result {
Err(SendError::DnsPermanent(msg)) => {
assert!(msg.contains("null MX"), "msg: {msg}")
}
other => panic!("expected DnsPermanent, got {other:?}"),
}
}
#[test]
fn null_mx_alongside_real_records_still_permanent() {
let d = EmailDomain::parse("mixed.nel.pet").unwrap();
let real = hickory_resolver::Name::from_ascii("mx1.nel.pet.").unwrap();
let result = interpret_lookup(vec![(10, real), (0, hickory_resolver::Name::root())], &d);
assert!(matches!(result, Err(SendError::DnsPermanent(_))));
}
#[test]
fn empty_lookup_uses_implicit_mx() {
let d = EmailDomain::parse("nel.pet").unwrap();
let result = interpret_lookup(Vec::<(u16, hickory_resolver::Name)>::new(), &d).unwrap();
assert_eq!(result.len(), 1);
assert_eq!(result[0].host.as_str(), "nel.pet");
}
#[test]
fn valid_records_pass_through_with_priority_sort() {
let d = EmailDomain::parse("nel.pet").unwrap();
let mx1 = hickory_resolver::Name::from_ascii("mx1.nel.pet.").unwrap();
let mx2 = hickory_resolver::Name::from_ascii("mx2.nel.pet.").unwrap();
let result = interpret_lookup(vec![(20, mx2), (10, mx1)], &d).unwrap();
assert_eq!(result.len(), 2);
assert_eq!(result[0].priority.as_u16(), 10);
assert_eq!(result[0].host.as_str(), "mx1.nel.pet");
assert_eq!(result[1].priority.as_u16(), 20);
}
}
@@ -0,0 +1,27 @@
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAtsQsUV8QpqrygsY+2+JCQ6Fw8/omM71IM2N/R8pPbzbgOl0p
78MZGsgPOQ2HSznjD0FPzsH8oO2B5Uftws04LHb2HJAYlz25+lN5cqfHAfa3fgmC
38FfwBkn7l582UtPWZ/wcBOnyCgb3yLcvJrXyrt8QxHJgvWO23ITrUVYszImbXQ6
7YGS0YhMrbixRzmo2tpm3JcIBtnHrEUMsT0NfFdfsZhTT8YbxBvA8FdODgEwx7u/
vf3J9qbi4+Kv8cvqyJuleIRSjVXPsIMnoejIn04APPKIjpMyQdnWlby7rNyQtE4+
CV+jcFjqJbE/Xilcvqxt6DirjFCvYeKYl1uHLwIDAQABAoIBAH7Mg2LA7bB0EWQh
XiL3SrnZG6BpAHAM9jaQ5RFNjua9z7suP5YUaSpnegg/FopeUuWWjmQHudl8bg5A
ZPgtoLdYoU8XubfUH19I4o1lUXBPVuaeeqn6Yw/HZCjAbSXkVdz8VbesK092ZD/e
0/4V/3irsn5lrMSq0L322yfvYKaRDFxKCF7UMnWrGcHZl6Msbv/OffLRk19uYB7t
4WGhK1zCfKIfgdLJnD0eoI6Q4wU6sJvvpyTe8NDDo8HpdAwNn3YSahSewKp9gHgg
VIQlTZUdsHxM+R+2RUwJZYj9WSTbq+s1nKICUmjQBPnWbrPW963BE5utQPFt3mOe
EWRzdsECgYEA3MBhJC1Okq+u5yrFE8plufdwNvm9fg5uYUYafvdlQiXsFTx+XDGm
FXpuWhP/bheOh1jByzPZ1rvjF57xiZjkIuzcvtePTs/b5fT82K7CydDchkc8qb0W
2dI40h+13e++sUPKYdC9aqjZHzOgl3kOlkDbyRCF3F8mNDujE49rLWcCgYEA0/MU
dX5A6VSDb5K+JCNq8vDaBKNGU8GAr2fpYAhtk/3mXLI+/Z0JN0di9ZgeNhhJr2jN
11OU/2pOButpsgnkIo2y36cOQPf5dQpSgXZke3iNDld3osuLIuPNJn/3C087AtOq
+w4YxZClZLAxiLCqX8SBVrB2IiFCQ70SJ++n8vkCgYEAzmi3rBsNEA1jblVIh1PF
wJhD/bOQ4nBd92iUV8m9jZdl4wl4YX4u/IBI9MMkIG24YIe2VOl7s9Rk5+4/jNg/
4QQ2998Y6aljxOZJEdZ+3jQELy4m49OhrTRq2ta5t/Z3CMsJTmLe6f9NXWZpr5iK
8iVdHOjtMXxqfYaR2jVNEtsCgYAl9uWUQiAoa037v0I1wO5YQ9IZgJGJUSDWynsg
C4JtPs5zji4ASY+sCipsqWnH8MPKGrC8QClxMr51ONe+30yw78a5jvfbpU9Wqpmq
vOU0xJwnlH1GeMUcY8eMfOFocjG0yOtYeubvBIDLr0/AFzz9WHp+Z69RX7m53nUR
GDlyKQKBgDGZVAbUBiB8rerqNbONBAxfipoa4IJ+ntBrFT2DtoIZNbSzaoK+nVbH
kbWMJycaV5PVOh1lfAiZeWCxQz5RcZh/RS8USnxyMG1j4dP/wLcbdasI8uRaSC6Y
hFHL5HjhLrIo0HRWySS2b2ztBI2FP1M+MaaGFPHDzm2OyZg85yr3
-----END RSA PRIVATE KEY-----
@@ -0,0 +1,164 @@
use std::sync::Arc;
use std::time::Duration;
use futures::StreamExt;
use hickory_resolver::TokioAsyncResolver;
use lettre::transport::smtp::AsyncSmtpTransport;
use lettre::transport::smtp::Error as SmtpError;
use lettre::transport::smtp::client::{Tls, TlsParameters};
use lettre::transport::smtp::extension::ClientId;
use lettre::{AsyncTransport, Message, Tokio1Executor};
use tokio::sync::Semaphore;
use super::message::recipient_domain;
use super::mx;
use super::types::{HeloName, MxRecord};
use crate::sender::SendError;
pub enum SendMode {
Smarthost {
transport: Box<AsyncSmtpTransport<Tokio1Executor>>,
total_timeout: Duration,
},
DirectMx {
resolver: Arc<TokioAsyncResolver>,
helo: HeloName,
command_timeout: Duration,
total_timeout: Duration,
require_tls: bool,
inflight: Arc<Semaphore>,
},
}
impl std::fmt::Debug for SendMode {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::Smarthost { total_timeout, .. } => {
write!(f, "SendMode::Smarthost(total_timeout={total_timeout:?})")
}
Self::DirectMx {
helo, require_tls, ..
} => write!(
f,
"SendMode::DirectMx({}, require_tls={require_tls})",
helo.as_str()
),
}
}
}
pub async fn dispatch(mode: &SendMode, message: Message) -> Result<(), SendError> {
match mode {
SendMode::Smarthost {
transport,
total_timeout,
} => with_total_timeout(*total_timeout, run_send(transport, message)).await,
SendMode::DirectMx {
resolver,
helo,
command_timeout,
total_timeout,
require_tls,
inflight,
} => {
with_total_timeout(*total_timeout, async {
let _permit =
inflight.clone().acquire_owned().await.map_err(|_| {
SendError::SmtpTransient("send semaphore closed".to_string())
})?;
send_direct(
resolver.as_ref(),
helo,
*command_timeout,
*require_tls,
message,
)
.await
})
.await
}
}
}
async fn with_total_timeout<F: std::future::Future<Output = Result<(), SendError>>>(
total: Duration,
fut: F,
) -> Result<(), SendError> {
tokio::time::timeout(total, fut)
.await
.unwrap_or(Err(SendError::Timeout))
}
async fn run_send(
transport: &AsyncSmtpTransport<Tokio1Executor>,
message: Message,
) -> Result<(), SendError> {
transport
.send(message)
.await
.map(|_| ())
.map_err(classify_smtp_error)
}
async fn send_direct(
resolver: &TokioAsyncResolver,
helo: &HeloName,
command_timeout: Duration,
require_tls: bool,
message: Message,
) -> Result<(), SendError> {
let domain = recipient_domain(&message)?;
let mxs = mx::resolve(resolver, &domain).await?;
let outcome = futures::stream::iter(mxs)
.fold(None::<Result<(), SendError>>, |acc, mx_record| {
let message = message.clone();
async move {
match &acc {
Some(Ok(())) | Some(Err(SendError::SmtpPermanent(_))) => acc,
_ => Some(
attempt_one_host(mx_record, helo, command_timeout, require_tls, message)
.await,
),
}
}
})
.await;
outcome.unwrap_or_else(|| {
Err(SendError::SmtpTransient(format!(
"no MX records returned for {}",
domain.as_str()
)))
})
}
async fn attempt_one_host(
mx_record: MxRecord,
helo: &HeloName,
command_timeout: Duration,
require_tls: bool,
message: Message,
) -> Result<(), SendError> {
let host = mx_record.host.as_str().to_string();
let tls_params = TlsParameters::new(host.clone())
.map_err(|e| SendError::SmtpTransient(format!("TLS params for {host}: {e}")))?;
let tls = match require_tls {
true => Tls::Required(tls_params),
false => Tls::Opportunistic(tls_params),
};
let transport: AsyncSmtpTransport<Tokio1Executor> =
AsyncSmtpTransport::<Tokio1Executor>::builder_dangerous(&host)
.port(25)
.tls(tls)
.hello_name(ClientId::Domain(helo.as_str().to_string()))
.timeout(Some(command_timeout))
.build();
run_send(&transport, message).await
}
fn classify_smtp_error(e: SmtpError) -> SendError {
match () {
_ if e.is_permanent() => SendError::SmtpPermanent(e.to_string()),
_ if e.is_timeout() => SendError::Timeout,
_ => SendError::SmtpTransient(e.to_string()),
}
}
+291
View File
@@ -0,0 +1,291 @@
use std::path::PathBuf;
#[derive(Debug, thiserror::Error)]
pub enum ParseError {
#[error("empty value")]
Empty,
#[error("invalid character {0:?}")]
InvalidChar(char),
#[error("zero {0}")]
Zero(&'static str),
#[error("invalid TLS mode {0:?}")]
InvalidTlsMode(String),
}
fn parse_token(raw: &str, lowercase: bool, strip_trailing_dot: bool) -> Result<String, ParseError> {
let mut s = raw.trim();
if strip_trailing_dot {
s = s.trim_end_matches('.');
}
match s {
"" => Err(ParseError::Empty),
_ if s.chars().any(char::is_whitespace) => Err(ParseError::InvalidChar(' ')),
_ => Ok(match lowercase {
true => s.to_lowercase(),
false => s.to_string(),
}),
}
}
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct SmtpHost(String);
impl SmtpHost {
pub fn parse(raw: &str) -> Result<Self, ParseError> {
parse_token(raw, true, false).map(Self)
}
pub fn as_str(&self) -> &str {
&self.0
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct SmtpPort(u16);
impl SmtpPort {
pub fn parse(raw: u16) -> Result<Self, ParseError> {
match raw {
0 => Err(ParseError::Zero("smtp port")),
n => Ok(Self(n)),
}
}
pub fn as_u16(self) -> u16 {
self.0
}
}
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct HeloName(String);
impl HeloName {
pub fn parse(raw: &str) -> Result<Self, ParseError> {
parse_token(raw, false, false).map(Self)
}
pub fn as_str(&self) -> &str {
&self.0
}
pub fn into_inner(self) -> String {
self.0
}
}
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct EmailDomain(String);
impl EmailDomain {
pub fn parse(raw: &str) -> Result<Self, ParseError> {
parse_token(raw, true, true).map(Self)
}
pub fn as_str(&self) -> &str {
&self.0
}
pub fn into_inner(self) -> String {
self.0
}
}
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct MxHost(String);
impl MxHost {
pub fn parse(raw: &str) -> Result<Self, ParseError> {
parse_token(raw, true, true).map(Self)
}
pub fn as_str(&self) -> &str {
&self.0
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct MxPriority(u16);
impl MxPriority {
pub fn new(value: u16) -> Self {
Self(value)
}
pub fn as_u16(self) -> u16 {
self.0
}
}
#[derive(Debug, Clone)]
pub struct MxRecord {
pub priority: MxPriority,
pub host: MxHost,
}
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct DkimSelector(String);
impl DkimSelector {
pub fn parse(raw: &str) -> Result<Self, ParseError> {
let trimmed = raw.trim();
let valid = !trimmed.is_empty() && trimmed.split('.').all(valid_subdomain);
match valid {
true => Ok(Self(trimmed.to_string())),
false => Err(ParseError::InvalidChar('?')),
}
}
pub fn into_inner(self) -> String {
self.0
}
}
fn valid_subdomain(seg: &str) -> bool {
let starts_alnum = seg
.chars()
.next()
.is_some_and(|c| c.is_ascii_alphanumeric());
let ends_alnum = seg
.chars()
.next_back()
.is_some_and(|c| c.is_ascii_alphanumeric());
let body_ok = seg.chars().all(|c| c.is_ascii_alphanumeric() || c == '-');
starts_alnum && ends_alnum && body_ok
}
#[derive(Debug, Clone)]
pub struct DkimKeyPath(PathBuf);
impl DkimKeyPath {
pub fn parse(raw: &str) -> Result<Self, ParseError> {
let trimmed = raw.trim();
match trimmed.is_empty() {
true => Err(ParseError::Empty),
false => Ok(Self(PathBuf::from(trimmed))),
}
}
pub fn as_path(&self) -> &std::path::Path {
&self.0
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SmtpUsername(String);
impl SmtpUsername {
pub fn parse(raw: &str) -> Result<Self, ParseError> {
match raw.is_empty() {
true => Err(ParseError::Empty),
false => Ok(Self(raw.to_string())),
}
}
pub fn into_inner(self) -> String {
self.0
}
}
#[derive(Clone)]
pub struct SmtpPassword(secrecy::SecretString);
impl SmtpPassword {
pub fn parse(raw: &str) -> Result<Self, ParseError> {
match raw.is_empty() {
true => Err(ParseError::Empty),
false => Ok(Self(secrecy::SecretString::from(raw.to_string()))),
}
}
pub fn expose(&self) -> &str {
use secrecy::ExposeSecret;
self.0.expose_secret()
}
}
impl std::fmt::Debug for SmtpPassword {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("SmtpPassword(***)")
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum TlsMode {
Implicit,
Starttls,
None,
}
impl TlsMode {
pub fn parse(raw: &str) -> Result<Self, ParseError> {
match raw.to_ascii_lowercase().as_str() {
"implicit" => Ok(Self::Implicit),
"starttls" => Ok(Self::Starttls),
"none" => Ok(Self::None),
other => Err(ParseError::InvalidTlsMode(other.to_string())),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn smtp_host_lowercases_and_trims() {
let h = SmtpHost::parse(" SMTP.NEL.PET ").unwrap();
assert_eq!(h.as_str(), "smtp.nel.pet");
}
#[test]
fn smtp_host_rejects_whitespace() {
assert!(SmtpHost::parse("a b").is_err());
}
#[test]
fn smtp_host_rejects_empty() {
assert!(SmtpHost::parse("").is_err());
assert!(SmtpHost::parse(" ").is_err());
}
#[test]
fn smtp_port_rejects_zero() {
assert!(SmtpPort::parse(0).is_err());
assert_eq!(SmtpPort::parse(587).unwrap().as_u16(), 587);
}
#[test]
fn email_domain_strips_trailing_dot() {
assert_eq!(EmailDomain::parse("Nel.pet.").unwrap().as_str(), "nel.pet");
}
#[test]
fn dkim_selector_validates() {
assert!(DkimSelector::parse("default").is_ok());
assert!(DkimSelector::parse("s1.nel.pet").is_ok());
assert!(DkimSelector::parse("s2024-q1").is_ok());
assert!(DkimSelector::parse("mailo-2024.nel.pet").is_ok());
assert!(DkimSelector::parse("a-b").is_ok());
assert!(DkimSelector::parse("").is_err());
assert!(DkimSelector::parse("a..b").is_err());
assert!(DkimSelector::parse("-leading").is_err());
assert!(DkimSelector::parse("trailing-").is_err());
assert!(DkimSelector::parse("s_under").is_err());
}
#[test]
fn tls_mode_parses_known_modes() {
assert_eq!(TlsMode::parse("STARTTLS").unwrap(), TlsMode::Starttls);
assert_eq!(TlsMode::parse("implicit").unwrap(), TlsMode::Implicit);
assert_eq!(TlsMode::parse("none").unwrap(), TlsMode::None);
assert!(TlsMode::parse("garbage").is_err());
}
#[test]
fn smtp_password_redacts_in_debug() {
let p = SmtpPassword::parse("hunter2").unwrap();
let dbg = format!("{:?}", p);
assert_eq!(dbg, "SmtpPassword(***)");
assert!(!dbg.contains("hunter2"));
}
}
+4 -2
View File
@@ -1,13 +1,15 @@
pub mod email;
mod locale;
mod sender;
mod types;
pub use email::EmailSender;
pub use locale::{
DEFAULT_LOCALE, NotificationStrings, VALID_LOCALES, format_message, get_strings,
validate_locale,
};
pub use sender::{
CommsSender, DiscordSender, EmailSender, SendError, SignalSender, TelegramSender,
is_valid_phone_number, is_valid_signal_username, mime_encode_header, sanitize_header_value,
CommsSender, DiscordSender, SendError, SignalSender, TelegramSender, is_valid_phone_number,
is_valid_signal_username,
};
pub use types::{CommsChannel, CommsStatus, CommsType, NewComms, QueuedComms};
+37 -1
View File
@@ -1,5 +1,5 @@
pub const DEFAULT_LOCALE: &str = "en";
pub const VALID_LOCALES: &[&str] = &["en", "zh", "ja", "ko", "sv", "fi"];
pub const VALID_LOCALES: &[&str] = &["en", "zh", "ja", "ko", "sv", "fi", "fr"];
pub fn validate_locale(locale: &str) -> &str {
if VALID_LOCALES.contains(&locale) {
@@ -44,6 +44,7 @@ pub fn get_strings(locale: &str) -> &'static NotificationStrings {
"ko" => &STRINGS_KO,
"sv" => &STRINGS_SV,
"fi" => &STRINGS_FI,
"fr" => &STRINGS_FR,
_ => &STRINGS_EN,
}
}
@@ -216,6 +217,34 @@ static STRINGS_FI: NotificationStrings = NotificationStrings {
channel_verification_body: "Vahvistuskoodisi on:\n{code}\n\nTai vahvista suoraan:\n{verify_link}",
};
static STRINGS_FR: NotificationStrings = NotificationStrings {
welcome_subject: "Bienvenue sur {hostname}",
welcome_body: "Bienvenue sur {hostname} !\n\nVotre identifiant est : @{handle}\n\nMerci de nous avoir rejoint.",
password_reset_subject: "Réinitialisation du mot de passe - {hostname}",
password_reset_body: "Bonjour @{handle},\n\nVotre code de réinitialisation du mot de passe est : {code}\n\nCe code expirera dans 10 minutes.\n\nSi vous n'avez pas demandé cela, veuillez ignorer ce message.",
email_update_subject: "Confirmer votre nouvelle adresse e-mail - {hostname}",
email_update_body: "Bonjour @{handle},\n\nVotre code de vérification est :\n{code}\n\nCopiez le code ci-dessus et saisissez-le ici :\n{verify_page}\n\nCe code expirera dans 10 minutes.\n\nOu si vous aimez vivre dangereusement :\n{verify_link}\n\nSi vous n'avez pas demandé cela, veuillez ignorer cet e-mail.",
short_token_body: "Bonjour @{handle},\n\nVotre code de vérification est :\n{code}\n\nCe code expirera dans 15 minutes.\n\nSi vous n'avez pas demandé cela, veuillez ignorer cet e-mail.",
account_deletion_subject: "Demande de suppression de compte - {hostname}",
account_deletion_body: "Bonjour @{handle},\n\nVotre code de confirmation de suppression de compte est : {code}\n\nCe code expirera dans 10 minutes.\n\nSi vous n'avez pas demandé cela, sécurisez votre compte immédiatement.",
plc_operation_subject: "{hostname} - Jeton d'opération PLC",
plc_operation_body: "Bonjour @{handle},\n\nVous avez demandé à signer une opération PLC pour votre compte.\n\nVotre jeton de vérification est : {token}\n\nCe jeton expirera dans 10 minutes.\n\nSi vous n'avez pas demandé cela, vous pouvez ignorer ce message en toute sécurité.",
two_factor_code_subject: "Vérification de connexion - {hostname}",
two_factor_code_body: "Bonjour @{handle},\n\nVotre code de vérification de connexion est : {code}\n\nCe code expirera dans 10 minutes.\n\nSi vous n'avez pas demandé cela, sécurisez votre compte immédiatement.",
passkey_recovery_subject: "Récupération de compte - {hostname}",
passkey_recovery_body: "Bonjour @{handle},\n\nVous avez demandé la récupération de votre compte à clé d'accès uniquement.\n\nCliquez sur le lien ci-dessous pour définir un mot de passe temporaire et retrouver l'accès :\n{url}\n\nCe lien expirera dans 1 heure.\n\nSi vous n'avez pas demandé cela, veuillez ignorer ce message. Votre compte reste sécurisé.",
signup_verification_subject: "Vérifier votre compte - {hostname}",
signup_verification_body: "Bienvenue ! Votre code de vérification est :\n{code}\n\nCopiez le code ci-dessus et saisissez-le ici :\n{verify_page}\n\nCe code expirera dans 30 minutes.\n\nOu si vous aimez vivre dangereusement :\n{verify_link}\n\nSi vous n'avez pas créé de compte sur {hostname}, veuillez ignorer ce message.",
legacy_login_subject: "Alerte de sécurité : Connexion classique détectée - {hostname}",
legacy_login_body: "Bonjour @{handle},\n\nUne connexion à votre compte a été détectée via une application classique (comme Bluesky) qui ne prend pas en charge la vérification TOTP.\n\nDétails :\n- Date : {timestamp}\n- Adresse IP : {ip}\n\nVotre protection TOTP a été contournée pour cette connexion. La session dispose de permissions limitées pour les opérations sensibles.\n\nSi ce n'était pas vous :\n1. Changez votre mot de passe immédiatement\n2. Vérifiez vos sessions actives\n3. Envisagez de désactiver les connexions d'applications classiques dans vos paramètres de sécurité\n\nRestez vigilant,\n{hostname}",
migration_verification_subject: "Vérifier votre adresse e-mail - {hostname}",
migration_verification_body: "Bienvenue sur {hostname} !\n\nVotre compte a été migré avec succès. Pour finaliser la configuration, veuillez vérifier votre adresse e-mail.\n\nVotre code de vérification est :\n{code}\n\nCopiez le code ci-dessus et saisissez-le ici :\n{verify_page}\n\nCe code expirera dans 48 heures.\n\nOu si vous aimez vivre dangereusement :\n{verify_link}\n\nSi vous n'avez pas migré votre compte, veuillez ignorer cet e-mail.",
channel_verified_subject: "Canal de notification vérifié - {hostname}",
channel_verified_body: "Bonjour {handle},\n\n{channel} a été vérifié comme canal de notification pour votre compte sur {hostname}.",
channel_verification_subject: "Vérifier votre canal - {hostname}",
channel_verification_body: "Votre code de vérification est :\n{code}\n\nOu vérifiez directement :\n{verify_link}",
};
pub fn format_message(template: &str, vars: &[(&str, &str)]) -> String {
vars.iter()
.fold(template.to_string(), |result, (key, value)| {
@@ -233,6 +262,9 @@ mod tests {
assert_eq!(validate_locale("zh"), "zh");
assert_eq!(validate_locale("ja"), "ja");
assert_eq!(validate_locale("ko"), "ko");
assert_eq!(validate_locale("sv"), "sv");
assert_eq!(validate_locale("fi"), "fi");
assert_eq!(validate_locale("fr"), "fr");
assert_eq!(validate_locale("invalid"), DEFAULT_LOCALE);
assert_eq!(validate_locale(""), DEFAULT_LOCALE);
}
@@ -252,5 +284,9 @@ mod tests {
let zh = get_strings("zh");
assert!(zh.welcome_subject.contains("{hostname}"));
assert!(zh.welcome_body.contains("欢迎"));
let fr = get_strings("fr");
assert!(fr.welcome_subject.contains("{hostname}"));
assert!(fr.welcome_body.contains("Bienvenue"));
}
}
+61 -114
View File
@@ -1,11 +1,7 @@
use async_trait::async_trait;
use base64::{Engine, engine::general_purpose::STANDARD as BASE64};
use reqwest::Client;
use serde_json::json;
use std::process::Stdio;
use std::time::Duration;
use tokio::io::AsyncWriteExt;
use tokio::process::Command;
use super::types::{CommsChannel, QueuedComms};
@@ -21,25 +17,51 @@ pub trait CommsSender: Send + Sync {
#[derive(Debug, thiserror::Error)]
pub enum SendError {
#[error("Failed to spawn {command}: {source}")]
ProcessSpawn {
command: String,
source: std::io::Error,
},
#[error("{command} exited with non-zero status: {detail}")]
ProcessFailed { command: String, detail: String },
#[error("Channel not configured: {0:?}")]
NotConfigured(CommsChannel),
#[error("External service error: {0}")]
ExternalService(String),
#[error("Email configuration invalid: {0}")]
ConfigInvalid(String),
#[error("Invalid recipient format: {0}")]
InvalidRecipient(String),
#[error("Message construction failed: {0}")]
MessageBuild(String),
#[error("transient DNS lookup failure: {0}")]
DnsTransient(String),
#[error("permanent DNS lookup failure: {0}")]
DnsPermanent(String),
#[error("SMTP transient error: {0}")]
SmtpTransient(String),
#[error("SMTP permanent error: {0}")]
SmtpPermanent(String),
#[error("DKIM signing failed: {0}")]
DkimSign(String),
#[error("External service error: {0}")]
ExternalService(String),
#[error("Request timeout")]
Timeout,
#[error("Max retries exceeded: {0}")]
MaxRetriesExceeded(String),
}
impl SendError {
pub fn is_permanent(&self) -> bool {
match self {
Self::SmtpPermanent(_)
| Self::DnsPermanent(_)
| Self::InvalidRecipient(_)
| Self::MessageBuild(_)
| Self::DkimSign(_)
| Self::ConfigInvalid(_) => true,
Self::SmtpTransient(_)
| Self::DnsTransient(_)
| Self::Timeout
| Self::ExternalService(_)
| Self::MaxRetriesExceeded(_)
| Self::NotConfigured(_) => false,
}
}
}
fn create_http_client() -> Client {
Client::builder()
.timeout(Duration::from_secs(HTTP_TIMEOUT_SECS))
@@ -100,19 +122,6 @@ where
))
}
pub fn sanitize_header_value(value: &str) -> String {
value.replace(['\r', '\n'], " ").trim().to_string()
}
pub fn mime_encode_header(value: &str) -> String {
if value.is_ascii() {
sanitize_header_value(value)
} else {
let sanitized = sanitize_header_value(value);
format!("=?UTF-8?B?{}?=", BASE64.encode(sanitized.as_bytes()))
}
}
pub fn escape_html(text: &str) -> String {
text.replace('&', "&amp;")
.replace('<', "&lt;")
@@ -135,93 +144,6 @@ pub fn is_valid_signal_username(username: &str) -> bool {
tranquil_signal::SignalUsername::parse(username).is_ok()
}
pub struct EmailSender {
from_address: String,
from_name: String,
sendmail_path: String,
}
impl EmailSender {
pub fn new(from_address: String, from_name: String, sendmail_path: String) -> Self {
Self {
from_address,
from_name,
sendmail_path,
}
}
pub fn from_config(cfg: &tranquil_config::TranquilConfig) -> Option<Self> {
let from_address = cfg.email.from_address.clone()?;
let from_name = cfg.email.from_name.clone();
let sendmail_path = cfg.email.sendmail_path.clone();
Some(Self::new(from_address, from_name, sendmail_path))
}
pub fn format_email(&self, notification: &QueuedComms) -> String {
let subject = mime_encode_header(notification.subject.as_deref().unwrap_or("Notification"));
let recipient = sanitize_header_value(&notification.recipient);
let from_header = if self.from_name.is_empty() {
self.from_address.clone()
} else {
format!(
"{} <{}>",
sanitize_header_value(&self.from_name),
self.from_address
)
};
format!(
"From: {}\r\nTo: {}\r\nSubject: {}\r\nContent-Type: text/plain; charset=utf-8\r\nMIME-Version: 1.0\r\n\r\n{}",
from_header, recipient, subject, notification.body
)
}
}
#[async_trait]
impl CommsSender for EmailSender {
fn channel(&self) -> CommsChannel {
CommsChannel::Email
}
async fn send(&self, notification: &QueuedComms) -> Result<(), SendError> {
let email_content = self.format_email(notification);
let mut child = Command::new(&self.sendmail_path)
.arg("-t")
.arg("-oi")
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.map_err(|e| SendError::ProcessSpawn {
command: self.sendmail_path.clone(),
source: e,
})?;
if let Some(mut stdin) = child.stdin.take() {
stdin
.write_all(email_content.as_bytes())
.await
.map_err(|e| SendError::ProcessSpawn {
command: self.sendmail_path.clone(),
source: e,
})?;
}
let output = child
.wait_with_output()
.await
.map_err(|e| SendError::ProcessSpawn {
command: self.sendmail_path.clone(),
source: e,
})?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
return Err(SendError::ProcessFailed {
command: self.sendmail_path.clone(),
detail: stderr.to_string(),
});
}
Ok(())
}
}
const DISCORD_API_BASE: &str = "https://discord.com/api/v10";
#[derive(Clone)]
@@ -299,7 +221,7 @@ impl DiscordSender {
"type": 1,
"options": [{
"name": "handle",
"description": "Your PDS handle (e.g. alice.example.com)",
"description": "Your PDS handle",
"type": 3,
"required": false
}]
@@ -610,3 +532,28 @@ impl CommsSender for SignalSender {
))
}
}
#[cfg(test)]
mod is_permanent_matrix {
use super::{CommsChannel, SendError};
#[test]
fn permanent_variants_are_permanent() {
assert!(SendError::SmtpPermanent("x".into()).is_permanent());
assert!(SendError::DnsPermanent("x".into()).is_permanent());
assert!(SendError::InvalidRecipient("x".into()).is_permanent());
assert!(SendError::MessageBuild("x".into()).is_permanent());
assert!(SendError::DkimSign("x".into()).is_permanent());
assert!(SendError::ConfigInvalid("x".into()).is_permanent());
}
#[test]
fn transient_variants_are_not_permanent() {
assert!(!SendError::SmtpTransient("x".into()).is_permanent());
assert!(!SendError::DnsTransient("x".into()).is_permanent());
assert!(!SendError::Timeout.is_permanent());
assert!(!SendError::ExternalService("x".into()).is_permanent());
assert!(!SendError::MaxRetriesExceeded("x".into()).is_permanent());
assert!(!SendError::NotConfigured(CommsChannel::Email).is_permanent());
}
}
+158
View File
@@ -0,0 +1,158 @@
use std::time::Duration;
use chrono::Utc;
use lettre::message::Mailbox;
use lettre::transport::smtp::AsyncSmtpTransport;
use lettre::transport::smtp::extension::ClientId;
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::net::{TcpListener, TcpStream};
use tranquil_comms::email::transport::SendMode;
use tranquil_comms::email::{EmailSender, types::HeloName};
use tranquil_comms::{CommsChannel, CommsSender, CommsStatus, CommsType, QueuedComms, SendError};
use uuid::Uuid;
fn fixture(recipient: &str, subject: &str, body: &str) -> QueuedComms {
QueuedComms {
id: Uuid::new_v4(),
user_id: None,
channel: CommsChannel::Email,
comms_type: CommsType::Welcome,
status: CommsStatus::Pending,
recipient: recipient.to_string(),
subject: Some(subject.to_string()),
body: body.to_string(),
metadata: None,
attempts: 0,
max_attempts: 3,
last_error: None,
created_at: Utc::now(),
updated_at: Utc::now(),
scheduled_for: Utc::now(),
processed_at: None,
}
}
fn build_smarthost_sender(host: &str, port: u16) -> EmailSender {
build_smarthost_sender_with_total_timeout(host, port, Duration::from_secs(10))
}
fn build_smarthost_sender_with_total_timeout(
host: &str,
port: u16,
total_timeout: Duration,
) -> EmailSender {
let from: Mailbox = "Tranquil Test <noreply@nel.pet>".parse().unwrap();
let helo = HeloName::parse("mta.nel.pet").unwrap();
let transport = AsyncSmtpTransport::<lettre::Tokio1Executor>::builder_dangerous(host)
.port(port)
.hello_name(ClientId::Domain(helo.into_inner()))
.timeout(Some(Duration::from_secs(5)))
.build();
EmailSender::new(
from,
SendMode::Smarthost {
transport: Box::new(transport),
total_timeout,
},
None,
)
}
async fn drive_stub(stream: TcpStream, rcpt_response: &'static [u8]) -> std::io::Result<()> {
let (read, mut write) = stream.into_split();
let mut reader = BufReader::new(read);
write.write_all(b"220 stub ESMTP\r\n").await?;
let mut line = String::new();
loop {
line.clear();
let n = reader.read_line(&mut line).await?;
if n == 0 {
return Ok(());
}
let upper = line.to_ascii_uppercase();
let response: &[u8] = match upper.split_whitespace().next() {
Some("EHLO") | Some("HELO") => b"250-stub\r\n250 SIZE 10240000\r\n",
Some("MAIL") => b"250 OK\r\n",
Some("RCPT") => rcpt_response,
Some("DATA") => b"354 end with .\r\n",
Some("RSET") => b"250 OK\r\n",
Some("QUIT") => b"221 bye\r\n",
_ => b"500 unknown\r\n",
};
write.write_all(response).await?;
if upper.starts_with("QUIT") {
return Ok(());
}
}
}
async fn spawn_stub(rcpt_response: &'static [u8]) -> u16 {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let port = listener.local_addr().unwrap().port();
tokio::spawn(async move {
let (stream, _) = listener.accept().await.unwrap();
let _ = drive_stub(stream, rcpt_response).await;
});
port
}
#[tokio::test]
async fn rcpt_550_classifies_as_smtp_permanent() {
let port = spawn_stub(b"550 5.1.1 user unknown\r\n").await;
let sender = build_smarthost_sender("127.0.0.1", port);
let result = sender.send(&fixture("nel@nel.pet", "x", "x")).await;
match result {
Err(SendError::SmtpPermanent(_)) => {}
other => panic!("expected SmtpPermanent, got {other:?}"),
}
}
#[tokio::test]
async fn rcpt_421_classifies_as_smtp_transient() {
let port = spawn_stub(b"421 4.7.0 try again later\r\n").await;
let sender = build_smarthost_sender("127.0.0.1", port);
let result = sender.send(&fixture("nel@nel.pet", "x", "x")).await;
match result {
Err(SendError::SmtpTransient(_)) => {}
other => panic!("expected SmtpTransient, got {other:?}"),
}
}
#[tokio::test]
async fn invalid_recipient_classifies_as_invalid_recipient() {
let port = spawn_stub(b"250 OK\r\n").await;
let sender = build_smarthost_sender("127.0.0.1", port);
let result = sender.send(&fixture("not-an-address", "x", "x")).await;
match result {
Err(SendError::InvalidRecipient(_)) => {}
other => panic!("expected InvalidRecipient, got {other:?}"),
}
}
async fn spawn_silent_stub() -> u16 {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let port = listener.local_addr().unwrap().port();
tokio::spawn(async move {
let (_stream, _) = listener.accept().await.unwrap();
std::future::pending::<()>().await;
});
port
}
#[tokio::test]
async fn smarthost_silent_relay_hits_total_timeout() {
let port = spawn_silent_stub().await;
let sender =
build_smarthost_sender_with_total_timeout("127.0.0.1", port, Duration::from_millis(500));
let start = std::time::Instant::now();
let result = sender.send(&fixture("nel@nel.pet", "x", "x")).await;
let elapsed = start.elapsed();
match result {
Err(SendError::Timeout) => {}
other => panic!("expected Timeout, got {other:?}"),
}
assert!(
elapsed < Duration::from_secs(2),
"send returned in {elapsed:?}, expected close to 500ms total_timeout"
);
}
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -7,7 +7,7 @@ use uuid::Uuid;
use crate::DbError;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum BacklinkPath {
Subject,
SubjectUri,
+2
View File
@@ -244,6 +244,8 @@ pub trait InfraRepository: Send + Sync {
async fn mark_comms_failed(&self, id: Uuid, error: &str) -> Result<(), DbError>;
async fn mark_comms_failed_permanent(&self, id: Uuid, error: &str) -> Result<(), DbError>;
async fn create_invite_code(
&self,
code: &str,
+15 -18
View File
@@ -319,7 +319,6 @@ pub struct ApplyCommitInput {
#[derive(Debug, Clone)]
pub struct ApplyCommitResult {
pub seq: i64,
pub is_account_active: bool,
}
@@ -445,19 +444,15 @@ pub trait RepoRepository: Send + Sync {
async fn count_user_blocks(&self, user_id: Uuid) -> Result<i64, DbError>;
async fn insert_commit_event(&self, data: &CommitEventData) -> Result<SequenceNumber, DbError>;
async fn insert_commit_event(&self, data: &CommitEventData) -> Result<(), DbError>;
async fn insert_identity_event(
&self,
did: &Did,
handle: Option<&Handle>,
) -> Result<SequenceNumber, DbError>;
) -> Result<(), DbError>;
async fn insert_account_event(
&self,
did: &Did,
status: AccountStatus,
) -> Result<SequenceNumber, DbError>;
async fn insert_account_event(&self, did: &Did, status: AccountStatus) -> Result<(), DbError>;
async fn insert_sync_event(
&self,
@@ -465,7 +460,7 @@ pub trait RepoRepository: Send + Sync {
commit_cid: &CidLink,
rev: Option<&str>,
commit_bytes: &[u8],
) -> Result<SequenceNumber, DbError>;
) -> Result<(), DbError>;
async fn insert_genesis_commit_event(
&self,
@@ -475,14 +470,18 @@ pub trait RepoRepository: Send + Sync {
rev: &str,
commit_bytes: &[u8],
mst_root_bytes: &[u8],
) -> Result<SequenceNumber, DbError>;
async fn delete_sequences_except(
&self,
did: &Did,
keep_seq: SequenceNumber,
) -> Result<(), DbError>;
async fn purge_did_events_keeping_latest(&self, did: &Did) -> Result<(), DbError>;
async fn assign_pending_sequences(&self) -> Result<u64, DbError> {
Ok(0)
}
async fn flush_pending_sequences(&self) -> Result<(), DbError> {
Ok(())
}
async fn prune_events_older_than(&self, cutoff: DateTime<Utc>) -> Result<PruneCount, DbError>;
async fn get_max_seq(&self) -> Result<SequenceNumber, DbError>;
@@ -526,8 +525,6 @@ pub trait RepoRepository: Send + Sync {
async fn get_repo_root_cid_by_user_id(&self, user_id: Uuid)
-> Result<Option<CidLink>, DbError>;
async fn notify_update(&self, seq: SequenceNumber) -> Result<(), DbError>;
async fn import_repo_data(
&self,
user_id: Uuid,
@@ -563,5 +560,5 @@ pub trait RepoEventNotifier: Send + Sync {
#[async_trait]
pub trait RepoEventReceiver: Send {
async fn recv(&mut self) -> Option<i64>;
async fn recv(&mut self) -> Option<()>;
}
+3 -1
View File
@@ -44,6 +44,7 @@ pub struct UserRow {
pub deactivated_at: Option<DateTime<Utc>>,
pub takedown_ref: Option<String>,
pub is_admin: bool,
pub inbound_migration: bool,
}
#[derive(Debug, Clone)]
@@ -550,7 +551,7 @@ pub trait UserRepository: Send + Sync {
limit: i64,
) -> Result<Vec<ScheduledDeletionAccount>, DbError>;
async fn delete_account_with_firehose(&self, user_id: Uuid, did: &Did) -> Result<i64, DbError>;
async fn delete_account_with_firehose(&self, user_id: Uuid, did: &Did) -> Result<(), DbError>;
async fn create_password_account(
&self,
@@ -995,6 +996,7 @@ pub struct CreatePasswordAccountInput {
pub telegram_username: Option<String>,
pub signal_username: Option<String>,
pub deactivated_at: Option<DateTime<Utc>>,
pub inbound_migration: bool,
pub encrypted_key_bytes: Vec<u8>,
pub encryption_version: i32,
pub reserved_key_id: Option<Uuid>,
@@ -35,9 +35,9 @@ pub struct PostgresRepoEventReceiver {
#[async_trait]
impl RepoEventReceiver for PostgresRepoEventReceiver {
async fn recv(&mut self) -> Option<i64> {
async fn recv(&mut self) -> Option<()> {
match self.listener.recv().await {
Ok(notification) => notification.payload().parse().ok(),
Ok(_) => Some(()),
Err(_) => None,
}
}
+24 -2
View File
@@ -65,9 +65,13 @@ impl InfraRepository for PostgresInfraRepository {
SET status = 'processing', updated_at = NOW()
WHERE id IN (
SELECT id FROM comms_queue
WHERE status = 'pending'
WHERE attempts < max_attempts
AND scheduled_for <= $1
AND attempts < max_attempts
AND (
status = 'pending'
OR (status = 'processing'
AND updated_at < $1 - INTERVAL '10 minutes')
)
ORDER BY scheduled_for ASC
LIMIT $2
FOR UPDATE SKIP LOCKED
@@ -127,6 +131,24 @@ impl InfraRepository for PostgresInfraRepository {
Ok(())
}
async fn mark_comms_failed_permanent(&self, id: Uuid, error: &str) -> Result<(), DbError> {
sqlx::query!(
r#"UPDATE comms_queue
SET status = 'failed'::comms_status,
attempts = max_attempts,
last_error = $2,
updated_at = NOW()
WHERE id = $1"#,
id,
error
)
.execute(&self.pool)
.await
.map_err(map_sqlx_error)?;
Ok(())
}
async fn create_invite_code(
&self,
code: &str,
+1 -1
View File
@@ -374,7 +374,7 @@ impl OAuthRepository for PostgresOAuthRepository {
WHERE id IN (
SELECT id FROM oauth_token
WHERE did = $1
ORDER BY updated_at ASC
ORDER BY created_at DESC
OFFSET $2
)
"#,
+145 -88
View File
@@ -111,6 +111,64 @@ fn map_sequenced_row(r: SequencedEventRow) -> Result<SequencedEvent, DbError> {
})
}
const SEQUENCER_LOCK_KEY: i64 = 0x0074_7261_6e73_6571;
const SEQUENCER_BATCH_SIZE: i64 = 1000;
async fn notify_repo_pending(pool: &PgPool) {
if let Err(e) = sqlx::query!("NOTIFY repo_updates").execute(pool).await {
tracing::warn!(error = %e, "failed to notify repo_updates listeners");
}
}
async fn assign_one_batch(
mut tx: sqlx::Transaction<'_, sqlx::Postgres>,
pool: &PgPool,
) -> Result<i64, DbError> {
let pending_ids: Vec<i64> = sqlx::query_scalar!(
r#"SELECT id as "id!" FROM repo_seq WHERE seq IS NULL ORDER BY id LIMIT $1"#,
SEQUENCER_BATCH_SIZE
)
.fetch_all(&mut *tx)
.await
.map_err(map_sqlx_error)?;
let count = pending_ids.len() as i64;
if count == 0 {
tx.commit().await.map_err(map_sqlx_error)?;
return Ok(0);
}
let start = sqlx::query_scalar!(r#"SELECT nextval('firehose_seq') as "seq!""#)
.fetch_one(&mut *tx)
.await
.map_err(map_sqlx_error)?;
if count > 1 {
sqlx::query!("SELECT setval('firehose_seq', $1)", start + count - 1)
.fetch_one(&mut *tx)
.await
.map_err(map_sqlx_error)?;
}
let assigned = sqlx::query!(
r#"UPDATE repo_seq r
SET seq = $1::bigint + (o.ord - 1)
FROM unnest($2::bigint[]) WITH ORDINALITY AS o(id, ord)
WHERE r.id = o.id"#,
start,
&pending_ids
)
.execute(&mut *tx)
.await
.map_err(map_sqlx_error)?
.rows_affected();
tx.commit().await.map_err(map_sqlx_error)?;
if assigned > 0 {
notify_repo_pending(pool).await;
}
Ok(count)
}
pub struct PostgresRepoRepository {
pool: PgPool,
}
@@ -716,13 +774,12 @@ impl RepoRepository for PostgresRepoRepository {
Ok(rows.into_iter().map(|(cid,)| cid).collect())
}
async fn insert_commit_event(&self, data: &CommitEventData) -> Result<SequenceNumber, DbError> {
async fn insert_commit_event(&self, data: &CommitEventData) -> Result<(), DbError> {
let (block_cids, block_data) = inline_to_paired_blocks(data.blocks.as_deref());
let seq = sqlx::query_scalar!(
sqlx::query!(
r#"
INSERT INTO repo_seq (did, event_type, commit_cid, prev_cid, ops, blobs, block_cids, block_data, prev_data_cid, rev)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)
RETURNING seq
"#,
data.did.as_str(),
data.event_type.as_str(),
@@ -735,67 +792,54 @@ impl RepoRepository for PostgresRepoRepository {
data.prev_data_cid.as_ref().map(|c| c.as_str()),
data.rev
)
.fetch_one(&self.pool)
.execute(&self.pool)
.await
.map_err(map_sqlx_error)?;
Ok(seq.into())
notify_repo_pending(&self.pool).await;
Ok(())
}
async fn insert_identity_event(
&self,
did: &Did,
handle: Option<&Handle>,
) -> Result<SequenceNumber, DbError> {
) -> Result<(), DbError> {
let handle_str = handle.map(|h| h.as_str());
let seq = sqlx::query_scalar!(
sqlx::query!(
r#"
INSERT INTO repo_seq (did, event_type, handle)
VALUES ($1, 'identity', $2)
RETURNING seq
"#,
did.as_str(),
handle_str
)
.fetch_one(&self.pool)
.execute(&self.pool)
.await
.map_err(map_sqlx_error)?;
sqlx::query(&format!("NOTIFY repo_updates, '{}'", seq))
.execute(&self.pool)
.await
.map_err(map_sqlx_error)?;
Ok(seq.into())
notify_repo_pending(&self.pool).await;
Ok(())
}
async fn insert_account_event(
&self,
did: &Did,
status: AccountStatus,
) -> Result<SequenceNumber, DbError> {
async fn insert_account_event(&self, did: &Did, status: AccountStatus) -> Result<(), DbError> {
let active = status.is_active();
let status_str = status.for_firehose();
let seq = sqlx::query_scalar!(
sqlx::query!(
r#"
INSERT INTO repo_seq (did, event_type, active, status)
VALUES ($1, 'account', $2, $3)
RETURNING seq
"#,
did.as_str(),
active,
status_str
)
.fetch_one(&self.pool)
.execute(&self.pool)
.await
.map_err(map_sqlx_error)?;
sqlx::query(&format!("NOTIFY repo_updates, '{}'", seq))
.execute(&self.pool)
.await
.map_err(map_sqlx_error)?;
Ok(seq.into())
notify_repo_pending(&self.pool).await;
Ok(())
}
async fn insert_sync_event(
@@ -804,18 +848,17 @@ impl RepoRepository for PostgresRepoRepository {
commit_cid: &CidLink,
rev: Option<&str>,
commit_bytes: &[u8],
) -> Result<SequenceNumber, DbError> {
) -> Result<(), DbError> {
let cid_bytes = commit_cid
.to_cid()
.map(|c| c.to_bytes())
.unwrap_or_default();
let block_cids: Vec<Vec<u8>> = vec![cid_bytes];
let block_data: Vec<Vec<u8>> = vec![commit_bytes.to_vec()];
let seq = sqlx::query_scalar!(
sqlx::query!(
r#"
INSERT INTO repo_seq (did, event_type, commit_cid, rev, block_cids, block_data)
VALUES ($1, 'sync', $2, $3, $4, $5)
RETURNING seq
"#,
did.as_str(),
commit_cid.as_str(),
@@ -823,16 +866,12 @@ impl RepoRepository for PostgresRepoRepository {
&block_cids as &[Vec<u8>],
&block_data as &[Vec<u8>]
)
.fetch_one(&self.pool)
.execute(&self.pool)
.await
.map_err(map_sqlx_error)?;
sqlx::query(&format!("NOTIFY repo_updates, '{}'", seq))
.execute(&self.pool)
.await
.map_err(map_sqlx_error)?;
Ok(seq.into())
notify_repo_pending(&self.pool).await;
Ok(())
}
async fn insert_genesis_commit_event(
@@ -843,7 +882,7 @@ impl RepoRepository for PostgresRepoRepository {
rev: &str,
commit_bytes: &[u8],
mst_root_bytes: &[u8],
) -> Result<SequenceNumber, DbError> {
) -> Result<(), DbError> {
let ops = serde_json::json!([]);
let blobs: Vec<String> = vec![];
let commit_cid_bytes = commit_cid
@@ -858,11 +897,10 @@ impl RepoRepository for PostgresRepoRepository {
let block_data: Vec<Vec<u8>> = vec![commit_bytes.to_vec(), mst_root_bytes.to_vec()];
let prev_cid: Option<&str> = None;
let seq = sqlx::query_scalar!(
sqlx::query!(
r#"
INSERT INTO repo_seq (did, event_type, commit_cid, prev_cid, ops, blobs, block_cids, block_data, rev)
VALUES ($1, 'commit', $2, $3::TEXT, $4, $5, $6, $7, $8)
RETURNING seq
"#,
did.as_str(),
commit_cid.as_str(),
@@ -873,27 +911,22 @@ impl RepoRepository for PostgresRepoRepository {
&block_data as &[Vec<u8>],
rev
)
.fetch_one(&self.pool)
.execute(&self.pool)
.await
.map_err(map_sqlx_error)?;
sqlx::query(&format!("NOTIFY repo_updates, '{}'", seq))
.execute(&self.pool)
.await
.map_err(map_sqlx_error)?;
Ok(seq.into())
notify_repo_pending(&self.pool).await;
Ok(())
}
async fn delete_sequences_except(
&self,
did: &Did,
keep_seq: SequenceNumber,
) -> Result<(), DbError> {
async fn purge_did_events_keeping_latest(&self, did: &Did) -> Result<(), DbError> {
sqlx::query!(
"DELETE FROM repo_seq WHERE did = $1 AND seq != $2",
did.as_str(),
keep_seq.as_i64()
r#"
DELETE FROM repo_seq
WHERE did = $1
AND id <> (SELECT id FROM repo_seq WHERE did = $1 ORDER BY id DESC LIMIT 1)
"#,
did.as_str()
)
.execute(&self.pool)
.await
@@ -902,6 +935,42 @@ impl RepoRepository for PostgresRepoRepository {
Ok(())
}
async fn assign_pending_sequences(&self) -> Result<u64, DbError> {
let mut total: u64 = 0;
loop {
let mut tx = self.pool.begin().await.map_err(map_sqlx_error)?;
let acquired = sqlx::query_scalar!(
r#"SELECT pg_try_advisory_xact_lock($1) as "locked!""#,
SEQUENCER_LOCK_KEY
)
.fetch_one(&mut *tx)
.await
.map_err(map_sqlx_error)?;
if !acquired {
return Ok(total);
}
let count = assign_one_batch(tx, &self.pool).await?;
total += count as u64;
if count < SEQUENCER_BATCH_SIZE {
return Ok(total);
}
}
}
async fn flush_pending_sequences(&self) -> Result<(), DbError> {
loop {
let mut tx = self.pool.begin().await.map_err(map_sqlx_error)?;
sqlx::query!("SELECT pg_advisory_xact_lock($1)", SEQUENCER_LOCK_KEY)
.execute(&mut *tx)
.await
.map_err(map_sqlx_error)?;
let count = assign_one_batch(tx, &self.pool).await?;
if count < SEQUENCER_BATCH_SIZE {
return Ok(());
}
}
}
async fn prune_events_older_than(&self, cutoff: DateTime<Utc>) -> Result<PruneCount, DbError> {
let result = sqlx::query!("DELETE FROM repo_seq WHERE created_at < $1", cutoff)
.execute(&self.pool)
@@ -965,7 +1034,7 @@ impl RepoRepository for PostgresRepoRepository {
Some(lim) => {
let rows = sqlx::query_as!(
SequencedEventRow,
r#"SELECT seq, did, created_at, event_type as "event_type: RepoEventType", commit_cid, prev_cid, prev_data_cid,
r#"SELECT seq as "seq!", did, created_at, event_type as "event_type: RepoEventType", commit_cid, prev_cid, prev_data_cid,
ops, blobs, block_cids, block_data, blocks_cids, handle, active, status, rev
FROM repo_seq
WHERE seq > $1
@@ -982,7 +1051,7 @@ impl RepoRepository for PostgresRepoRepository {
None => {
let rows = sqlx::query_as!(
SequencedEventRow,
r#"SELECT seq, did, created_at, event_type as "event_type: RepoEventType", commit_cid, prev_cid, prev_data_cid,
r#"SELECT seq as "seq!", did, created_at, event_type as "event_type: RepoEventType", commit_cid, prev_cid, prev_data_cid,
ops, blobs, block_cids, block_data, blocks_cids, handle, active, status, rev
FROM repo_seq
WHERE seq > $1
@@ -1004,7 +1073,7 @@ impl RepoRepository for PostgresRepoRepository {
) -> Result<Vec<SequencedEvent>, DbError> {
let rows = sqlx::query_as!(
SequencedEventRow,
r#"SELECT seq, did, created_at, event_type as "event_type: RepoEventType", commit_cid, prev_cid, prev_data_cid,
r#"SELECT seq as "seq!", did, created_at, event_type as "event_type: RepoEventType", commit_cid, prev_cid, prev_data_cid,
ops, blobs, block_cids, block_data, blocks_cids, handle, active, status, rev
FROM repo_seq
WHERE seq > $1 AND seq < $2
@@ -1024,7 +1093,7 @@ impl RepoRepository for PostgresRepoRepository {
) -> Result<Option<SequencedEvent>, DbError> {
let row = sqlx::query_as!(
SequencedEventRow,
r#"SELECT seq, did, created_at, event_type as "event_type: RepoEventType", commit_cid, prev_cid, prev_data_cid,
r#"SELECT seq as "seq!", did, created_at, event_type as "event_type: RepoEventType", commit_cid, prev_cid, prev_data_cid,
ops, blobs, block_cids, block_data, blocks_cids, handle, active, status, rev
FROM repo_seq
WHERE seq = $1"#,
@@ -1043,7 +1112,7 @@ impl RepoRepository for PostgresRepoRepository {
) -> Result<Vec<SequencedEvent>, DbError> {
let rows = sqlx::query_as!(
SequencedEventRow,
r#"SELECT seq, did, created_at, event_type as "event_type: RepoEventType", commit_cid, prev_cid, prev_data_cid,
r#"SELECT seq as "seq!", did, created_at, event_type as "event_type: RepoEventType", commit_cid, prev_cid, prev_data_cid,
ops, blobs, block_cids, block_data, blocks_cids, handle, active, status, rev
FROM repo_seq
WHERE seq > $1
@@ -1104,14 +1173,6 @@ impl RepoRepository for PostgresRepoRepository {
Ok(cid.map(CidLink::from))
}
async fn notify_update(&self, seq: SequenceNumber) -> Result<(), DbError> {
sqlx::query(&format!("NOTIFY repo_updates, '{}'", seq.as_i64()))
.execute(&self.pool)
.await
.map_err(map_sqlx_error)?;
Ok(())
}
async fn import_repo_data(
&self,
user_id: Uuid,
@@ -1402,28 +1463,27 @@ impl RepoRepository for PostgresRepoRepository {
let event = input.commit_event;
let (event_block_cids, event_block_data) = inline_into_paired_blocks(event.blocks);
let seq: i64 = sqlx::query_scalar(
sqlx::query!(
r#"
INSERT INTO repo_seq (did, event_type, commit_cid, prev_cid, ops, blobs, block_cids, block_data, prev_data_cid, rev)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)
RETURNING seq
"#,
event.did.as_str(),
event.event_type.as_str(),
event.commit_cid.as_ref().map(|c| c.as_str()),
event.prev_cid.as_ref().map(|c| c.as_str()),
event.ops,
event.blobs.as_deref(),
&event_block_cids as &[Vec<u8>],
&event_block_data as &[Vec<u8>],
event.prev_data_cid.as_ref().map(|c| c.as_str()),
event.rev
)
.bind(event.did.as_str())
.bind(event.event_type.as_str())
.bind(event.commit_cid.as_ref().map(|c| c.as_str()))
.bind(event.prev_cid.as_ref().map(|c| c.as_str()))
.bind(&event.ops)
.bind(&event.blobs)
.bind(&event_block_cids)
.bind(&event_block_data)
.bind(event.prev_data_cid.as_ref().map(|c| c.as_str()))
.bind(&event.rev)
.fetch_one(&mut *tx)
.execute(&mut *tx)
.await
.map_err(|e| ApplyCommitError::Database(e.to_string()))?;
sqlx::query(&format!("NOTIFY repo_updates, '{}'", seq))
sqlx::query!("NOTIFY repo_updates")
.execute(&mut *tx)
.await
.map_err(|e| ApplyCommitError::Database(e.to_string()))?;
@@ -1432,10 +1492,7 @@ impl RepoRepository for PostgresRepoRepository {
.await
.map_err(|e| ApplyCommitError::Database(e.to_string()))?;
Ok(tranquil_db_traits::ApplyCommitResult {
seq,
is_account_active,
})
Ok(tranquil_db_traits::ApplyCommitResult { is_account_active })
}
async fn get_users_without_blocks(&self) -> Result<Vec<UserWithoutBlocks>, DbError> {
+15 -12
View File
@@ -47,7 +47,7 @@ pub(crate) fn map_sqlx_error(e: sqlx::Error) -> DbError {
impl UserRepository for PostgresUserRepository {
async fn get_by_did(&self, did: &Did) -> Result<Option<UserRow>, DbError> {
let row = sqlx::query!(
r#"SELECT id, did, handle, email, created_at, deactivated_at, takedown_ref, is_admin
r#"SELECT id, did, handle, email, created_at, deactivated_at, takedown_ref, is_admin, inbound_migration
FROM users WHERE did = $1"#,
did.as_str()
)
@@ -64,12 +64,13 @@ impl UserRepository for PostgresUserRepository {
deactivated_at: r.deactivated_at,
takedown_ref: r.takedown_ref,
is_admin: r.is_admin,
inbound_migration: r.inbound_migration,
}))
}
async fn get_by_handle(&self, handle: &Handle) -> Result<Option<UserRow>, DbError> {
let row = sqlx::query!(
r#"SELECT id, did, handle, email, created_at, deactivated_at, takedown_ref, is_admin
r#"SELECT id, did, handle, email, created_at, deactivated_at, takedown_ref, is_admin, inbound_migration
FROM users WHERE handle = $1"#,
handle.as_str()
)
@@ -86,6 +87,7 @@ impl UserRepository for PostgresUserRepository {
deactivated_at: r.deactivated_at,
takedown_ref: r.takedown_ref,
is_admin: r.is_admin,
inbound_migration: r.inbound_migration,
}))
}
@@ -1863,7 +1865,7 @@ impl UserRepository for PostgresUserRepository {
async fn activate_account(&self, did: &Did) -> Result<bool, DbError> {
let result = sqlx::query!(
"UPDATE users SET deactivated_at = NULL WHERE did = $1",
"UPDATE users SET deactivated_at = NULL, inbound_migration = FALSE WHERE did = $1",
did.as_str()
)
.execute(&self.pool)
@@ -2289,7 +2291,7 @@ impl UserRepository for PostgresUserRepository {
.collect())
}
async fn delete_account_with_firehose(&self, user_id: Uuid, did: &Did) -> Result<i64, DbError> {
async fn delete_account_with_firehose(&self, user_id: Uuid, did: &Did) -> Result<(), DbError> {
let mut tx = self.pool.begin().await.map_err(map_sqlx_error)?;
sqlx::query!("DELETE FROM blobs WHERE created_by_user = $1", user_id)
@@ -2368,11 +2370,11 @@ impl UserRepository for PostgresUserRepository {
.await
.map_err(map_sqlx_error)?;
let account_seq: i64 = sqlx::query_scalar!(
let event_id: i64 = sqlx::query_scalar!(
r#"
INSERT INTO repo_seq (did, event_type, active, status)
VALUES ($1, 'account', false, 'deleted')
RETURNING seq
RETURNING id
"#,
did.as_str()
)
@@ -2381,9 +2383,9 @@ impl UserRepository for PostgresUserRepository {
.map_err(map_sqlx_error)?;
sqlx::query!(
"DELETE FROM repo_seq WHERE did = $1 AND seq != $2",
"DELETE FROM repo_seq WHERE did = $1 AND id <> $2",
did.as_str(),
account_seq
event_id
)
.execute(&mut *tx)
.await
@@ -2391,12 +2393,12 @@ impl UserRepository for PostgresUserRepository {
tx.commit().await.map_err(map_sqlx_error)?;
sqlx::query(&format!("NOTIFY repo_updates, '{}'", account_seq))
sqlx::query!("NOTIFY repo_updates")
.execute(&self.pool)
.await
.map_err(map_sqlx_error)?;
Ok(account_seq)
Ok(())
}
async fn create_password_account(
@@ -2426,8 +2428,8 @@ impl UserRepository for PostgresUserRepository {
handle, email, did, password_hash,
preferred_comms_channel,
discord_username, telegram_username, signal_username,
is_admin, deactivated_at, email_verified
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, FALSE) RETURNING id"#,
is_admin, deactivated_at, inbound_migration, email_verified
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, FALSE) RETURNING id"#,
)
.bind(input.handle.as_str())
.bind(&input.email)
@@ -2439,6 +2441,7 @@ impl UserRepository for PostgresUserRepository {
.bind(&input.signal_username)
.bind(is_first_user)
.bind(input.deactivated_at)
.bind(input.inbound_migration)
.fetch_one(&mut *tx)
.await;
+1
View File
@@ -24,3 +24,4 @@ urlencoding = { workspace = true, optional = true }
[dev-dependencies]
wiremock = { workspace = true }
tokio = { workspace = true }
futures = { workspace = true }
+422 -31
View File
@@ -5,46 +5,102 @@ use std::collections::{HashMap, VecDeque};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, Instant};
use tokio::sync::Notify;
const NEGATIVE_CACHE_TTL: Duration = Duration::from_secs(24 * 60 * 60);
const POSITIVE_CACHE_TTL: Duration = Duration::from_secs(24 * 60 * 60);
const REFRESH_FAILURE_BACKOFF: Duration = Duration::from_secs(60);
const MAX_DYNAMIC_SCHEMAS: usize = 1024;
struct NegativeEntry {
expires_at: Instant,
}
struct PositiveEntry {
doc: Arc<LexiconDoc>,
expires_at: Instant,
}
pub(crate) enum CacheEntry {
Fresh(Arc<LexiconDoc>),
Stale(Arc<LexiconDoc>),
}
impl CacheEntry {
#[cfg(test)]
fn is_fresh(&self) -> bool {
matches!(self, Self::Fresh(_))
}
}
struct SchemaStore {
schemas: HashMap<String, Arc<LexiconDoc>>,
schemas: HashMap<String, PositiveEntry>,
insertion_order: VecDeque<String>,
}
pub struct DynamicRegistry {
store: RwLock<SchemaStore>,
negative_cache: RwLock<HashMap<String, NegativeEntry>>,
in_flight: RwLock<HashMap<String, Arc<Notify>>>,
network_disabled: AtomicBool,
}
struct InFlightGuard<'a> {
registry: &'a DynamicRegistry,
nsid: String,
}
impl Drop for InFlightGuard<'_> {
fn drop(&mut self) {
let notify = self.registry.in_flight.write().remove(&self.nsid);
if let Some(n) = notify {
n.notify_waiters();
}
}
}
impl DynamicRegistry {
pub fn new() -> Self {
let network_disabled =
std::env::var("TRANQUIL_LEXICON_OFFLINE").is_ok_and(|v| v == "1" || v == "true");
Self {
store: RwLock::new(SchemaStore {
schemas: HashMap::new(),
insertion_order: VecDeque::new(),
}),
negative_cache: RwLock::new(HashMap::new()),
network_disabled: AtomicBool::new(network_disabled),
in_flight: RwLock::new(HashMap::new()),
network_disabled: AtomicBool::new(false),
}
}
#[allow(dead_code)]
pub fn from_env() -> Self {
let registry = Self::new();
let disabled =
std::env::var("TRANQUIL_LEXICON_OFFLINE").is_ok_and(|v| v == "1" || v == "true");
registry.set_network_disabled(disabled);
registry
}
pub fn set_network_disabled(&self, disabled: bool) {
self.network_disabled.store(disabled, Ordering::Relaxed);
}
pub fn get(&self, nsid: &str) -> Option<Arc<LexiconDoc>> {
self.store.read().schemas.get(nsid).cloned()
pub fn get_cached(&self, nsid: &str) -> Option<Arc<LexiconDoc>> {
self.store
.read()
.schemas
.get(nsid)
.map(|e| Arc::clone(&e.doc))
}
pub(crate) fn get_entry(&self, nsid: &str) -> Option<CacheEntry> {
let now = Instant::now();
self.store.read().schemas.get(nsid).map(|e| {
if e.expires_at > now {
CacheEntry::Fresh(Arc::clone(&e.doc))
} else {
CacheEntry::Stale(Arc::clone(&e.doc))
}
})
}
pub fn is_negative_cached(&self, nsid: &str) -> bool {
@@ -56,7 +112,7 @@ impl DynamicRegistry {
fn insert_negative(&self, nsid: &str) {
let mut cache = self.negative_cache.write();
if cache.len() > MAX_DYNAMIC_SCHEMAS {
if cache.len() >= MAX_DYNAMIC_SCHEMAS {
let now = Instant::now();
cache.retain(|_, entry| entry.expires_at > now);
}
@@ -87,29 +143,95 @@ impl DynamicRegistry {
});
}
if store
.schemas
.insert(nsid.clone(), Arc::clone(&arc))
.is_some()
{
let entry = PositiveEntry {
doc: Arc::clone(&arc),
expires_at: Instant::now() + POSITIVE_CACHE_TTL,
};
if store.schemas.insert(nsid.clone(), entry).is_some() {
store.insertion_order.retain(|k| k != &nsid);
}
store.insertion_order.push_back(nsid.clone());
drop(store);
self.negative_cache.write().remove(&arc.id);
arc
}
fn bump_expiry(&self, nsid: &str, duration: Duration) {
let mut store = self.store.write();
if let Some(entry) = store.schemas.get_mut(nsid) {
entry.expires_at = Instant::now() + duration;
}
}
pub async fn resolve_and_cache(&self, nsid: &str) -> Result<Arc<LexiconDoc>, ResolveError> {
if let Some(doc) = self.get(nsid) {
return Ok(doc);
self.resolve_and_cache_with(nsid, |n| async move { resolve_lexicon(&n).await })
.await
}
async fn resolve_and_cache_with<F, Fut>(
&self,
nsid: &str,
resolver: F,
) -> Result<Arc<LexiconDoc>, ResolveError>
where
F: FnOnce(String) -> Fut,
Fut: std::future::Future<Output = Result<LexiconDoc, ResolveError>>,
{
match self.get_entry(nsid) {
Some(CacheEntry::Fresh(doc)) => Ok(doc),
Some(CacheEntry::Stale(stale)) => self.refresh_stale(nsid, stale, resolver).await,
None => self.resolve_fresh(nsid, resolver).await,
}
}
async fn refresh_stale<F, Fut>(
&self,
nsid: &str,
stale: Arc<LexiconDoc>,
resolver: F,
) -> Result<Arc<LexiconDoc>, ResolveError>
where
F: FnOnce(String) -> Fut,
Fut: std::future::Future<Output = Result<LexiconDoc, ResolveError>>,
{
if self.network_disabled.load(Ordering::Relaxed) {
return Ok(stale);
}
match self.acquire_leadership(nsid) {
Some(_guard) => match resolver(nsid.to_string()).await {
Ok(doc) => Ok(self.insert_schema(doc)),
Err(e) => {
self.bump_expiry(nsid, REFRESH_FAILURE_BACKOFF);
tracing::warn!(
nsid = nsid,
error = %e,
"lexicon refresh failed, serving stale cached entry"
);
Ok(stale)
}
},
None => {
self.wait_for_leader(nsid).await;
Ok(self.get_cached(nsid).unwrap_or(stale))
}
}
}
async fn resolve_fresh<F, Fut>(
&self,
nsid: &str,
resolver: F,
) -> Result<Arc<LexiconDoc>, ResolveError>
where
F: FnOnce(String) -> Fut,
Fut: std::future::Future<Output = Result<LexiconDoc, ResolveError>>,
{
if self.network_disabled.load(Ordering::Relaxed) {
return Err(ResolveError::NetworkDisabled);
}
if self.is_negative_cached(nsid) {
return Err(ResolveError::NegativelyCached {
nsid: nsid.to_string(),
@@ -117,19 +239,73 @@ impl DynamicRegistry {
});
}
match resolve_lexicon(nsid).await {
Ok(doc) => Ok(self.insert_schema(doc)),
Err(e) => {
tracing::debug!(nsid = nsid, error = %e, "caching negative resolution result");
self.insert_negative(nsid);
Err(e)
match self.acquire_leadership(nsid) {
Some(_guard) => match resolver(nsid.to_string()).await {
Ok(doc) => Ok(self.insert_schema(doc)),
Err(e) => {
self.insert_negative(nsid);
tracing::debug!(nsid = nsid, error = %e, "caching negative resolution result");
Err(e)
}
},
None => {
self.wait_for_leader(nsid).await;
match self.get_cached(nsid) {
Some(doc) => Ok(doc),
None if self.is_negative_cached(nsid) => Err(ResolveError::NegativelyCached {
nsid: nsid.to_string(),
ttl_secs: NEGATIVE_CACHE_TTL.as_secs(),
}),
None => Err(ResolveError::LeaderAborted {
nsid: nsid.to_string(),
}),
}
}
}
}
fn acquire_leadership(&self, nsid: &str) -> Option<InFlightGuard<'_>> {
let mut map = self.in_flight.write();
if map.contains_key(nsid) {
None
} else {
map.insert(nsid.to_string(), Arc::new(Notify::new()));
Some(InFlightGuard {
registry: self,
nsid: nsid.to_string(),
})
}
}
async fn wait_for_leader(&self, nsid: &str) {
let notify = {
let map = self.in_flight.read();
match map.get(nsid) {
Some(n) => Arc::clone(n),
None => return,
}
};
let notified = notify.notified();
tokio::pin!(notified);
notified.as_mut().enable();
let still_active = self.in_flight.read().contains_key(nsid);
if !still_active {
return;
}
notified.as_mut().await;
}
pub fn schema_count(&self) -> usize {
self.store.read().schemas.len()
}
#[cfg(test)]
fn expire_now(&self, nsid: &str) {
let mut store = self.store.write();
if let Some(entry) = store.schemas.get_mut(nsid) {
entry.expires_at = Instant::now();
}
}
}
impl Default for DynamicRegistry {
@@ -171,7 +347,7 @@ mod tests {
#[test]
fn test_empty_lookup() {
let registry = DynamicRegistry::new();
assert!(registry.get("com.example.nonexistent").is_none());
assert!(registry.get_cached("com.example.nonexistent").is_none());
assert_eq!(registry.schema_count(), 0);
}
@@ -188,9 +364,12 @@ mod tests {
assert_eq!(arc.id, "com.example.test");
assert_eq!(registry.schema_count(), 1);
let retrieved = registry.get("com.example.test");
let retrieved = registry.get_cached("com.example.test");
assert!(retrieved.is_some());
assert_eq!(retrieved.unwrap().id, "com.example.test");
let entry = registry.get_entry("com.example.test").unwrap();
assert!(entry.is_fresh(), "freshly inserted entry must be fresh");
}
#[test]
@@ -210,6 +389,218 @@ mod tests {
assert!(!registry.is_negative_cached("com.example.test"));
}
#[test]
fn test_positive_entry_reports_stale_after_ttl() {
let registry = DynamicRegistry::new();
let doc = LexiconDoc {
lexicon: 1,
id: "pet.nel.stale".to_string(),
defs: HashMap::new(),
};
registry.insert_schema(doc);
assert!(registry.get_entry("pet.nel.stale").unwrap().is_fresh());
registry.expire_now("pet.nel.stale");
assert!(
!registry.get_entry("pet.nel.stale").unwrap().is_fresh(),
"entry past expiry must be reported stale"
);
}
#[tokio::test]
async fn test_stale_served_on_resolve_failure() {
let registry = DynamicRegistry::new();
let doc = LexiconDoc {
lexicon: 1,
id: "pet.nel.flaky".to_string(),
defs: HashMap::new(),
};
registry.insert_schema(doc);
registry.expire_now("pet.nel.flaky");
let result = registry
.resolve_and_cache_with("pet.nel.flaky", |n| async move {
Err::<LexiconDoc, _>(ResolveError::DnsLookup {
domain: n,
reason: "simulated failure".to_string(),
})
})
.await;
let served = result.expect("stale entry must be served when refresh fails");
assert_eq!(served.id, "pet.nel.flaky");
assert!(
registry.get_entry("pet.nel.flaky").unwrap().is_fresh(),
"failed refresh must bump expiry so subsequent lookups skip the resolver"
);
assert!(
!registry.is_negative_cached("pet.nel.flaky"),
"stale refresh failure must not poison negative cache"
);
}
#[tokio::test]
async fn test_fresh_hit_skips_resolver() {
let registry = DynamicRegistry::new();
let doc = LexiconDoc {
lexicon: 1,
id: "pet.nel.fresh".to_string(),
defs: HashMap::new(),
};
registry.insert_schema(doc);
let result = registry
.resolve_and_cache_with("pet.nel.fresh", |_| async move {
panic!("resolver must not run on fresh hit")
})
.await;
assert!(result.is_ok());
}
#[tokio::test]
async fn test_stale_served_when_network_disabled() {
let registry = DynamicRegistry::new();
let doc = LexiconDoc {
lexicon: 1,
id: "pet.nel.offline".to_string(),
defs: HashMap::new(),
};
registry.insert_schema(doc);
registry.expire_now("pet.nel.offline");
registry.set_network_disabled(true);
let result = registry
.resolve_and_cache_with("pet.nel.offline", |_| async move {
panic!("resolver must not run when network disabled")
})
.await;
assert!(result.is_ok());
}
#[tokio::test]
async fn test_successful_refresh_updates_cached_at() {
let registry = DynamicRegistry::new();
let doc = LexiconDoc {
lexicon: 1,
id: "pet.nel.refresh".to_string(),
defs: HashMap::new(),
};
registry.insert_schema(doc);
registry.expire_now("pet.nel.refresh");
assert!(!registry.get_entry("pet.nel.refresh").unwrap().is_fresh());
let refreshed = registry
.resolve_and_cache_with("pet.nel.refresh", |n| async move {
Ok(LexiconDoc {
lexicon: 1,
id: n,
defs: HashMap::new(),
})
})
.await
.unwrap();
assert_eq!(refreshed.id, "pet.nel.refresh");
assert!(
registry.get_entry("pet.nel.refresh").unwrap().is_fresh(),
"refresh must restore freshness"
);
}
#[tokio::test]
async fn test_single_flight_dedups_concurrent_resolves() {
use std::sync::atomic::AtomicUsize;
let registry = Arc::new(DynamicRegistry::new());
let calls = Arc::new(AtomicUsize::new(0));
let tasks: Vec<_> = (0..16)
.map(|_| {
let registry = Arc::clone(&registry);
let calls = Arc::clone(&calls);
tokio::spawn(async move {
registry
.resolve_and_cache_with("pet.nel.herd", |n| {
let calls = Arc::clone(&calls);
async move {
calls.fetch_add(1, Ordering::SeqCst);
tokio::time::sleep(Duration::from_millis(50)).await;
Ok(LexiconDoc {
lexicon: 1,
id: n,
defs: HashMap::new(),
})
}
})
.await
})
})
.collect();
let results = futures_collect(tasks).await;
results
.iter()
.for_each(|r| assert!(r.is_ok(), "all single-flight callers must succeed"));
assert_eq!(
calls.load(Ordering::SeqCst),
1,
"single-flight must coalesce concurrent resolves"
);
assert_eq!(registry.schema_count(), 1);
}
#[tokio::test]
async fn test_single_flight_followers_observe_leader_failure() {
use std::sync::atomic::AtomicUsize;
let registry = Arc::new(DynamicRegistry::new());
let calls = Arc::new(AtomicUsize::new(0));
let tasks: Vec<_> = (0..8)
.map(|_| {
let registry = Arc::clone(&registry);
let calls = Arc::clone(&calls);
tokio::spawn(async move {
registry
.resolve_and_cache_with("pet.nel.failHerd", |n| {
let calls = Arc::clone(&calls);
async move {
calls.fetch_add(1, Ordering::SeqCst);
tokio::time::sleep(Duration::from_millis(50)).await;
Err::<LexiconDoc, _>(ResolveError::DnsLookup {
domain: n,
reason: "simulated".to_string(),
})
}
})
.await
})
})
.collect();
let results = futures_collect(tasks).await;
results
.iter()
.for_each(|r| assert!(r.is_err(), "all followers must observe leader failure"));
assert_eq!(
calls.load(Ordering::SeqCst),
1,
"single-flight must coalesce failing resolves too"
);
assert!(registry.is_negative_cached("pet.nel.failHerd"));
}
async fn futures_collect<T>(handles: Vec<tokio::task::JoinHandle<T>>) -> Vec<T> {
futures::future::join_all(handles)
.await
.into_iter()
.map(|r| r.expect("task panicked"))
.collect()
}
#[test]
fn test_eviction_is_fifo() {
let registry = DynamicRegistry::new();
@@ -217,7 +608,7 @@ mod tests {
(0..MAX_DYNAMIC_SCHEMAS).for_each(|i| {
let doc = LexiconDoc {
lexicon: 1,
id: format!("com.example.schema{}", i),
id: format!("pet.nel.schema{}", i),
defs: HashMap::new(),
};
registry.insert_schema(doc);
@@ -226,23 +617,23 @@ mod tests {
let trigger = LexiconDoc {
lexicon: 1,
id: "com.example.trigger".to_string(),
id: "pet.nel.trigger".to_string(),
defs: HashMap::new(),
};
registry.insert_schema(trigger);
assert!(
registry.get("com.example.schema0").is_none(),
registry.get_cached("pet.nel.schema0").is_none(),
"oldest entry should be evicted"
);
assert!(
registry.get("com.example.trigger").is_some(),
registry.get_cached("pet.nel.trigger").is_some(),
"newly inserted entry should exist"
);
let evict_count = MAX_DYNAMIC_SCHEMAS / 4;
assert!(
registry
.get(&format!("com.example.schema{}", evict_count))
.get_cached(&format!("pet.nel.schema{}", evict_count))
.is_some(),
"entry after eviction window should survive"
);
@@ -253,7 +644,7 @@ mod tests {
let registry = DynamicRegistry::new();
let doc = LexiconDoc {
lexicon: 1,
id: "com.example.tracked".to_string(),
id: "pet.nel.tracked".to_string(),
defs: HashMap::new(),
};
let arc = registry.insert_schema(doc);
@@ -265,7 +656,7 @@ mod tests {
(0..MAX_DYNAMIC_SCHEMAS).for_each(|i| {
registry.insert_schema(LexiconDoc {
lexicon: 1,
id: format!("com.example.filler{}", i),
id: format!("pet.nel.filler{}", i),
defs: HashMap::new(),
});
});
+2 -2
View File
@@ -25,7 +25,7 @@ impl LexiconRegistry {
Self {
schemas: HashMap::new(),
#[cfg(feature = "resolve")]
dynamic: crate::dynamic::DynamicRegistry::new(),
dynamic: crate::dynamic::DynamicRegistry::from_env(),
}
}
@@ -43,7 +43,7 @@ impl LexiconRegistry {
self.schemas.get(nsid).cloned().or_else(|| {
#[cfg(feature = "resolve")]
{
self.dynamic.get(nsid)
self.dynamic.get_cached(nsid)
}
#[cfg(not(feature = "resolve"))]
{
+2
View File
@@ -70,6 +70,8 @@ pub enum ResolveError {
NegativelyCached { nsid: String, ttl_secs: u64 },
#[error("network resolution disabled")]
NetworkDisabled,
#[error("leader task for {nsid} aborted before completion")]
LeaderAborted { nsid: String },
}
pub fn nsid_to_authority(nsid: &str) -> Result<String, ResolveError> {
@@ -310,6 +310,7 @@ pub async fn authorize_post(
State(state): State<AppState>,
_rate_limit: OAuthRateLimited<OAuthAuthorizeLimit>,
headers: HeaderMap,
client_ip: ClientIp,
Json(form): Json<AuthorizeSubmit>,
) -> Response {
let json_response = wants_json(&headers);
@@ -488,8 +489,7 @@ pub async fn authorize_post(
if !password_valid {
return show_login_error("Invalid identifier or password.", json_response);
}
let is_verified = user.channel_verification.has_any_verified();
if !is_verified {
if tranquil_api::server::verification_blocks_login(&user.channel_verification) {
let resend_info = tranquil_api::server::auto_resend_verification(&state, &user.did).await;
let handle = resend_info
.as_ref()
@@ -617,7 +617,7 @@ pub async fn authorize_post(
let device_data = DeviceData {
session_id: SessionId::generate(),
user_agent: extract_user_agent(&headers),
ip_address: extract_client_ip(&headers, None),
ip_address: client_ip.into_string(),
last_seen_at: Utc::now(),
};
if state
@@ -854,8 +854,7 @@ pub async fn authorize_select(
);
}
};
let is_verified = user.channel_verification.has_any_verified();
if !is_verified {
if tranquil_api::server::verification_blocks_login(&user.channel_verification) {
let resend_info = tranquil_api::server::auto_resend_verification(&state, &did).await;
return (
StatusCode::FORBIDDEN,
@@ -23,7 +23,7 @@ use tranquil_pds::rate_limit::{
};
use tranquil_pds::state::AppState;
use tranquil_pds::types::{Did, Handle, PlainPassword};
use tranquil_pds::util::extract_client_ip;
use tranquil_pds::util::ClientIp;
use tranquil_types::{AuthorizationCode, ClientId, DeviceId as DeviceIdType, RequestId};
use urlencoding::encode as url_encode;
@@ -175,10 +175,7 @@ pub async fn passkey_start(
}
}
async fn passkey_start_discoverable(
state: AppState,
request_id: RequestId,
) -> Response {
async fn passkey_start_discoverable(state: AppState, request_id: RequestId) -> Response {
let (rcr, auth_state) = match state.webauthn_config.start_discoverable_authentication() {
Ok(result) => result,
Err(e) => {
@@ -292,9 +289,7 @@ async fn passkey_start_named(
.into_response();
}
let is_verified = user.channel_verification.has_any_verified();
if !is_verified {
if tranquil_api::server::verification_blocks_login(&user.channel_verification) {
let resend_info = tranquil_api::server::auto_resend_verification(&state, &user.did).await;
return (
StatusCode::FORBIDDEN,
@@ -570,16 +565,13 @@ pub async fn passkey_finish(
Err(response) => return response,
},
None => {
let result = match passkey_finish_discoverable(
&state,
&credential,
&passkey_finish_request_id,
)
.await
{
Ok(result) => result,
Err(response) => return response,
};
let result =
match passkey_finish_discoverable(&state, &credential, &passkey_finish_request_id)
.await
{
Ok(result) => result,
Err(response) => return response,
};
if state
.repos
.oauth
@@ -160,8 +160,10 @@ pub async fn register_complete(
.into_response();
}
let is_verified = match state.repos.user.get_session_info_by_did(&did).await {
Ok(Some(info)) => info.channel_verification.has_any_verified(),
let login_blocked = match state.repos.user.get_session_info_by_did(&did).await {
Ok(Some(info)) => {
tranquil_api::server::verification_blocks_login(&info.channel_verification)
}
Ok(None) => {
return (
StatusCode::FORBIDDEN,
@@ -189,7 +191,7 @@ pub async fn register_complete(
}
};
if !is_verified {
if login_blocked {
let resend_info = tranquil_api::server::auto_resend_verification(&state, &did).await;
return (
StatusCode::FORBIDDEN,
@@ -300,6 +302,7 @@ pub async fn register_complete(
pub async fn establish_session(
State(state): State<AppState>,
headers: HeaderMap,
client_ip: ClientIp,
auth: tranquil_pds::auth::Auth<tranquil_pds::auth::Active>,
) -> Response {
let did = &auth.did;
@@ -317,7 +320,7 @@ pub async fn establish_session(
let device_data = DeviceData {
session_id: SessionId::generate(),
user_agent: extract_user_agent(&headers),
ip_address: extract_client_ip(&headers, None),
ip_address: client_ip.into_string(),
last_seen_at: Utc::now(),
};
@@ -75,6 +75,7 @@ pub async fn authorize_2fa_post(
State(state): State<AppState>,
_rate_limit: OAuthRateLimited<OAuthAuthorizeLimit>,
headers: HeaderMap,
client_ip: ClientIp,
Json(form): Json<Authorize2faSubmit>,
) -> Response {
let json_error = |status: StatusCode, error: &str, description: &str| -> Response {
@@ -251,7 +252,7 @@ pub async fn authorize_2fa_post(
let device_data = DeviceData {
session_id: SessionId::generate(),
user_agent: extract_user_agent(&headers),
ip_address: extract_client_ip(&headers, None),
ip_address: client_ip.into_string(),
last_seen_at: Utc::now(),
};
if state
@@ -12,7 +12,7 @@ use tranquil_pds::oauth::client::{build_client_metadata, delegation_oauth_urls};
use tranquil_pds::rate_limit::{LoginLimit, OAuthRateLimited, TotpVerifyLimit};
use tranquil_pds::state::AppState;
use tranquil_pds::types::PlainPassword;
use tranquil_pds::util::extract_client_ip;
use tranquil_pds::util::ClientIp;
use tranquil_types::did_doc::{extract_handle, extract_pds_endpoint};
use tranquil_types::{Did, RequestId};
@@ -402,6 +402,7 @@ pub struct DelegationTokenAuthSubmit {
pub async fn delegation_auth_token(
State(state): State<AppState>,
headers: HeaderMap,
client_ip: ClientIp,
auth: Auth<Active>,
Json(form): Json<DelegationTokenAuthSubmit>,
) -> Response {
@@ -428,7 +429,7 @@ pub async fn delegation_auth_token(
return resp;
}
let ip = extract_client_ip(&headers, None);
let ip = client_ip.into_string();
let user_agent = tranquil_pds::util::extract_user_agent(&headers);
finalize_delegation_auth(
@@ -138,21 +138,29 @@ pub async fn pushed_authorization_request(
}
fn determine_client_auth(request: &ParRequest) -> Result<ClientAuth, OAuthError> {
if let (Some(assertion), Some(assertion_type)) =
(&request.client_assertion, &request.client_assertion_type)
{
let assertion = request
.client_assertion
.as_deref()
.filter(|s| !s.is_empty());
let assertion_type = request
.client_assertion_type
.as_deref()
.filter(|s| !s.is_empty());
let secret = request.client_secret.as_deref().filter(|s| !s.is_empty());
if let (Some(assertion), Some(assertion_type)) = (assertion, assertion_type) {
if assertion_type != "urn:ietf:params:oauth:client-assertion-type:jwt-bearer" {
return Err(OAuthError::InvalidRequest(
"Unsupported client_assertion_type".to_string(),
));
}
return Ok(ClientAuth::PrivateKeyJwt {
client_assertion: assertion.clone(),
client_assertion: assertion.to_string(),
});
}
if let Some(secret) = &request.client_secret {
if let Some(secret) = secret {
return Ok(ClientAuth::SecretPost {
client_secret: secret.clone(),
client_secret: secret.to_string(),
});
}
Ok(ClientAuth::None)
@@ -138,13 +138,17 @@ impl TokenRequest {
}
};
let client_auth = match (self.client_assertion, self.client_assertion_type) {
let assertion = self.client_assertion.filter(|s| !s.is_empty());
let assertion_type = self.client_assertion_type.filter(|s| !s.is_empty());
let client_secret = self.client_secret.filter(|s| !s.is_empty());
let client_auth = match (assertion, assertion_type) {
(Some(assertion), Some(assertion_type)) => RequestClientAuth::PrivateKeyJwt {
client_id: self.client_id,
assertion,
assertion_type,
},
_ => match self.client_secret {
_ => match client_secret {
Some(secret) => RequestClientAuth::SecretPost {
client_id: self.client_id,
client_secret: secret,
@@ -402,13 +402,15 @@ async fn handle_sso_login(
}
};
let is_verified = match state
let login_blocked = match state
.repos
.user
.get_session_info_by_did(&identity.did)
.await
{
Ok(Some(info)) => info.channel_verification.has_any_verified(),
Ok(Some(info)) => {
tranquil_api::server::verification_blocks_login(&info.channel_verification)
}
Ok(None) => {
tracing::error!("User not found for SSO login: {}", identity.did);
return redirect_to_error("Account not found");
@@ -419,7 +421,7 @@ async fn handle_sso_login(
}
};
if !is_verified {
if login_blocked {
tracing::warn!(
did = %identity.did,
provider = %provider.as_str(),
+3
View File
@@ -23,3 +23,6 @@ sha2 = { workspace = true }
tokio = { workspace = true }
tracing = { workspace = true }
uuid = { workspace = true }
[features]
native-tls-roots = []
+15 -12
View File
@@ -78,18 +78,21 @@ impl ClientMetadataCache {
Self {
cache: Arc::new(RwLock::new(HashMap::new())),
jwks_cache: Arc::new(RwLock::new(HashMap::new())),
http_client: Client::builder()
.timeout(std::time::Duration::from_secs(30))
.connect_timeout(std::time::Duration::from_secs(10))
.pool_max_idle_per_host(10)
.pool_idle_timeout(std::time::Duration::from_secs(90))
.user_agent(concat!(
"Tranquil-PDS/",
env!("CARGO_PKG_VERSION"),
" (ATProto; +https://tangled.org/tranquil.farm/tranquil-pds)"
))
.build()
.unwrap_or_else(|_| Client::new()),
http_client: {
let builder = Client::builder()
.timeout(std::time::Duration::from_secs(30))
.connect_timeout(std::time::Duration::from_secs(10))
.pool_max_idle_per_host(10)
.pool_idle_timeout(std::time::Duration::from_secs(90))
.user_agent(concat!(
"Tranquil-PDS/",
env!("CARGO_PKG_VERSION"),
" (ATProto; +https://tangled.org/tranquil.farm/tranquil-pds)"
));
#[cfg(feature = "native-tls-roots")]
let builder = builder.danger_accept_invalid_certs(true);
builder.build().unwrap_or_else(|_| Client::new())
},
cache_ttl_secs,
}
}
+3
View File
@@ -55,6 +55,7 @@ metrics-exporter-prometheus = { workspace = true }
multibase = { workspace = true }
multihash = { workspace = true }
p256 = { workspace = true }
parking_lot = { workspace = true }
rand = { workspace = true }
redis = { workspace = true, optional = true }
regex = { workspace = true }
@@ -90,8 +91,10 @@ s3-storage = ["tranquil-storage/s3", "dep:aws-config", "dep:aws-sdk-s3"]
s3 = ["s3-storage"]
valkey = ["tranquil-cache/valkey", "dep:redis"]
frontend = []
native-tls-roots = ["tranquil-oauth/native-tls-roots"]
[dev-dependencies]
tempfile = "3"
ciborium = { workspace = true }
ctor = { workspace = true }
testcontainers = { workspace = true }
+72 -6
View File
@@ -15,12 +15,15 @@ struct ErrorBody<'a> {
#[derive(Debug)]
pub enum ApiError {
InternalError(Option<String>),
RepoCorruption,
AuthenticationRequired,
AuthenticationFailed(Option<String>),
InvalidRequest(String),
InvalidToken(Option<String>),
ExpiredToken(Option<String>),
OAuthExpiredToken(Option<String>),
UseDpopNonce(String),
InvalidDpopProof(String),
TokenRequired,
AccountDeactivated,
AccountTakedown,
@@ -119,10 +122,34 @@ pub enum ApiError {
},
}
const MST_NODE_MISSING_MARKER: &str = "MST node not found";
impl ApiError {
pub fn is_repo_corruption(&self) -> bool {
matches!(self, Self::RepoCorruption)
}
pub fn detail_is_repo_corruption(detail: &str) -> bool {
detail.contains(tranquil_store::blockstore::BLOCK_CORRUPTION_MARKER)
|| detail.contains(MST_NODE_MISSING_MARKER)
}
pub fn from_mst_error(context: &str, e: &jacquard_repo::error::RepoError) -> Self {
let detail = format!("{e:#}");
if Self::detail_is_repo_corruption(&detail) {
tracing::warn!("{context}: repairable MST damage: {detail}");
Self::RepoCorruption
} else {
tracing::error!("{context}: {detail}");
Self::InternalError(None)
}
}
fn status_code(&self) -> StatusCode {
match self {
Self::InternalError(_) | Self::DatabaseError => StatusCode::INTERNAL_SERVER_ERROR,
Self::InternalError(_) | Self::RepoCorruption | Self::DatabaseError => {
StatusCode::INTERNAL_SERVER_ERROR
}
Self::UpstreamFailure | Self::UpstreamUnavailable(_) | Self::UpstreamErrorMsg(_) => {
StatusCode::BAD_GATEWAY
}
@@ -137,6 +164,8 @@ impl ApiError {
| Self::InvalidToken(_)
| Self::PasskeyCounterAnomaly
| Self::OAuthExpiredToken(_)
| Self::UseDpopNonce(_)
| Self::InvalidDpopProof(_)
| Self::ReauthRequired { .. } => StatusCode::UNAUTHORIZED,
Self::InvalidCode(_) => StatusCode::BAD_REQUEST,
Self::ExpiredToken(_) => StatusCode::BAD_REQUEST,
@@ -219,7 +248,9 @@ impl ApiError {
}
fn error_name(&self) -> Cow<'static, str> {
match self {
Self::InternalError(_) | Self::DatabaseError => Cow::Borrowed("InternalServerError"),
Self::InternalError(_) | Self::RepoCorruption | Self::DatabaseError => {
Cow::Borrowed("InternalServerError")
}
Self::UpstreamFailure | Self::UpstreamUnavailable(_) | Self::UpstreamErrorMsg(_) => {
Cow::Borrowed("UpstreamError")
}
@@ -236,6 +267,8 @@ impl ApiError {
Self::AuthenticationFailed(_) => Cow::Borrowed("AuthenticationFailed"),
Self::InvalidToken(_) => Cow::Borrowed("InvalidToken"),
Self::ExpiredToken(_) | Self::OAuthExpiredToken(_) => Cow::Borrowed("ExpiredToken"),
Self::UseDpopNonce(_) => Cow::Borrowed("use_dpop_nonce"),
Self::InvalidDpopProof(_) => Cow::Borrowed("invalid_dpop_proof"),
Self::TokenRequired => Cow::Borrowed("TokenRequired"),
Self::AccountDeactivated => Cow::Borrowed("AccountDeactivated"),
Self::AccountTakedown => Cow::Borrowed("AccountTakedown"),
@@ -326,6 +359,7 @@ impl ApiError {
Self::InternalError(msg) => msg
.clone()
.unwrap_or_else(|| "Internal Server Error".into()),
Self::RepoCorruption => "Internal Server Error".into(),
Self::AuthenticationFailed(msg) => msg
.clone()
.unwrap_or_else(|| "Authentication failed".into()),
@@ -335,6 +369,8 @@ impl ApiError {
Self::ExpiredToken(msg) | Self::OAuthExpiredToken(msg) => {
msg.clone().unwrap_or_else(|| "Token has expired".into())
}
Self::UseDpopNonce(_) => "DPoP nonce required".into(),
Self::InvalidDpopProof(msg) => msg.clone(),
Self::RepoNotFound(msg) => msg
.clone()
.unwrap_or_else(|| "Repository not found".into()),
@@ -560,6 +596,36 @@ impl IntoResponse for ApiError {
),
);
}
Self::UseDpopNonce(nonce) => {
match HeaderValue::from_str(nonce) {
Ok(val) => {
response
.headers_mut()
.insert(crate::util::HEADER_DPOP_NONCE, val);
}
Err(err) => {
tracing::error!(
?err,
nonce_len = nonce.len(),
"generated DPoP nonce is not a valid header value"
);
}
}
response.headers_mut().insert(
http::header::WWW_AUTHENTICATE,
HeaderValue::from_static(
"DPoP error=\"use_dpop_nonce\", error_description=\"Resource server requires nonce in DPoP proof\"",
),
);
}
Self::InvalidDpopProof(_) => {
response.headers_mut().insert(
http::header::WWW_AUTHENTICATE,
HeaderValue::from_static(
"DPoP error=\"invalid_dpop_proof\", error_description=\"Invalid DPoP proof\"",
),
);
}
_ => {}
}
response
@@ -596,6 +662,8 @@ impl From<crate::auth::TokenValidationError> for ApiError {
crate::auth::TokenValidationError::InvalidToken => {
Self::AuthenticationFailed(Some("Invalid token format".to_string()))
}
crate::auth::TokenValidationError::UseDpopNonce(nonce) => Self::UseDpopNonce(nonce),
crate::auth::TokenValidationError::InvalidDpopProof(msg) => Self::InvalidDpopProof(msg),
}
}
}
@@ -625,10 +693,8 @@ impl From<crate::auth::extractor::AuthError> for ApiError {
crate::auth::extractor::AuthError::OAuthExpiredToken(msg) => {
Self::OAuthExpiredToken(Some(msg))
}
crate::auth::extractor::AuthError::UseDpopNonce(_)
| crate::auth::extractor::AuthError::InvalidDpopProof(_) => {
Self::AuthenticationFailed(None)
}
crate::auth::extractor::AuthError::UseDpopNonce(nonce) => Self::UseDpopNonce(nonce),
crate::auth::extractor::AuthError::InvalidDpopProof(msg) => Self::InvalidDpopProof(msg),
}
}
}

Some files were not shown because too many files have changed in this diff Show More