Better discipline in constructing verificationtarget
for verifying comms channels. It's still not perfect and
there's still email-defaultism but I want this bandaid in
before we do more work later on to make email not special.
Better hygiene here also for secrecy vs not secret notifs.
More type upgrades to come after this!
Lewis: May this revision serve well! <did:plc:3fwecdnvtcscjnrx2p4n7alz>
This endpoint was lacking rate limits, even though it can be a very costly operation. I set it to a 5 per 60s per ip rate limit, which should support normal behaviors and mitigate any misbehaving relays.
The spec allows for a maximum of 2MB of total record block bytes in a commit event. In indigo-relay, a commit between 2MB and 2MiB results in the commit being silently dropped. Over 2MiB the commit is rejected, the connection closed and then immediately re-opened and the same commit read again. This leads to the relays essentially DDoSing the server by reading the same commit over and over again.
Once indigo-relay implements backoff, the effect will instead be that the relay can't progress and your account goes dark.
Left an explanation in the code as well but basically although the spec doesn't set a hard limit on the size of a CBOR record, there is a hard limit on individual commit event record blocks in the sync 1.1 spec. indigo-relay doesn't seem to validate the size of your commits, but hits a hard limit in the websocket library if the commit is too large, causing the connection to drop and instantly reconnect.
I had a 10MiB record on my PDS and I had 9 relays hammering my server constantly trying to backfill that same event and choking on it over and over, until I cleaned it up.
With this PR tranquil introduces a 1MB limit on CBOR records, ensuring we never go over the sync commit event limits.
As part of a subscriber getting caught up, we look up the oldest repo_seq row since the cursor they provided, reading out lots of data from the row, then throwing it all away and just looking at `created_at`. As happened to my poor baby PDS, relays can be pretty aggressive with trying to backfill and will keep triggering these lookups with no backoff, so it's worth avoiding the extra work here and limiting the query to get the created_at field only. That's about a 600x reduction in the number of bytes this lookup represents based on my PDS's data, from ~5K to 8 bytes.
I have not done the same optimization for metastore, I don't think it's as affected. The ROI is lower and the effort is higher. Might be a fun thing for someone to add if they wanted to.
Typing a struct for enqueuing messages out to comms.
Also, after recent PRs showing me the way in which the
previously unused metadata field would possibly be used going
forward, I think I deem it too much of a hazard to be
left alive, hence removing it and migrating any inflight
comms to extract html_body (only existing use for it).
Lewis: May this revision serve well! <did:plc:3fwecdnvtcscjnrx2p4n7alz>
dalek derive 0.1.1 is just a duplicate now! So nix wasn't happy
with it, considering name-version syml went
into a RO dir.
Lewis: May this revision serve well! <did:plc:3fwecdnvtcscjnrx2p4n7alz>
- use tranquil_pds not comms since the validate email and templates are reexported from pds package
- implement handlebars syntax
- update templates to be transport agnostic, email, discord, telegram with handlebar examples with conditions
- refactor template and notice logic from PR #269, and #299 to be self contained enforcing all substitutions to be defined within the enum properties
- fix open TODOs, store.rs uses get_string within the ids.rs get_built_ins method
- UTC logging for User facing notices restored
- new api error for verification channel requires bot
- fix tests to use metadata for html body...
- remove unwrap and handle template rendering error explicitly
- update verify_link to use handlebars trusted url syntax {{{verify_link}}}
- fix verification enqueueing to include channel in template rendering
- use locale enum for validation and display instead of string array separately maintained
- support text only templates for signal and discord
- fix missing verify_link and verify_page in migration and signup verification templates with unit tests
- consolidated body and subject notice methods into a single rendered function
Instead of moving :latest, this adds :main for the intrepid adventurers who wish to live dangerously.
And then we can have :latest track the latest proper release instead.
We were just using them for S3 support. This PR switches us over to the s3 crate which works the same way, with some minor differences.
The main implication of this change is that we're no longer doing all the AWS stuff, like checking aws profile in home directory, or automatic discovery of metadata.
This change is covered by our brand new s3 test suite, but I also did the old school `just test-full` with minio.
Introduces s3s and s3s-fs to run an in-memory s3 server in tests. With this we can promote s3 tests to run in `just test`, they won't need containers, and they're super fast. Adds `just test-s3` to run them more easily.
```
Nextest run ID db9a9530-738e-4c92-ad3a-23289252cac2 with nextest profile: default Starting 6 tests across 1 binary PASS [ 0.356s] tranquil-storage::s3 put_stream_error_aborts_upload PASS [ 0.356s] tranquil-storage::s3 put_stream_empty_aborts_upload PASS [ 0.358s] tranquil-storage::s3 copy PASS [ 0.360s] tranquil-storage::s3 put_get_head_delete PASS [ 0.360s] tranquil-storage::s3 put_stream PASS [ 0.447s] tranquil-storage::s3 put_stream_multipart ──────────── Summary [ 0.448s] 6 tests run: 6 passed, 0 skipped
```
Also optimizes more hashing functions because it matters in tests.
We can make this test only by adding a feature for it and checking, but it didn't seem immediately worth it to me. Ping me if you want me to change it.
Shaves about 15-20% off of a j2 test run, because we spent about 200ms on every one of the 500 tests against the server just initializing the governor clock.
We've got a bunch of tests that never pass because they time out on their own, or that take 10s because they test a 10s timeout, that we can rethink. Together they introduce a lot of slowness.
This puts anything over 10s in CI into test jail.
LewisandTangled
Johanna LarssonandTangled
SteffoandTangled
straiforosandTangled
Natalie BridgersandTangled
Hayden YoungandTangled
teqandTangled
nekomimi.petandTangled
juneandTangled