Introduces s3s and s3s-fs to run an in-memory s3 server in tests. With this we can promote s3 tests to run in `just test`, they won't need containers, and they're super fast. Adds `just test-s3` to run them more easily.
```
Nextest run ID db9a9530-738e-4c92-ad3a-23289252cac2 with nextest profile: default Starting 6 tests across 1 binary PASS [ 0.356s] tranquil-storage::s3 put_stream_error_aborts_upload PASS [ 0.356s] tranquil-storage::s3 put_stream_empty_aborts_upload PASS [ 0.358s] tranquil-storage::s3 copy PASS [ 0.360s] tranquil-storage::s3 put_get_head_delete PASS [ 0.360s] tranquil-storage::s3 put_stream PASS [ 0.447s] tranquil-storage::s3 put_stream_multipart ──────────── Summary [ 0.448s] 6 tests run: 6 passed, 0 skipped
```
Also optimizes more hashing functions because it matters in tests.
We can make this test only by adding a feature for it and checking, but it didn't seem immediately worth it to me. Ping me if you want me to change it.
Shaves about 15-20% off of a j2 test run, because we spent about 200ms on every one of the 500 tests against the server just initializing the governor clock.
We've got a bunch of tests that never pass because they time out on their own, or that take 10s because they test a 10s timeout, that we can rethink. Together they introduce a lot of slowness.
This puts anything over 10s in CI into test jail.
This removes the old workflows that had not been working for a while and introduces a new microvm based one that does some basic checks and runs the default test suite.
All of the codebase uses ring explicitly or implicitly, but the HTTP client used for S3 requests was using aws-lc instead. This commit removes that extra dependency.
Anecdotally it shaves 10s off a dev build on my machine, but slower machines would benefit more from it.
Many of these don't finish anyway on my computer, I gave up around 40 minutes. But for the ones that did, it's a big speed up! For the others I guess they need more love.
gauntlet-pr: 67.9s with it, 120.9s without
test-store: 508.8s with it, still running at 1313s (I gave up)
test-store-sim-nightly: ran for like 40 minutes each attempt without finishing. I just gave up.
test-store-asan: only linux? but no reason why we can't apply the fix here too
Overall `just test` runs about 10-20s faster on this branch than current main, for me, with around 170s instead of around 180-190s. But the cool thing is the compile time. Making a simple edit to any file involved and running `just test` on main takes about 60s to recompile the test binaries. On this branch it takes less than 15s.
This is done by replacing a ton of separate test binaries with a single megabinary.
Only test files that looked safe have been included, avoiding ones that do stuff like setting env.
Summary [ 166.110s] 1453 tests run: 1453 passed (1 slow), 15 skipped
Above is before this commit, below is after.
Summary [ 140.410s] 1453 tests run: 1453 passed, 15 skipped
The mechanism here is the same as for the metastore tests. By skipping fsync we can run the tests much faster, and fsync doesn't matter in tests. Unless you're testing postgres itself I guess
Replace sleeps with the existing polling function instead. Exposes that the stress test isn't much of a stress test, it probably needs more data or something, but if it's heavy it should maybe not be part of `just test`
`just test` doesn't use docker, but the cleanup was still pruning containers over and over. This switches to only cleaning up containers created by the tests.
test-unit was building a second copy of the code with different features, and then the main test command ran the same tests anyway, so we just built and ran the tests twice
Locally shaves a minute off of a full build+test, on this branch (so both runs include the fsync change etc)
We're at [ 173.444s] overall for `just test` now
Build optimized versions of bcrypt and blowfish to speed them up, referencing things like https://github.com/oxidecomputer/omicron/blob/main/Cargo.toml#L981-L982
Password hashing is not a dominating cost, so this doesn't have a significant impact on the overall time, but a lot of tests like `**tranquil-pds::legacy_2fa** **test_legacy_2fa_code_consumed_after_use**` speed up. It goes from 5.6s to 2.7s on my machine.
Flushing to disk all the time ensures no partial states when writing to disk during awkwardnesses like crashes in prod risking real user data. That's good!
We're also doing it while running tests. Less good! Well, at least not important!
Comparing a clean test run (`just test --no-fail-fast`) and a run on this branch, on my MacOS M4 Pro, I get:
`**Summary** [ 517.723s] **1424** tests run: **1405** **passed** (**7** **slow**), **19** **failed**, **15** **skipped**`
before and
`**Summary** [ 326.369s] **1424** tests run: **1405** **passed**, **19** **failed**, **15** **skipped**`
after.
The reason I run `--no-fail-fast` is that a bunch of tests always fail and time out and stuff
`rpc:*?aud=did:web:api.bsky.app#bsky_appview` fails in the permissions check even though it's a valid scope. This makes it resolve correctly.
Also this commit fixes a bug where `rpc` as a bare scope is interpreted as all methods for all services, but it's really an invalid scope. Can drop it if it's a problem, but it just looks like a bug in the implementation rather than something intended?
Note that the parser treats `rpc:*` as "allow everything for no services", but aud is actually required, so it's not a valid scope? Didn't mess with it though because delegation uses it and that part of the code treats it as allow everything for every service. We should maybe split the scope parsers so we have one for delegated scopes and one for regular?
However, to avoid the same bug as bare `rpc`, I changed the enforcement step in "regular" OAuth to disallow `rpc:*` (this matches the existing behavior on main, no actual change, just a little acknowledgement).
Updates the default runner debian image sha to the multi-arch image instead of amd64-only. Also symlinks .containerignore -> .dockerignore since docker refuses to read .containerignore.
With that one can build and push multi-arch, like so:
```
docker buildx create --name tranquil-multiarch --driver docker-container --bootstrap
docker buildx build \
--builder tranquil-multiarch \
--platform linux/amd64,linux/arm64 \
-t atcr.io/tranquil.farm/tranquil-pds:0.6.7 \
--push \
.
docker buildx imagetools create \
-t atcr.io/tranquil.farm/tranquil-pds:latest \
atcr.io/tranquil.farm/tranquil-pds:0.6.7
```
if one were so inclined, that is.
This improves mobile behavior for the inputs without affecting other things.
Two inputs set to email, rest to URL pre recommendation in https://tangled.org/tranquil.farm/tranquil-pds/issues/140
Also disables autocapitalize, autocorrect, and spellcheck, while we're at it. All really annoying when typing in handles on a mobile phone.
My last PR introduced a migration to blobs, changing them from being unique per cid to being unique per cid+user. This is because two users can upload the same blob, get the same cid, and then previously in tranquil the second user would never actually get their ownership recorded, meaning listBlobs was missing the blob, and a migration _off_ of tranquil would leave them sans blob https://tangled.org/tranquil.farm/tranquil-pds/pulls/261/round/3
This PR attempts to add a job to repair existing instances where blobs have been deduplicated and not attributed to each owning user. It does this by pulling all records for each user, and then in batches crawling the records to find blob references. For each blob reference, it checks whether that user has its ownership recorded, and if not, repairs it by adding the entry.
To avoid re-running this potentially heavy job on every startup, I've used the config table to store a little record of whether it has been run.
This also means we can clean up the last blob operations that were per cid rather than user+cid 🪓
We should be setting ourselves up to be able to go garbage collect blobs safely so deleting the logo blobs manually won't matter anyway.
Johanna LarssonandTangled
LewisandTangled
juneandTangled
Louis EscherandTangled
nelind
Anastasia CalicoandTangled
TrezyandTangled
LunaandTangled
xan.lolandTangled