The current tranquil database design only allows each blob to be owned by one account. This means that if a second account also has that blob, tranquil skips associated the blob with the account. That works fine a lot of the time, since blobs are looked up by cid and the blob exists. However, it can lead to loss of data under certain scenarios.
One example is where I upload a blob, the blob already exists in my instance so insertion is skipped (postgres requires cid to be unique in blobs, fjall only allows one owner per blob cid), I then decide to migrate off tranquil, the blob does not come with me since it is not mine.
Another example is where an account is deleted. If a blob was uploaded for account a, then account b uploads the same blob tranquil skips storing it since it exists. Then I delete account a, now account b's blob is missing.
I accidentally stumbled upon this when I migrated my account to my own tranuil instance and list blobs now lists 2 fewer blobs than before, two images that had been uploaded by accounts already on the PDS.
ps I found record_blobs a bit confusing, at first it looked like a blob ownership table, but then it turns out to just be used for migrations!
This PR makes the blob primary key be cid+user for postgres, and updates the queries to account for there being multiple "blobs" with the same cid. For queries that just care about the blob existing, it doesn't matter "whose" blob it is, so limit 1.
Most of the work is on the metastore side. Adds ref_count to track how many are referencing the blob since we can't just check for other rows. Instead of storing blobs directly, we now store a per account cid, and the blob reference itself is shared and keyed by cid only. This means some of these operations now require updating two places, so they're done in `batch`es.
With the new layout get_blob_value becomes simpler, all blob data is a single "table" or whatever it's called, so we just grab it using cid instead of looking it up for the user.
Migrates blobs rather than maintaining two different versions of the tables, although it seems like that could be supported.
I removed a test that asserted the old behavior, and added a reasonable (?) set of new tests that assert the new behavior, including a parity test.
Johanna LarssonandTangled
LewisandTangled
Lewis