From 3bb825fda1e7420204dbfc220502f8c10d253563 Mon Sep 17 00:00:00 2001 From: William Banfield Date: Fri, 10 Dec 2021 14:31:12 -0500 Subject: [PATCH] add prototype impl from @creachadair --- docs/rfc/rfc-008-deterministic-proto-bytes.md | 27 ++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/docs/rfc/rfc-008-deterministic-proto-bytes.md b/docs/rfc/rfc-008-deterministic-proto-bytes.md index 1bb8813c7..1e39135f4 100644 --- a/docs/rfc/rfc-008-deterministic-proto-bytes.md +++ b/docs/rfc/rfc-008-deterministic-proto-bytes.md @@ -35,10 +35,24 @@ used within the Tendermint consensus algorithm to prevent any issue from occurri Proto has the following points of variability that can produce non-deterministic byte representation: 1. Encoding order of fields within a message. + +Proto allows fields to be encoded in any order and even be repeated. + 2. Encoding order of elements of a repeated field. + +`repeated` fields in a proto message can be serialized in any order. + 3. Presence or absence of default values. -4. Inclusion/exclusion of unknown fields. -5. Padded length of varint fields. + +Types in proto have defined default values similar to Go's zero values. +Writing or omitting a default value are both legal ways of encoding a wire message. + +4. Serialization of 'unknown' fields. + +Unknown fields can be present when a message is created by a binary with a newer +version of the proto that contains fields that the deserializer in a different +binary does not yet know about. Deserializers in binaries that do not know about the field +will maintain the bytes of the unknown field but not place them into the deserialized structure. We have a few options to consider when producing this stable representation. @@ -63,11 +77,17 @@ This could be implemented as a function in many languages that performed the fol 1. Reordered all fields to be in tag-sorted order. 2. Reordered all `repeated` sub-fields to be in lexicographically sorted order. 3. Deleted all default values from the representation. -5. Trimmed all varints to be of minimum length. +4. Set our proto decoder package to remove unknown fields on deserialization. This would still require that messages never unmarshal data structures with unknown fields. This can be accomplished by defining adding fields to a structure that needs canonicalization as a breaking changes within our semantic versioning and disallowing it within a major version. +I.e., two versions of Tendermint that have different fields in proto messages which require +canonical byte representation will not be able to interoperate. + +A prototype implementation by @creachadair of this can be found in [the wirepb repo][wire-pb]. +This could be implemented in multiple languages more simply than ensuring that there are +canonical proto serializers that match in each language. Finally, we should add clear documentation to the Tendermint codebase every time we compare hashes of proto messages or use proto serialized bytes to produces a @@ -80,4 +100,5 @@ properly. [spec-issue]: https://github.com/tendermint/tendermint/issues/5005 [cosmos-sdk-adr-27]: https://github.com/cosmos/cosmos-sdk/blob/master/docs/architecture/adr-027-deterministic-protobuf-serialization.md [cer-proto-3]: https://github.com/regen-network/canonical-proto3 +[wire-pb]: https://github.com/creachadair/wirepb