mirrors/tendermint
1
0
Fork 0
mirror of https://github.com/tendermint/tendermint.git synced 2026-08-27 19:37:08 +00:00
Code Issues Packages Projects Releases Wiki Activity

add prototype impl from @creachadair

Browse Source
This commit is contained in:
William Banfield
2021-12-10 14:31:12 -05:00
parent 7421b8380f
commit 3bb825fda1
1 changed files with 24 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/rfc/rfc-008-deterministic-proto-bytes.md
+24 -3
View File
@@ -35,10 +35,24 @@ used within the Tendermint consensus algorithm to prevent any issue from occurri
Proto has the following points of variability that can produce non-deterministic byte representation:
1. Encoding order of fields within a message.
Proto allows fields to be encoded in any order and even be repeated.
2. Encoding order of elements of a repeated field.
`repeated` fields in a proto message can be serialized in any order.
3. Presence or absence of default values.
4. Inclusion/exclusion of unknown fields.
5. Padded length of varint fields.
Types in proto have defined default values similar to Go's zero values.
Writing or omitting a default value are both legal ways of encoding a wire message.
4. Serialization of 'unknown' fields.
Unknown fields can be present when a message is created by a binary with a newer
version of the proto that contains fields that the deserializer in a different
binary does not yet know about. Deserializers in binaries that do not know about the field
will maintain the bytes of the unknown field but not place them into the deserialized structure.
We have a few options to consider when producing this stable representation.
@@ -63,11 +77,17 @@ This could be implemented as a function in many languages that performed the fol
1. Reordered all fields to be in tag-sorted order.
2. Reordered all `repeated` sub-fields to be in lexicographically sorted order.
3. Deleted all default values from the representation.
5. Trimmed all varints to be of minimum length.
4. Set our proto decoder package to remove unknown fields on deserialization.
This would still require that messages never unmarshal data structures with unknown fields.
This can be accomplished by defining adding fields to a structure that needs canonicalization
as a breaking changes within our semantic versioning and disallowing it within a major version.
I.e., two versions of Tendermint that have different fields in proto messages which require
canonical byte representation will not be able to interoperate.
A prototype implementation by @creachadair of this can be found in [the wirepb repo][wire-pb].
This could be implemented in multiple languages more simply than ensuring that there are
canonical proto serializers that match in each language.
Finally, we should add clear documentation to the Tendermint codebase every time we
compare hashes of proto messages or use proto serialized bytes to produces a
@@ -80,4 +100,5 @@ properly.
[spec-issue]: https://github.com/tendermint/tendermint/issues/5005
[cosmos-sdk-adr-27]: https://github.com/cosmos/cosmos-sdk/blob/master/docs/architecture/adr-027-deterministic-protobuf-serialization.md
[cer-proto-3]: https://github.com/regen-network/canonical-proto3
[wire-pb]: https://github.com/creachadair/wirepb