mirrors/tendermint
1
0
Fork 0
mirror of https://github.com/tendermint/tendermint.git synced 2026-08-28 11:56:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

clarify reasons for canonicalization rules

Browse Source
This commit is contained in:
William Banfield
2021-12-15 17:45:00 -05:00
parent c932e7926b
commit 1e6e9c9e59
1 changed files with 42 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/rfc/rfc-007-deterministic-proto-bytes.md
+42 -15
View File
@@ -56,7 +56,9 @@ will maintain the bytes of the unknown field but not place them into the deseria
We have a few options to consider when producing this stable representation.
### Use only compliant serializers and constrain field usage
### Options for deterministic byte representation
#### Use only compliant serializers and constrain field usage
According to [Cosmos-SDK ADR-27][cosmos-sdk-adr-27], when message types obey a simple
set of rules, gogoproto produces a consistent byte representation of serialized messages.
@@ -65,40 +67,65 @@ produces a consistent set of bytes on serialized messages. This would solve the
within Tendermint as written in Go, but would require ensuring that there are similar
serializers written in other languages that produce the same output as gogoproto.
### Reorder serialized bytes to ensure determinism.
#### Reorder serialized bytes to ensure determinism.
The serialized form of a proto message can be transformed into a canonical representation
by applying simple rules to the serialized bytes. Re-ordering the serialized bytes
would allow Tendermint to produce a canonical byte representation without having to
simultaneously maintain a custom proto marshaller.
This could be implemented as a function in many languages that performed the following when hashing:
This could be implemented as a function in many languages that performed the following
producing bytes to sign or hashing:
1. Reordered all fields to be in tag-sorted order.
2. Does not add any of the data from unknown fields into the type to hash.
3. Reordered all non-scalar `repeated` sub-fields to be in lexicographically sorted order.
4. Deleted all default values from the byte representation.
1. Does not add any of the data from unknown fields into the type to hash.
Tendermint should not run into a case where it needs to verify the integrity of
data with unknown fields for the following reasons:
When a process is checking hash equality, it knows the concrete data that it is trying to
compare to the hash. The purpose of checking hash equality within Tendermint is to ensure that
the data that process knows about matches the data that the network agreed on. There should
The purpose of checking hash equality within Tendermint is to ensure that
its local copy of data matches the data that the network agreed on. There should
therefore not be a case where a process is checking hash equality using data that it did not expect
to receive. What the data represent may be opaque to the process, such as when checking the
hashes of transactions in a block, _but the validator still expected to receive this data_,
despite not understanding what their internal structure is.
transactions in a block, _but the process will still have expected to receive this data_,
despite not understanding what their internal structure is. It's not clear what it would
mean to verify that a block contains data that a process does not know about.
The same reasoning applies for signature verification within Tendermint. Processes
verify that a digital signature signed over a set of bytes by locally reconstructing the
data structure that the digital signature signed.
data structure that the digital signature signed using the process's local data.
A prototype implementation by @creachadair of this can be found in [the wirepb repo][wire-pb].
2. Reordered all message fields to be in tag-sorted order.
Tag-sorting top-level fields will place all fields of the same tag in a adjacent
to eachother within the serialized representation.
3. Reordered the contents of all `repeated` fields to be in lexicographically sorted order.
`repeated` fields will appear in a message as having the same tag but will contain different
contents. Therefore, lexicographical sorting will produce a stable ordering of
fields with the same tag.
4. Deleted all default values from the byte representation.
Encoders can include default values or omit them. Most encoders appear to omit them
but we may wish to delete them just to be safe.
5. Recursively performed these operations on any length-delimited subfields.
Length delimited fields may contain messages, strings, or just bytes. However,
it's not possible to know what data is being represented by such a field.
A 'string' may happen to have the same structure as an embedded message and we cannot
disambiguate. For this reason, we must apply these same rules to all subfields that
may contain messages. Because we cannot know if we have totally mangled the interior 'string'
or not, this data should never be deserialized or used for anything beyond hashing.
A **prototype** implementation by @creachadair of this can be found in [the wirepb repo][wire-pb].
This could be implemented in multiple languages more simply than ensuring that there are
canonical proto serializers that match in each language.
Finally, we should add clear documentation to the Tendermint codebase every time we
### Future work
We should add clear documentation to the Tendermint codebase every time we
compare hashes of proto messages or use proto serialized bytes to produces a
digital signatures that we have been careful to ensure that the hashes are performed
properly.