There's no case where we recieve an error during handshake and don't
just return/continue, and it's at a point during startup where not
much is going on in the process, so having some classes of errors
return errors and some return panics is confusing and doesn't protect
anything.