mirror of
https://github.com/versity/versitygw.git
synced 2026-10-22 06:12:39 +00:00
Kyeongpyo Kim
368dae0d4e
The hipobj-rc-v2 server was written against RoCEv2, where the GRH
carries the whole route and the LID in the address handle is unused.
modifyQpToRtr therefore hard-coded ah_attr.dlid = 0, the PREPARE
handler dropped the LID the client token carries, and encodeReplyToken
never filled in our own LID. On an InfiniBand link layer the fabric
switches on the DLID, so every RC session fell over at the data phase:
the client's RDMA write failed with a transport retry exceeded or a
remote invalid request as soon as the QP pair left RTR.
Carry the LID end to end:
- DeviceHandle gains localLid, read from ibv_query_port at init.
- The PREPARE handler stashes the client LID from the token next to
the peer GID; the loopback fallback uses our own LID.
- READY passes that LID into the RTR transition, and modifyQpToRtr
sets ah_attr.dlid from it. is_global stays set: a GRH is valid on
IB and RoCE still needs it, and on RoCE the callers keep passing 0.
- encodeReplyToken fills token.lid so the client can address us.
The 44-byte token has always had the lid field; no wire change.
Verified on a native InfiniBand fabric (ConnectX-6, 200G HDR, SM lid 8)
with the gateway on a Rocky 8.10 storage node using inbox verbs and an
RC client that consumes the reply token LID: PUT and GET complete with
the DLID set, and fail exactly as described with the previous dlid = 0.