【verbs】ibv_modify_qp()|RDMA
目录
原文:ibv_modify_qp() - RDMAmojo RDMAmojo
描述
ibv_modify_qp()修改队列对的属性。
更改的属性描述了QP的发送和接收属性。在UC和RC QP中,这意味着将QP与远程QP连接。
在Infiniband中,其中一个应该执行到子网管理员(SA)的路径查询(one should perform path query to the Subnet Administrator (SA)),以便确定应该为QP配置哪些属性或作为最佳解决方案,请使用Communication Manager(CM)或通用RDMA CM代理(CMA)连接QP。但是,有些应用程序倾向于自己连接QP,并通过sockets交换数据来决定要使用哪些QP属性。
在RoCE中,其中一个应该为连接的QP配置在QP属性中的GRH,或者为UD QP配置在地址句柄(Address Handle中)的GRH。
在iWARP中,只能使用通用RDMA CM代理(CMA)连接QP。
struct ibv_qp_attr描述了队列对的属性。
struct ibv_qp_attr 描述QP的属性
struct ibv_qp_attr { enum ibv_qp_state qp_state; enum ibv_qp_state cur_qp_state; enum ibv_mtu path_mtu; enum ibv_mig_state path_mig_state; uint32_t qkey; uint32_t rq_psn; uint32_t sq_psn; uint32_t dest_qp_num; int qp_access_flags; struct ibv_qp_cap cap; struct ibv_ah_attr ah_attr; struct ibv_ah_attr alt_ah_attr; uint16_t pkey_index; uint16_t alt_pkey_index; uint8_t en_sqd_async_notify; uint8_t sq_draining; uint8_t max_rd_atomic; uint8_t max_dest_rd_atomic; uint8_t min_rnr_timer; uint8_t port_num; uint8_t timeout; uint8_t retry_cnt; uint8_t rnr_retry; uint8_t alt_port_num; uint8_t alt_timeout; };
struct ibv_qp_attr的完整说明
qp_state | The next QP state. It can be one of the following enumerated values: 下一个 QP 状态。它可以是以下枚举值之一:
|
cur_qp_state | Assume that this is the current QP state. This is useful if it is known to the application that the QP state is different from the assumed state by the low-level driver. It can be one of the enumerated values as qp_state 假设这是当前的QP状态。如果应用程序知道QP状态与底层驱动程序假定的状态不同,这将很有用。它可以是qp_state枚举值之一 |
path_mtu | The path MTU (Maximum Transfer Unit) i.e. the maximum payload size of a packet that can be transferred in the path. It can be one of the following enumerated values:
For UC and RC QPs, when needed, the RDMA device will automatically fragment the messages to packet of this size. |
path_mig_state | The state of the QP's path migration state machine if supported by the device (IBV_DEVICE_AUTO_PATH_MIG is set in dev_cap.device_cap_flags). It can be one of the following enumerated values:
|
qkey | The Q_Key that incoming messages are check against and possibly used as the outgoing Q_Key (if the MSB of the q_key in the Send Request is set). Relevant only for UD QPs |
rq_psn | A 24 bits value of the Packet Sequence Number of the received packets for RC and UC QPs |
sq_psn | A 24 bits value of the Packet Sequence Number of the sent packets for any QP |
dest_qp_num | A 24 bits value of the remote QP number of RC and UC QPs; when sending data, packets will be sent to this QP number and when receiving data, packets will be accepted only from this QP number |
qp_access_flags | Allowed access flags of the remote operations for incoming packets of RC and UC QPs. It is either 0 or the bitwise OR of one or more of the following flags:
|
cap | New attributes to the number of Wore Requests in the Queue Pair, as described in the table below, if resize QP is supported by the device (IBV_DEVICE_RESIZE_MAX_WR is set in dev_cap.device_cap_flags). Upon a successful Queue Pair resize, this structure will hold the actual Queue Pair attributes |
ah_attr | Address vector of the primary path which describes the path information to the remote QP as described in the table below |
alt_ah_attr | Address vector of the alternate path which describes the path information to the remote QP as described in the table below. Can be used only if supported by the device (IBV_DEVICE_AUTO_PATH_MIG is set in dev_cap.device_cap_flags) |
pkey_index | Primary P_Key index. The value of the entry in the P_Key table that outgoing packets from this QP will be sent with and incoming packets to this QP will be verified within the Primary path |
alt_pkey_index | Alternate P_Key index. The value of the entry in the P_Key table that outgoing packets from this QP will be sent with and incoming packets to this QP will be verified within the Alternate path |
en_sqd_async_notify | If non-zero, generate the affiliated asynchronous event IBV_EVENT_SQ_DRAINED when the QP state becomes SQD.drained, i.e. the Send Queue is drained |
sq_draining | If set, indication that Send Queue draining is in progress. Not relevant for ibv_modify_qp() |
max_rd_atomic | The number of RDMA Reads & atomic operations outstanding at any time that can be handled by this QP as an initiator. Relevant only for RC QPs |
max_dest_rd_atomic | The number of RDMA Reads & atomic operations outstanding at any time that can be handled by this QP as a destination. Relevant only for RC QPs |
min_rnr_timer | Minimum RNR NAK Timer Field Value. When an incoming message to this QP should consume a Work Request from the Receive Queue, but not Work Request is outstanding on that Queue, the QP will send an RNR NAK packet to the initiator. It does not affect RNR NAKs sent for other reasons. The value can be one of the following numeric values since those values aren’t enumerated:
Relevant only for RC QPs |
port_num | Primary physical port number associated with this QP |
timeout | The minimum timeout that a QP waits for ACK/NACK from remote QP before retransmitting the packet. The value zero is special value which means wait an infinite time for the ACK/NACK (useful for debugging). For any other value of timeout, the time calculation is: ![]()
Relevant only to RC QPs |
retry_cnt | A 3 bits value of the total number of times that the QP will try to resend the packets before reporting an error because the remote side doesn't answer in the primary path |
rnr_retry | A 3 bits value of the total number of times that the QP will try to resend the packets when an RNR NACK was sent by the remote QP before reporting an error. The value 7 is special and specify to retry infinite times in case of RNR |
alt_port_num | Alternate physical port number associated with this QP |
alt_timeout | The total number of times that the QP will try to resend the packets before reporting an error because the remote side doesn't answer in the alternate path |
关于QP属性中某些特定值的一些警告:
- 在开发过程中,将attr.retry_cnt和attr.rnr_retry设置为0是一个好习惯。如果代码中存在竞争,最好在开发阶段检测它们
- attr.timeout中的值0表示一直等待ACK或NACK。这意味着,如果消息中的任何数据包丢失并且没有发送ACK或NACK,则不会发生任何重试,并且QP只会停止发送数据
- attr.rnr_retry中的值7表示在远程端发送RNR Nack时,发送消息重试次数无上限
struct ibv_qp_cap 描述QP的size(用于发送和接收队列)。
struct ibv_qp_cap { uint32_t max_send_wr; uint32_t max_recv_wr; uint32_t max_send_sge; uint32_t max_recv_sge; uint32_t max_inline_data; };
struct ibv_qp_cap: 说明
max_send_wr | The maximum number of outstanding Work Requests that can be posted to the Send Queue in that Queue Pair. Value can be [1..dev_cap.max_qp_wr] |
max_recv_wr | The maximum number of outstanding Work Requests that can be posted to the Receive Queue in that Queue Pair. Value can be [1..dev_cap.max_qp_wr]. This value is ignored if the Queue Pair is associated with an SRQ |
max_send_sge | The maximum number of scatter/gather elements in any Work Request that can be posted to the Send Queue in that Queue Pair. Value can be [1..dev_cap.max_sge] |
max_recv_sge | The maximum number of scatter/gather elements in any Work Request that can be posted to the Receive Queue in that Queue Pair. Value can be [1..dev_cap.max_sge]. This value is ignored if the Queue Pair is associated with an SRQ |
max_inline_data | The maximum message size (in bytes) that can be posted inline to the Send Queue. 0, if no inline message is requested |
struct ibv_ah_attr 描述了QP的地址向量
|
struct ibv_ah_attr:说明
grh | Attributes of the Global Routing Headers (GRH), as described in the table below. This is useful when sending packets to another subnet 全局路由头(GRH)的属性,如下表所述。将数据包发送到另一个子网时,这很有用 |
dlid | If the destination is in same subnet, the LID of the port to which the subnet delivers the packets to. If the destination is in another subnet, the LID of the Router |
sl | 4 bits. The Service Level to be used |
src_path_bits | The used Source Path Bits. This is useful when LMC is used in the port, i.e. each port covers a range of LIDs. The packets are being sent with the port's base LID, bitwised ORed with the value of the source path bits. The value 0 indicates the port's base LID is used |
static_rate | A value which limits the rate of packets that being sent to the subnet. This can be useful if the rate of the packet origin is higher than the rate of the destination |
is_global | If this value contains any value other than zero, then GRH information exists in this AH, thus the field grh if valid |
port_num | The local physical port that the packets will be sent from |
struct ibv_global_route AH中描述将要发送的数据包的GRH中使用的值
|
struct ibv_global_route属性说明:
dgid | The GID that is used to identify the destination port of the packets |
flow_label | 20 bits. If this value is set to a non-zero value, it gives a hint for switches and routers with multiple outbound paths that these sequence of packets must be delivered in order, those staying on the same path, so that they won't be reordered. |
sgid_index | An index in the port's GID table that will be used to identify the originator of the packet |
hop_limit | The number of hops (i.e. the number of routers) that the packet is permitted to take before being discarded. This ensures that a packet will not loop indefinitely between routers if a routing loop occur. Each router decrement by one this value at the packet and when this value reaches 0, this packet is discarded. Setting the value to 0 or 1 will ensure that the packet won't leave the local subnet. |
traffic_class | Using this value, the originator of the packets specifies the required delivery priority for handling them by the routers 使用此值,数据包始发者指定它们的路由器传递优先级 |
attr_mask specifies the QP attributes to be modified. It is either 0 or the bitwise OR of one or more of the following flags:
attr_mask指定要修改QP的哪些属性。它是0或以下一个或多个标志的按位或:
IBV_QP_STATE | Use the value of attr->qp_state |
IBV_QP_CUR_STATE | Use the value of attr->cur_qp_state |
IBV_QP_EN_SQD_ASYNC_NOTIFY | Use the value of attr->en_sqd_async_notify |
IBV_QP_ACCESS_FLAGS | Use the value of attr->qp_access_flags |
IBV_QP_PKEY_INDEX | Use the value of attr->pkey_index |
IBV_QP_PORT | Use the value of attr->port_num |
IBV_QP_QKEY | Use the value of attr->qkey |
IBV_QP_AV | Use the value of attr->ah_attr |
IBV_QP_PATH_MTU | Use the value of attr->path_mtu |
IBV_QP_TIMEOUT | Use the value of attr->timeout |
IBV_QP_RETRY_CNT | Use the value of attr->retry_cnt |
IBV_QP_RNR_RETRY | Use the value of attr->rnr_retry |
IBV_QP_RQ_PSN | Use the value of attr->rq_psn |
IBV_QP_MAX_QP_RD_ATOMIC | Use the value of attr->max_rd_atomic |
IBV_QP_ALT_PATH | Use the value of attr->alt_ah_attr, attr->alt_pkey_index, attr->alt_port_num, attr->alt_timeout |
IBV_QP_MIN_RNR_TIMER | Use the value of attr->min_rnr_timer |
IBV_QP_SQ_PSN | Use the value of attr->sq_psn |
IBV_QP_MAX_DEST_RD_ATOMIC | Use the value of attr->max_dest_rd_atomic |
IBV_QP_PATH_MIG_STATE | Use the value of attr->path_mig_state |
IBV_QP_CAP | Use the value of attr->cap |
IBV_QP_DEST_QPN | Use the value of attr->dest_qp_num |
下表指定了服务类型为IBV_QPT_UD的QP的支持的状态转换及其接受的属性:
Transition | 必填属性(Required Attributes) | 可选属性(Optional Attributes) |
---|---|---|
*->RESET | IBV_QP_STATE | |
*->ERR | IBV_QP_STATE | |
RESET->INIT | IBV_QP_STATE, IBV_QP_PKEY_INDEX, IBV_QP_PORT, IBV_QP_QKEY | |
INIT->INIT | IBV_QP_PKEY_INDEX, IBV_QP_PORT, IBV_QP_QKEY | |
INIT->RTR | IBV_QP_STATE | IBV_QP_PKEY_INDEX, IBV_QP_QKEY |
RTR->RTS | IBV_QP_STATE, IBV_QP_SQ_PSN | IBV_QP_CUR_STATE, IBV_QP_QKEY |
RTS->RTS | IBV_QP_CUR_STATE, IBV_QP_QKEY | |
RTS->SQD | IBV_QP_STATE | IBV_QP_EN_SQD_ASYNC_NOTIFY |
SQD->RTS | IBV_QP_STATE | IBV_QP_CUR_STATE, IBV_QP_QKEY |
SQD->SQD | IBV_QP_PKEY_INDEX, IBV_QP_QKEY | |
SQE->RTS | IBV_QP_STATE | IBV_QP_CUR_STATE, IBV_QP_QKEY |
下表指定了服务类型为IBV_QPT_UC 的QP的支持的状态转换及其接受的属性:
Transition | 必填属性(Required Attributes) | 可选属性(Optional Attributes) |
---|---|---|
*->RESET | IBV_QP_STATE | |
*->ERR | IBV_QP_STATE | |
RESET->INIT | IBV_QP_STATE, IBV_QP_PKEY_INDEX, IBV_QP_PORT, IBV_QP_ACCESS_FLAGS | |
INIT->INIT | IBV_QP_PKEY_INDEX, IBV_QP_PORT, IBV_QP_ACCESS_FLAGS | |
INIT->RTR | IBV_QP_STATE, IBV_QP_AV, IBV_QP_PATH_MTU, IBV_QP_DEST_QPN, IBV_QP_RQ_PSN | IBV_QP_PKEY_INDEX, IBV_QP_ACCESS_FLAGS, IBV_QP_ALT_PATH |
RTR->RTS | IBV_QP_STATE, IBV_QP_SQ_PSN | IBV_QP_CUR_STATE, IBV_QP_ACCESS_FLAGS, IBV_QP_ALT_PATH, IBV_QP_PATH_MIG_STATE |
RTS->RTS | IBV_QP_CUR_STATE, IBV_QP_ACCESS_FLAGS, IBV_QP_ALT_PATH, IBV_QP_PATH_MIG_STATE | |
RTS->SQD | IBV_QP_STATE | IBV_QP_EN_SQD_ASYNC_NOTIFY |
SQD->RTS | IBV_QP_STATE | IBV_QP_CUR_STATE, IBV_QP_ACCESS_FLAGS, IBV_QP_ALT_PATH, IBV_QP_PATH_MIG_STATE |
SQD->SQD | IBV_QP_PKEY_INDEX, IBV_QP_ACCESS_FLAGS, IBV_QP_AV, IBV_QP_ALT_PATH, IBV_QP_PATH_MIG_STATE | |
SQE->RTS | IBV_QP_STATE | IBV_QP_CUR_STATE, IBV_QP_ACCESS_FLAGS |
下表指定了服务类型为IBV_QPT_RC 的QP的支持的状态转换及其接受的属性:
Transition | 必填属性(Required Attributes) | 可选属性(Optional Attributes) |
---|---|---|
*->RESET | IBV_QP_STATE | |
*->ERR | IBV_QP_STATE | |
RESET->INIT | IBV_QP_STATE, IBV_QP_PKEY_INDEX, IBV_QP_PORT, IBV_QP_ACCESS_FLAGS | |
INIT->INIT | IBV_QP_PKEY_INDEX, IBV_QP_PORT, IBV_QP_ACCESS_FLAGS | |
INIT->RTR | IBV_QP_STATE, IBV_QP_AV, IBV_QP_PATH_MTU, IBV_QP_DEST_QPN, IBV_QP_RQ_PSN, IBV_QP_MAX_DEST_RD_ATOMIC, IBV_QP_MIN_RNR_TIMER | IBV_QP_PKEY_INDEX, IBV_QP_ACCESS_FLAGS, IBV_QP_ALT_PATH |
RTR->RTS | IBV_QP_STATE, IBV_QP_SQ_PSN, IBV_QP_TIMEOUT, IBV_QP_RETRY_CNT, IBV_QP_RNR_RETRY, IBV_QP_MAX_QP_RD_ATOMIC | IBV_QP_CUR_STATE, IBV_QP_ACCESS_FLAGS, IBV_QP_MIN_RNR_TIMER, IBV_QP_ALT_PATH, IBV_QP_PATH_MIG_STATE |
RTS->RTS | IBV_QP_CUR_STATE, IBV_QP_ACCESS_FLAGS, IBV_QP_MIN_RNR_TIMER, IBV_QP_ALT_PATH, IBV_QP_PATH_MIG_STATE | |
RTS->SQD | IBV_QP_STATE | IBV_QP_EN_SQD_ASYNC_NOTIFY |
SQD->RTS | IBV_QP_STATE | IBV_QP_CUR_STATE, IBV_QP_ACCESS_FLAGS, IBV_QP_MIN_RNR_TIMER, IBV_QP_ALT_PATH, IBV_QP_PATH_MIG_STATE |
SQD->SQD | IBV_QP_PKEY_INDEX, IBV_QP_PORT, IBV_QP_ACCESS_FLAGS, IBV_QP_AV, IBV_QP_MAX_QP_RD_ATOMIC, IBV_QP_MIN_RNR_TIMER, IBV_QP_ALT_PATH, IBV_QP_TIMEOUT, IBV_QP_RETRY_CNT, IBV_QP_RNR_RETRY, IBV_QP_MAX_DEST_RD_ATOMIC, IBV_QP_PATH_MIG_STATE |
参数
Name | Direction | Description |
---|---|---|
qp | 入参 | ibv_create_qp() 返回的QP句柄 |
attr | 入参/出参 | Requested attributes to be modified in the QP. If the QP is being resized, it will hold the actual attributes of the QP 对QP进行修改的属性集。如果要调整QP的大小,它将保留QP的实际属性 |
attr_mask | 入参 | Mask of the QP attributes to be modified 要修改的QP属性的掩码,指明attr哪些属性项要修改。(bit=1 需要改,bit=0的 保持) |
返回值
Value | Description |
---|---|
0 | 成功 |
errno | 失败且不会对QP进行任何更改 |
EINVAL | 在attr或attr_mask中提供的值无效 errno 22 |
ENOMEM | 没有足够的资源来完成此操作 errno 12 |
示例
1) Modify a UD QP from the RESET state to the RTS state:
将UD QP从RESET状态修改为RTS状态:
|
2) Modify a UC QP from the RESET state to the RTS state:
将UC QP从RESET状态修改为RTS状态:
|
3) Modify an RC QP from the RESET state to the RTS state:
将RC QP从RESET状态修改为RTS状态:
|
常见问题
为什么需要调用ibv_modify_qp()?
您需要调用ibv_modify_qp()才能将QP修改为可以接收和发送数据的状态
我可以将QP移到RTR状态并保持在该状态吗?
是的你可以。如果QP只是接收者
我如何知道要配置QP的哪些属性?
In Infiniband, you should perform path query to the SA or use CM or CMA, in iWARP, you should use the CMA
在Infiniband中,应该执行到SA的路径查询,或者使用CM或CMA;在iWARP中,应该使用CMA。
ibv_modify_qp()失败了,该QP的属性现在是什么?
与调用ibv_modify_qp()之前完全相同,未对QP进行任何更改。尽管该失败可能是由于某个错误导致RDMA设备将QP移至SQE或ERROR状态引起的
ibv_modify_qp()在通过RoCE已连接QP 进行INIT-> RTR转换时失败,这是什么原因?
使用RoCE时,必须配置GRH。
GRH在连接的QP中:作为QP属性的一部分
GRH在UD QP中:作为“地址句柄”属性的一部分
-
Igor R. says:February 11, 2014
如果您能阐明以下主题,我将不胜感激。
根据“ Infinibeand网络体系结构”(*参见下面的引用),在RDMA读取request期间,允许重试时从内存中重新读取响应者的RQ logic 。是否可以通过设置retry_cnt = rnr_cnt = 0来完全禁用此功能?
换句话说,可以确保在这种情况下,单个RDMA 读取请求 永远不会导致对任何远程位置的双重读取吗? (如果这是设备内存,则这可能至关重要,并且访问可能会产生副作用。)(I'd appreciate if you could shed some light on the following subject.
As per "Infinibeand Network Architecture"(* see quotations below), during RDMA Read request, the responder's RQ logic is allowed to re-read from memory when re-trying. Can we totally disable this functionality by setting retry_cnt=rnr_cnt=0?
In other words, can one be sure in such a case, that a single RDMA Read request will never cause double read of any remote location? (This can be crucial if it's a device memory, and an access might have side effects.Thanks!)
* Infiniband Network Architecture, p.34:(Infiniband网络体系结构,第34页:)
<<
This service is referred to as "reliable" for the following reasons:
-- The destination QP's RQ Logic verifies the PSN in each request packet to ensure that all of the message's request packets are received in order, that none are missing, and, if any duplicate request packets are received, that they are only processed once. There is one exception; upon receipt of a duplicate memory read, the data is read from memory again.>>
And in details (p.394):(还有详细说明(第394页):)
<<
Effects of Retry on Requester and Responder The following list defines the sequence of events:6. RQ Logic receives duplicate copy of request. When the second copy (the retried one) of the request packet eventually arrives at the RQ Logic, it has the same PSN as the original request packet. Its PSN, therefore, falls into the Ack’ d PSN region (also referred to as the Duplicate region), thereby identifying it as a duplicate request packet. The actions taken by the RQ Logic are determined by the request type:
— If it’ s a duplicate RDMA Read:
– The RQ Logic re-executes the read from its local memory.
– It sends back the requested read data in a series of one or more RDMA Read response packets.
– The first response packet uses the PSN of the original request packet, and each subsequent response packet increments the PSN by one.
– The RQ Logic does not change its ePSN (even if an error is detected while generating the response to the duplicate request).
– After issuing the response to the duplicate request, the RQ Logic resumes waiting for request packet with the ePSN
.Reply Igor R. says:February 12, 2014
好吧,这里的重试计数器似乎是“红鲱鱼”,因为真正的问题是远程HCA无法缓冲读取的数据。因此,在发生故障的情况下,即使不会发生自动重传,也将需要手动进行一次-以便无论如何将数据读取两次。因此,仅禁用自动重试是不够的,并且需要更复杂的方法。
Well, it seems that the retry counter is a "red herring" here, as the real problem is that remote HCA doesn't buffer the read data. So, in case of failure, even if the automatic retransmission won't occur, a manual one will be required - so that the data will be read twice anyway. Thus, solely disabling automatic retries is not sufficient, and more complicated approach is needed.
Reply Dotan Barak says:February 12, 2014
我必须承认,我不太了解您要执行的操作...
是的,如果retry_cnt为0,则不会发生重传,
QP将在请求方进入错误状态。
这是您所需的预期行为吗?
如果您必须从一侧读取数据而无需从设备读取任何数据,
也许RDMA读取不是正确的方法
(也许将数据缓存在本地内存中并执行RDMA写操作会更好。)
谢谢
多坦I must admit that I don't really understand what you are trying to perform...
Yes, if the retry_cnt will be 0 retransmission won't happen,
and the QP will get into the error state in the requester side.Is this is the expected behavior that you need?
If you must read the data from one side without any read from the device,
maybe RDMA Read isn't the right approach
(maybe cache the data in local memory and perform RDMA Write is better..)Thanks
Dotan
-
-
Igor R. says:February 12, 2014
-
感谢您回复。
在我的情况下,响应方(设备所在的位置)必须尽可能保持被动状态。我只是想考虑各种可能性并了解其后果...Thanks for the input.
In my case the responding side (where the device is) must remain as passive as possible. I'm just trying to consider various possibilities and to understand their consequences...
-
-
Alan says:August 29, 2014
Hi Dotan,
Do we have to memset the ibv_qp_attr structure before using it in the ibv_modify_qp() function? I am thinking there are flags in the function API to indicate which fields are being used why we still need to memset it?
Thanks.
-
Dotan Barak says:August 29, 2014
-
Hi Alan.
In general - you are correct (and in most cases it will work without any problem).
However:
* Most ibv_* structu don't have bitmask to specify the valid attributes
* The struct ibv_qp_attr *may* be extended (although the developers try hard to keep backward binary compatibility), and in that case calling memset() on the struct will keep the behavior as it was in the pastI believe, as a developer, in defensive programming and this is a good practice to handle API which is evolving.
I hope that I answered your question.
Thanks
Dotan -
Yanfang Le says:October 12, 2015
-
Hi Dotan,
I am trying to flush the work requests in qp, so I first turn the qp state to error state, and then convert to the RTS state. During this process, I don't know where can I get the ah_attr information. Could you tell me how can I get the ah_attr information? Thanks.
I am trying to copy the information from the old qp (before turn the state to error), however, it doesn't work.
-
Dotan Barak says:October 17, 2015
-
Hi.
When the QP is in ERROR state,
you cannot trust the attributes which were queried from it, expect for the QP state.If you must keep the AH attributes, maybe you can:
1) Save a local copy of the attributes as a function of the QP number
2) Use the qp_context pointer to hold a buffer which holds the AH attributes.Thanks
Dotan -
Haomai says:January 21, 2015
-
-
-
Hi, if I make a connected qp to error state, is it possible that peer side can detect this.
For example, if we use tcp socket to communicate, if we close one side's socket, other side will get "0" when read. So peer side can know peer socket already closed. Is it possible that rdma can achieve this?
Thanks!
-
Dotan Barak says:January 23, 2015
-
Hi Haomai.
In RDMA there aren't any keep-alive messages that send automatically,
if needed you need to perform this in your application.If you close one side, only if you are using rdmacm, the remote side will be informed on this;
but not in "standard" data verbs.If the other side (which didn't move to Error) won't send any message - it won't know that its remote
QP moved to Error.Thanks
DotanReplyAdrian says:March 23, 2015
-
-
Hi Dotan,
I've just started with verbs and when I try to modify a qp from INIT to RTR state, a 101 error is returned. The 101 error means #define ENETUNREACH 101 /* Network is unreachable */. The values are exchanged via TCP sockets (lid=4 qp_num=116 psn=9105394 from client to server and lid=3 qp_num=116 psn=14618275 from server to client). I don't know what is happening and there is difficult to obtain information.
Thanks
ReplyDotan Barak says:March 24, 2015
-
Hi Adrian.
Are you using explicit ibv_modify_qp(() or do you call librdmacm function?
Are you using IB or RoCE?Thanks
DotanReplyAdrian says:March 24, 2015
-
I'm using IB and explicit ibv_modify_qp()Dotan Barak says:March 24, 2015
-
It is weird; since this return value isn't really returned ...
Which QP transport type are you using?
Can you please write explicit the values that you set + the used mask in the ibv_modify_qp()?Thanks
DotanAdrian says:March 24, 2015
-
-
-
Hi Dotan,
the code isattr.qp_state = IBV_QPS_RTR;
attr.path_mtu = mtu;
attr.dest_qp_num = rmsg.qp_num[0];
attr.rq_psn = rmsg.psn[0];
attr.ah_attr.dlid = rmsg.lid;
attr.max_dest_rd_atomic = 1;
attr.min_rnr_timer = 12;
attr.ah_attr.is_global = 0;
attr.ah_attr.sl = 0;
attr.ah_attr.src_path_bits = 0;if (ibv_modify_qp(qp, &attr, IBV_QP_STATE | IBV_QP_AV | IBV_QP_PATH_MTU |
IBV_QP_DEST_QPN | IBV_QP_RQ_PSN | IBV_QP_MIN_RNR_TIMER |
IBV_QP_MAX_DEST_RD_ATOMIC)){
cerr << "Failed to modify QP 1 to RTR." << endl;
abort();
}lid, qp_num and psn are sent via TCP sockets.
Thanks
ReplyDotan Barak says:March 26, 2015
-
Hi.
I assume that you are using RC QP.
What about attr.ah_attr.port_num?Thanks
DotanReplyAdrian says:March 31, 2015
-
Yes, I use RC QP and the port number is 1. Finally, we reinstall the OFED drivers and it works.
Thank you for your time.Dotan Barak says:March 31, 2015
-
Sure.
Maybe there were some inconsistencies between the various libraries..
I'm glad at the end everything worked for you
:)
DotanSangwook says:April 14, 2015
-
-
Hi Adrian & Dotan. Same error happens in my code too.
Explicit ibv_modify_qp call to change QP state to RTR fails and returns 101. (My code is based on ibv_rc_pingpong program, an example from libibverb library. And ibv_rc_pingpong also fails in my environment.)
You said the problem is solved after reinstalling OFED drivers.. Can you share your system environment and OFED version?
(I installed OFED ver 2.4-1.0.4 on Intel Xeon E5-2670 + MLNX ConnectX-3 machine running Ubuntu 14.04 with kernel 3.13.0-48.)ReplySangwook says:April 14, 2015
-
Hi, I wrote a comment to ask for Adrian's system environment. But I figured out the cause of my problem- I had to add "gid-index" option in my program because I'm using RoCE. My program worked fine with that option. Anyway, thanks!
ReplyDotan Barak says:April 16, 2015
-
I'm glad that everything is fine.
I'll add this note on RoCE, so people won't fail on it again...
Thanks
DotanJesus Camacho says:May 5, 2015
-
-
-
Hi Dotan!
I am running the ib_write_lat microbenchmark with OpenSM.
I am disabling a link while running them and then OpenSM reconfigures all the network and the benchmark continues working without problems.
If have reduce the number of retries in the QPs to 1 or 2, then ib_wirte_lat crash in some cases now.
However, if I change the timeout of the QPs, I have the same behaviour.
I expected to have less retries if I increase the QP timeout. I understand that every retry is triggered after the QP timeout expires.
Am i right?
If not, how is it affecting the QP timeout to the number of retries?
Thanks for your help!
Best regards,
Jesus CamachoReplyDotan Barak says:May 5, 2015
-
Hi Jesus.
Yes, you are right: every retry is triggered after a QP timeout expires.
There is an internal clock in the RDMA device, and at least after the timeout expires
(+ a delta which depends on this clock resolution), there is a retransmission.If (in your experience) the link will be stable until the last retransmission,
ib_write_lat will continue working..Thanks
Dotan
If until the
What is "important" is theReplyJesus Camacho says:May 6, 2015
-
Hi Dotan,
I am replacing a LFT instead of switching on the link back.
It takes less than 2 ms from the time I send the new block to the LFT and I receive its ACK later. Then the new path should be ready in this short period of time.In one of my experiments I am using only one retry and timeout 10 (4194.304 usec).
Although I spend only 4 ms until the next retransmission, I have to spend up to 0.5 seconds for a successful completion when polling the completion queue (ibv_poll_cq using RC).
Do you know why this takes all this time?Thank you!
JesusDotan Barak says:May 7, 2015 -
Hi Jesus.
What does LFT mean? (I don't really familiar with this term)
Are you performing Automatic Path Migartion?
What are the values of the timeout/retry_count?Thanks
DotanJesus Camacho says:May 8, 2015
-
-
-
Hi Dotan,
instead of using Automatic Path Migration, I am replacing the forwarding tables (trough OpenSM) in some of the switches to avoid the broken link (I just say the switches to take another port to avoid the link that is down). This takes about 2 milliseconds.
I am using this values in the ib_write_lat microbenchmark:
timeout 10
retry_cnt 1This seems to work (I am using a small network). Then I understand that the retransmission is OK in less than timeout+timeout.
I am measuring the time to send a packet from one node to another and this takes usually several milliseconds.
However, it takes longer after the fault. And most of the time is spent polling the completion queue -ibv_poll_cq- (up to 0.5 seconds).
Do you know what could be the reason for this?
Thanks for your time :-)
JesusReplyDotan Barak says:May 8, 2015
-
You are welcome
:)I don't have a solid answer here (from knowledge), but I have an assumption:
RDMA is very fine tune to handle good flow (since this happen most of the time).In case of error flow starts (i.e. retransmission, violation, etc.), maybe this flow isn't fully tuned.
Another option is that this happening because of other issues
(for example: is the NUMA node is local or remote? do you use CPU affinity to the running process? etc.)I would suggest to use a sniffing tool and try to measure this (when the message is send, when an ack is returned, and when there is a Work Completion available in the CQ) or send an email to the support team of the HW vendor about this.
BTW, making the SM change the subnet may cause the subnet to start recovery flows
(since *maybe* client reregister events are created in the subnet's nodes).Thanks
Dotan
-
-
Anonymous says:October 1, 2015
Hi Dotan,
When I tried to modify retry_cnt to 7 using ibv_modify_qp,
I got "Invalid argument". Do you have any suggestion ?
I listed my code here:struct ibv_qp_attr attr;
int qret;
attr.retry_cnt = (uint8_t)7;
attr.timeout = (uint8_t)14;
attr.rnr_retry = (uint8_t)7;
qret = ibv_modify_qp(id->qp, &attr, IBV_QP_RETRY_CNT|IBV_QP_TIMEOU T|IBV_QP_RNR_RETRY);Thanks
ReplyDotan Barak says:October 1, 2015
-
Hi.
From which QP state to which QP state did you try to do this?
I suspect that you need more flags in the QP transition that you made.
Thanks
DotanReplyJunhyun says:August 12, 2016
-
-
Hi Dotan, I recently built an event-driven RDMA send/recv IO engine for multiplexing multiple QP events and so far I worked out most kinks, but it suffers when more than one connection is introduced.
What frustrates me is that the connection works perfectly until I get the WC for my first WR! (QP transition and post_send doesn't return any errors, even with the second connection)
The second connection keeps throwing me status 12 (RETRY_EXC_ERROR) WCs while the first connection keeps on exchanging data smoothly. When I retry the post_send, I get status 5, vendor_err 249. I took it to mean that the QP has already transitioned to an error state.Environment:
I use dual-port Mellanox CX-3 EN 10GBe on RoCE. Port 1 acts both as the port for my shell connection and RoCE.
I have made it so that all QPs from the same device share device context and pd, but they use disjoint MRs.
Here's the code I use for creating and transitioning QPs.
Every created QPs follow this exact path.
I wish I could find more explanation as to why this keeps failing.
Please tell me if I'm doing anything wrong.
conn.port is the local port I chose for use, and remote_info members are obtained from the other host over TCP connection. (byte-reordered)
Again, thanks for maintaining such a helpful site.struct ibv_qp_init_attr qp_init_attr;
memset(&qp_init_attr, 0, sizeof(qp_init_attr));
qp_init_attr.qp_type = IBV_QPT_RC;
qp_init_attr.sq_sig_all = 1;
qp_init_attr.send_cq = conn.cq;
qp_init_attr.recv_cq = conn.cq;
qp_init_attr.cap.max_send_wr = DEFAULT_CQ_SIZE / 2;
qp_init_attr.cap.max_recv_wr = DEFAULT_CQ_SIZE / 2;
qp_init_attr.cap.max_send_sge = std::min((int)MAX_SGE, _per_device_attr[device].max_sge);
qp_init_attr.cap.max_recv_sge = std::min((int)MAX_SGE, _per_device_attr[device].max_sge);
CHECK_PTR(conn.qp = ibv_create_qp(conn.pd, &qp_init_attr), "Failed to create QP for device " << device << ", port " << port << ", error=" << strerror(errno) << ", pd=" < INIT
struct ibv_qp_attr to_init;
int flags;
memset(&to_init, 0, sizeof(to_init));
to_init.qp_state = IBV_QPS_INIT;
to_init.port_num = conn.port_num;
to_init.pkey_index = 0;
to_init.qp_access_flags = IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE;
flags = IBV_QP_STATE | IBV_QP_PKEY_INDEX | IBV_QP_PORT | IBV_QP_ACCESS_FLAGS;
CHECK_EQ0(ibv_modify_qp(conn.qp, &to_init, flags), "QP transition failed (RESET -> INIT): " <RTR
struct ibv_qp_attr to_rtr;
int flags;
memset(&to_rtr, 0, sizeof(to_rtr));
to_rtr.qp_state = IBV_QPS_RTR;
to_rtr.path_mtu = DEFAULT_QP_MTU;
to_rtr.dest_qp_num = remote_info.qp_num;
to_rtr.rq_psn = 0;
to_rtr.max_dest_rd_atomic = 0;
to_rtr.min_rnr_timer = 12;
to_rtr.ah_attr.dlid = remote_info.lid;
to_rtr.ah_attr.sl = 0;
to_rtr.ah_attr.src_path_bits = 0;
to_rtr.ah_attr.port_num = conn.port_num;
// Begin RoCE specific conf (currently universal)
to_rtr.ah_attr.is_global = 1;
memcpy(&to_rtr.ah_attr.grh.dgid, remote_info.gid, sizeof(remote_info.gid));
to_rtr.ah_attr.grh.flow_label = 0;
to_rtr.ah_attr.grh.hop_limit = 1;
to_rtr.ah_attr.grh.sgid_index = 0;
to_rtr.ah_attr.grh.traffic_class = 0;
flags = IBV_QP_STATE | IBV_QP_AV | IBV_QP_PATH_MTU | IBV_QP_DEST_QPN |
IBV_QP_RQ_PSN | IBV_QP_MAX_DEST_RD_ATOMIC | IBV_QP_MIN_RNR_TIMER;
CHECK_EQ0(ibv_modify_qp(conn.qp, &to_rtr, flags), "QP transition failed (INIT -> RTR): " < RTS
struct ibv_qp_attr to_rts;
int flags;
memset(&to_rts, 0, sizeof(to_rts));
to_rts.qp_state = IBV_QPS_RTS;
to_rts.timeout = 14;
to_rts.retry_cnt = 7;
to_rts.rnr_retry = 7;
to_rts.sq_psn = 0;
to_rts.max_rd_atomic = 0;
flags = IBV_QP_STATE | IBV_QP_TIMEOUT | IBV_QP_RETRY_CNT |
IBV_QP_RNR_RETRY | IBV_QP_SQ_PSN | IBV_QP_MAX_QP_RD_ATOMIC;
CHECK_EQ0(ibv_modify_qp(conn.qp, &to_rts, flags), "QP transition failed (RTR -> RTS)" << getQPDescription(local_info));ReplyDotan Barak says:August 18, 2016
-
Hi.
Please make the following:
1) verify that the DEFAULT_QP_MTU is less than the interface MTU
(i.e. if the network I/F MTU is 1500, this value should be 1024).
2) All remote attributes are correct
3) Make sure that there is a synchronization between both sides
(to prevent a case where the sender send a message and the receiver isn't in RTR state,
or in Error state).Thanks
Dotan
-
-
Arvind says:April 10, 2019
Why does the state have to go to RTR and then to RTS. Can't we just switch from Init to RTS?
ReplyDotan Barak says:April 11, 2019
-
Hi.
* When transitioning to RTR - only the receive attributes are filled (the QP can stay at this state, if it will only receive messages)
* When transitioning to RTS - only the send attributes are filledAt the end of the day, this is the what the InfiniBand spec specifies...
Thanks
DotanReplyqiuhaonan says:April 19, 2019
-
-
Hello Dotan.
I have met an interesting problem. At first, I connected two QPs by exchanging the meta info(QPN, GID and etc) and modifying them to RTS. Then exchanging some data on them. Finally, I closed one QP by ibv_destroy_qp and the other QP got errors when sending data to the closed QP, but when I tried to modify the error QP to RTS, it could return to the RTS state and got errors when trying to send data again. The curious thing was why the error QP could be modified to RTS state when the remote target QP had been destroyed. Aren't there some SYN-like packets to be exchanged between the two QPs? Thanks.
HaonanReplyDotan Barak says:April 19, 2019
-
Hi.
I have a feeling that you used Reliable Connected QP.
When you got the error in the QP, you transitioned it back to RTS;
however, there wasn't any remote side - so you got (again) transport error.No, there isn't any "SYN" like packets - they are replaced with Communication Manager
(but you didn't use the Communication Manager, you replaced it with a socket).The behavior is as expected
:)Thanks
DotanReplyqiuhaonan says:April 19, 2019
-
-
Hi Dotan.
Yes, I used RC QP and verbs for this test.
Thanks.ReplyWeizhi Liao says:December 30, 2019
-
Hi Dotan:
After reading this blog,I feel confused about the required and optional attributes of QP on status transitions 。 the Linux manpages 、rdma_aware_network_programming_user_manual and your blog have different description 。which is right? ThanksReplyDotan Barak says:February 22, 2020
-
Hi.
What do you mean?
Can you give me an example?Thanks
DotanReplyraphael says:May 21, 2020
-
-
Hi Dotan
I will use an RC queue pair and Write verb from an FPGA device (w/o RDMA network card, 100GbE MAC). For simulation purpose with a workstation and a connectx5 I use python / Scapy to build a RoCEv2 Write_IMM packet from the data and receiver qp_num, rkey, addr, etc. The receiver is also a Connectx-5.
it works, I get the data and the Immediate value, but only for 8 transfers. The last RC acknowledge from receiver is a RNR Nak.
Is it possible to configure the Receiver HCA to continuously recive the data ?
thanksReplyDotan Barak says:July 10, 2020
-
Hi.
RNR Nak means that the "Receiver Not Ready", this means that there isn't any Receiver Request in that (responder) QP.
So, I suggest that you'll post more Receive Requests for that side.Thanks
DotanReplyXiangyu Zhang says:July 13, 2020
-
-
Hi Dotan:
These days I encounter an error:after run ibv_post_send to issue an RDMA READ operation, there is nothing happened, this means that the RDMA READ operation don't appears in CQ. After to read "Help, I've posted and Send Request and it wasn't completed with a corresponding Work Completion. What happened?" in "ibv_post_send" page, I decide to try advise in it. The problem is I can't modify qp attribute, the ibv_modity_qp return -22 all the time, below is my code:
int ret = 0;
rdma_addrinfo* addrinfo;
rdma_addrinfo hints = {};
struct ibv_qp_attr qp_attr;
hints.ai_port_space = RDMA_PS_TCP;
struct ibv_qp *client_qp = nullptr;
if (rdma_getaddrinfo(const_cast(host.c_str()),
const_cast(port.c_str()), &hints, &addrinfo)) {
return errors::InvalidArgument(
strerror(errno), ": ", "cannot connect to rdma://", host, ":", port);
}ibv_qp_init_attr init_attr = {};
init_attr.qp_type = IBV_QPT_RC;
init_attr.cap.max_recv_wr = 1;
init_attr.cap.max_send_wr = 1024;
init_attr.cap.max_recv_sge = 1;
init_attr.cap.max_send_sge = 1;rdma_cm_id* id;
if (rdma_create_ep(&id, addrinfo, nullptr, &init_attr)) {
rdma_freeaddrinfo(addrinfo);
return errors::Unavailable(strerror(errno), ": ",
"cannot create endpoint to rdma://", host, ":",
port);
}
rdma_freeaddrinfo(addrinfo);client_qp = id->qp;
// motify qp's attr.
bzero(&qp_attr, sizeof(qp_attr));
qp_attr.qp_state = IBV_QPS_RTS;
qp_attr.timeout = 14;
qp_attr.retry_cnt = 7;ret = ibv_modify_qp(client_qp, &qp_attr,
IBV_QP_TIMEOUT |
IBV_QP_STATE |
IBV_QP_RETRY_CNT);
if (ret) {
fprintf(stderr, "failed to motify qp, %s\n", strerror(errno));
abort();
}
if (rdma_connect(id, nullptr)) {
rdma_destroy_ep(id);
return errors::Unavailable(strerror(errno), ": ",
"cannot connect to rdma://", host, ":", port);
}My rdma NIC type is RoCE, could you help me to see why I can't motify qp statue?
ReplyDotan Barak says:July 15, 2020
-
Hi.
What is the QP state before you called ibv_modify_qp()?
Thanks
Dotan
-
cdef class QPInitAttr(PyverbsObject): | |
def __init__(self, qp_type=e.IBV_QPT_UD, qp_context=None, | |
PyverbsObject scq=None, PyverbsObject rcq=None, | |
SRQ srq=None, QPCap cap=None, sq_sig_all=1): | |
""" | |
Initializes a QpInitAttr object representing ibv_qp_init_attr struct. | |
Note that SRQ object is not yet supported in pyverbs so can't be passed | |
as a parameter. None should be used until such support is added. | |
:param qp_type: The desired QP type (see enum ibv_qp_type) | |
:param qp_context: Associated QP context | |
:param scq: Send CQ to be used for this QP | |
:param rcq: Receive CQ to be used for this QP | |
:param srq: Shared receive queue to be used as RQ in QP | |
:param cap: A QPCap object | |
:param sq_sig_all: If set, each send WR will generate a completion | |
entry | |
:return: A QpInitAttr object | |
""" |
/*If set, each Work Request (WR) submitted to the SQ*/ |