Merge tag '6.16-rc-part2-smb3-client-fixes' of git://git.samba.org/sfrench/cifs-2.6

Pull more smb client updates from Steve French:

- multichannel/reconnect fixes

- move smbdirect (smb over RDMA) defines to fs/smb/common so they will
be able to be used in the future more broadly, and a documentation
update explaining setting up smbdirect mounts

- update email address for Paulo

* tag '6.16-rc-part2-smb3-client-fixes' of git://git.samba.org/sfrench/cifs-2.6:
cifs: update internal version number
MAINTAINERS, mailmap: Update Paulo Alcantara's email address
cifs: add documentation for smbdirect setup
cifs: do not disable interface polling on failure
cifs: serialize other channels when query server interfaces is pending
cifs: deal with the channel loading lag while picking channels
smb: client: make use of common smbdirect_socket_parameters
smb: smbdirect: introduce smbdirect_socket_parameters
smb: client: make use of common smbdirect_socket
smb: smbdirect: add smbdirect_socket.h
smb: client: make use of common smbdirect.h
smb: smbdirect: add smbdirect.h with public structures
smb: client: make use of common smbdirect_pdu.h
smb: smbdirect: add smbdirect_pdu.h with protocol definitions

+532 -287
+6
.mailmap
··· 602 Paul Mackerras <paulus@ozlabs.org> <paulus@au1.ibm.com> 603 Paul Moore <paul@paul-moore.com> <paul.moore@hp.com> 604 Paul Moore <paul@paul-moore.com> <pmoore@redhat.com> 605 Pavankumar Kondeti <quic_pkondeti@quicinc.com> <pkondeti@codeaurora.org> 606 Peter A Jonsson <pj@ludd.ltu.se> 607 Peter Oruba <peter.oruba@amd.com>
··· 602 Paul Mackerras <paulus@ozlabs.org> <paulus@au1.ibm.com> 603 Paul Moore <paul@paul-moore.com> <paul.moore@hp.com> 604 Paul Moore <paul@paul-moore.com> <pmoore@redhat.com> 605 + Paulo Alcantara <pc@manguebit.org> <pcacjr@zytor.com> 606 + Paulo Alcantara <pc@manguebit.org> <paulo@paulo.ac> 607 + Paulo Alcantara <pc@manguebit.org> <pc@cjr.nz> 608 + Paulo Alcantara <pc@manguebit.org> <palcantara@suse.de> 609 + Paulo Alcantara <pc@manguebit.org> <palcantara@suse.com> 610 + Paulo Alcantara <pc@manguebit.org> <pc@manguebit.com> 611 Pavankumar Kondeti <quic_pkondeti@quicinc.com> <pkondeti@codeaurora.org> 612 Peter A Jonsson <pj@ludd.ltu.se> 613 Peter Oruba <peter.oruba@amd.com>
+1
Documentation/filesystems/smb/index.rst
··· 8 9 ksmbd 10 cifsroot
··· 8 9 ksmbd 10 cifsroot 11 + smbdirect
+103
Documentation/filesystems/smb/smbdirect.rst
···
··· 1 + .. SPDX-License-Identifier: GPL-2.0 2 + 3 + =========================== 4 + SMB Direct - SMB3 over RDMA 5 + =========================== 6 + 7 + This document describes how to set up the Linux SMB client and server to 8 + use RDMA. 9 + 10 + Overview 11 + ======== 12 + The Linux SMB kernel client supports SMB Direct, which is a transport 13 + scheme for SMB3 that uses RDMA (Remote Direct Memory Access) to provide 14 + high throughput and low latencies by bypassing the traditional TCP/IP 15 + stack. 16 + SMB Direct on the Linux SMB client can be tested against KSMBD - a 17 + kernel-space SMB server. 18 + 19 + Installation 20 + ============= 21 + - Install an RDMA device. As long as the RDMA device driver is supported 22 + by the kernel, it should work. This includes both software emulators (soft 23 + RoCE, soft iWARP) and hardware devices (InfiniBand, RoCE, iWARP). 24 + 25 + - Install a kernel with SMB Direct support. The first kernel release to 26 + support SMB Direct on both the client and server side is 5.15. Therefore, 27 + a distribution compatible with kernel 5.15 or later is required. 28 + 29 + - Install cifs-utils, which provides the `mount.cifs` command to mount SMB 30 + shares. 31 + 32 + - Configure the RDMA stack 33 + 34 + Make sure that your kernel configuration has RDMA support enabled. Under 35 + Device Drivers -> Infiniband support, update the kernel configuration to 36 + enable Infiniband support. 37 + 38 + Enable the appropriate IB HCA support or iWARP adapter support, 39 + depending on your hardware. 40 + 41 + If you are using InfiniBand, enable IP-over-InfiniBand support. 42 + 43 + For soft RDMA, enable either the soft iWARP (`RDMA _SIW`) or soft RoCE 44 + (`RDMA_RXE`) module. Install the `iproute2` package and use the 45 + `rdma link add` command to load the module and create an 46 + RDMA interface. 47 + 48 + e.g. if your local ethernet interface is `eth0`, you can use: 49 + 50 + .. code-block:: bash 51 + 52 + sudo rdma link add siw0 type siw netdev eth0 53 + 54 + - Enable SMB Direct support for both the server and the client in the kernel 55 + configuration. 56 + 57 + Server Setup 58 + 59 + .. code-block:: text 60 + 61 + Network File Systems ---> 62 + <M> SMB3 server support 63 + [*] Support for SMB Direct protocol 64 + 65 + Client Setup 66 + 67 + .. code-block:: text 68 + 69 + Network File Systems ---> 70 + <M> SMB3 and CIFS support (advanced network filesystem) 71 + [*] SMB Direct support 72 + 73 + - Build and install the kernel. SMB Direct support will be enabled in the 74 + cifs.ko and ksmbd.ko modules. 75 + 76 + Setup and Usage 77 + ================ 78 + 79 + - Set up and start a KSMBD server as described in the `KSMBD documentation 80 + <https://www.kernel.org/doc/Documentation/filesystems/smb/ksmbd.rst>`_. 81 + Also add the "server multi channel support = yes" parameter to ksmbd.conf. 82 + 83 + - On the client, mount the share with `rdma` mount option to use SMB Direct 84 + (specify a SMB version 3.0 or higher using `vers`). 85 + 86 + For example: 87 + 88 + .. code-block:: bash 89 + 90 + mount -t cifs //server/share /mnt/point -o vers=3.1.1,rdma 91 + 92 + - To verify that the mount is using SMB Direct, you can check dmesg for the 93 + following log line after mounting: 94 + 95 + .. code-block:: text 96 + 97 + CIFS: VFS: RDMA transport established 98 + 99 + Or, verify `rdma` mount option for the share in `/proc/mounts`: 100 + 101 + .. code-block:: bash 102 + 103 + cat /proc/mounts | grep cifs
+2 -2
MAINTAINERS
··· 5986 COMMON INTERNET FILE SYSTEM CLIENT (CIFS and SMB3) 5987 M: Steve French <sfrench@samba.org> 5988 M: Steve French <smfrench@gmail.com> 5989 - R: Paulo Alcantara <pc@manguebit.com> (DFS, global name space) 5990 R: Ronnie Sahlberg <ronniesahlberg@gmail.com> (directory leases, sparse files) 5991 R: Shyam Prasad N <sprasad@microsoft.com> (multichannel) 5992 R: Tom Talpey <tom@talpey.com> (RDMA, smbdirect) ··· 9280 9281 FILESYSTEMS [NETFS LIBRARY] 9282 M: David Howells <dhowells@redhat.com> 9283 - M: Paulo Alcantara <pc@manguebit.com> 9284 L: netfs@lists.linux.dev 9285 L: linux-fsdevel@vger.kernel.org 9286 S: Supported
··· 5986 COMMON INTERNET FILE SYSTEM CLIENT (CIFS and SMB3) 5987 M: Steve French <sfrench@samba.org> 5988 M: Steve French <smfrench@gmail.com> 5989 + R: Paulo Alcantara <pc@manguebit.org> (DFS, global name space) 5990 R: Ronnie Sahlberg <ronniesahlberg@gmail.com> (directory leases, sparse files) 5991 R: Shyam Prasad N <sprasad@microsoft.com> (multichannel) 5992 R: Tom Talpey <tom@talpey.com> (RDMA, smbdirect) ··· 9280 9281 FILESYSTEMS [NETFS LIBRARY] 9282 M: David Howells <dhowells@redhat.com> 9283 + M: Paulo Alcantara <pc@manguebit.org> 9284 L: netfs@lists.linux.dev 9285 L: linux-fsdevel@vger.kernel.org 9286 S: Supported
+14 -9
fs/smb/client/cifs_debug.c
··· 362 c = 0; 363 spin_lock(&cifs_tcp_ses_lock); 364 list_for_each_entry(server, &cifs_tcp_ses_list, tcp_ses_list) { 365 /* channel info will be printed as a part of sessions below */ 366 if (SERVER_IS_CHAN(server)) 367 continue; ··· 387 seq_printf(m, "\nSMBDirect transport not available"); 388 goto skip_rdma; 389 } 390 391 seq_printf(m, "\nSMBDirect (in hex) protocol version: %x " 392 "transport status: %x", 393 server->smbd_conn->protocol, 394 - server->smbd_conn->transport_status); 395 seq_printf(m, "\nConn receive_credit_max: %x " 396 "send_credit_target: %x max_send_size: %x", 397 - server->smbd_conn->receive_credit_max, 398 - server->smbd_conn->send_credit_target, 399 - server->smbd_conn->max_send_size); 400 seq_printf(m, "\nConn max_fragmented_recv_size: %x " 401 "max_fragmented_send_size: %x max_receive_size:%x", 402 - server->smbd_conn->max_fragmented_recv_size, 403 - server->smbd_conn->max_fragmented_send_size, 404 - server->smbd_conn->max_receive_size); 405 seq_printf(m, "\nConn keep_alive_interval: %x " 406 "max_readwrite_size: %x rdma_readwrite_threshold: %x", 407 - server->smbd_conn->keep_alive_interval, 408 - server->smbd_conn->max_readwrite_size, 409 server->smbd_conn->rdma_readwrite_threshold); 410 seq_printf(m, "\nDebug count_get_receive_buffer: %x " 411 "count_put_receive_buffer: %x count_send_empty: %x",
··· 362 c = 0; 363 spin_lock(&cifs_tcp_ses_lock); 364 list_for_each_entry(server, &cifs_tcp_ses_list, tcp_ses_list) { 365 + #ifdef CONFIG_CIFS_SMB_DIRECT 366 + struct smbdirect_socket_parameters *sp; 367 + #endif 368 + 369 /* channel info will be printed as a part of sessions below */ 370 if (SERVER_IS_CHAN(server)) 371 continue; ··· 383 seq_printf(m, "\nSMBDirect transport not available"); 384 goto skip_rdma; 385 } 386 + sp = &server->smbd_conn->socket.parameters; 387 388 seq_printf(m, "\nSMBDirect (in hex) protocol version: %x " 389 "transport status: %x", 390 server->smbd_conn->protocol, 391 + server->smbd_conn->socket.status); 392 seq_printf(m, "\nConn receive_credit_max: %x " 393 "send_credit_target: %x max_send_size: %x", 394 + sp->recv_credit_max, 395 + sp->send_credit_target, 396 + sp->max_send_size); 397 seq_printf(m, "\nConn max_fragmented_recv_size: %x " 398 "max_fragmented_send_size: %x max_receive_size:%x", 399 + sp->max_fragmented_recv_size, 400 + sp->max_fragmented_send_size, 401 + sp->max_recv_size); 402 seq_printf(m, "\nConn keep_alive_interval: %x " 403 "max_readwrite_size: %x rdma_readwrite_threshold: %x", 404 + sp->keepalive_interval_msec * 1000, 405 + sp->max_read_write_size, 406 server->smbd_conn->rdma_readwrite_threshold); 407 seq_printf(m, "\nDebug count_get_receive_buffer: %x " 408 "count_put_receive_buffer: %x count_send_empty: %x",
+2 -2
fs/smb/client/cifsfs.h
··· 145 #endif /* CONFIG_CIFS_NFSD_EXPORT */ 146 147 /* when changing internal version - update following two lines at same time */ 148 - #define SMB3_PRODUCT_BUILD 54 149 - #define CIFS_VERSION "2.54" 150 #endif /* _CIFSFS_H */
··· 145 #endif /* CONFIG_CIFS_NFSD_EXPORT */ 146 147 /* when changing internal version - update following two lines at same time */ 148 + #define SMB3_PRODUCT_BUILD 55 149 + #define CIFS_VERSION "2.55" 150 #endif /* _CIFSFS_H */
+1
fs/smb/client/cifsglob.h
··· 1085 }; 1086 1087 #define CIFS_SES_FLAG_SCALE_CHANNELS (0x1) 1088 1089 /* 1090 * Session structure. One of these for each uid session with a particular host
··· 1085 }; 1086 1087 #define CIFS_SES_FLAG_SCALE_CHANNELS (0x1) 1088 + #define CIFS_SES_FLAGS_PENDING_QUERY_INTERFACES (0x2) 1089 1090 /* 1091 * Session structure. One of these for each uid session with a particular host
+1 -5
fs/smb/client/connect.c
··· 116 rc = server->ops->query_server_interfaces(xid, tcon, false); 117 free_xid(xid); 118 119 - if (rc) { 120 - if (rc == -EOPNOTSUPP) 121 - return; 122 - 123 cifs_dbg(FYI, "%s: failed to query server interfaces: %d\n", 124 __func__, rc); 125 - } 126 127 queue_delayed_work(cifsiod_wq, &tcon->query_interfaces, 128 (SMB_INTERFACE_POLL_INTERVAL * HZ));
··· 116 rc = server->ops->query_server_interfaces(xid, tcon, false); 117 free_xid(xid); 118 119 + if (rc) 120 cifs_dbg(FYI, "%s: failed to query server interfaces: %d\n", 121 __func__, rc); 122 123 queue_delayed_work(cifsiod_wq, &tcon->query_interfaces, 124 (SMB_INTERFACE_POLL_INTERVAL * HZ));
+10 -4
fs/smb/client/smb2ops.c
··· 504 wsize = min_t(unsigned int, wsize, server->max_write); 505 #ifdef CONFIG_CIFS_SMB_DIRECT 506 if (server->rdma) { 507 if (server->sign) 508 /* 509 * Account for SMB2 data transfer packet header and ··· 514 */ 515 wsize = min_t(unsigned int, 516 wsize, 517 - server->smbd_conn->max_fragmented_send_size - 518 SMB2_READWRITE_PDU_HEADER_SIZE - 519 sizeof(struct smb2_transform_hdr)); 520 else 521 wsize = min_t(unsigned int, 522 - wsize, server->smbd_conn->max_readwrite_size); 523 } 524 #endif 525 if (!(server->capabilities & SMB2_GLOBAL_CAP_LARGE_MTU)) ··· 555 rsize = min_t(unsigned int, rsize, server->max_read); 556 #ifdef CONFIG_CIFS_SMB_DIRECT 557 if (server->rdma) { 558 if (server->sign) 559 /* 560 * Account for SMB2 data transfer packet header and ··· 565 */ 566 rsize = min_t(unsigned int, 567 rsize, 568 - server->smbd_conn->max_fragmented_recv_size - 569 SMB2_READWRITE_PDU_HEADER_SIZE - 570 sizeof(struct smb2_transform_hdr)); 571 else 572 rsize = min_t(unsigned int, 573 - rsize, server->smbd_conn->max_readwrite_size); 574 } 575 #endif 576
··· 504 wsize = min_t(unsigned int, wsize, server->max_write); 505 #ifdef CONFIG_CIFS_SMB_DIRECT 506 if (server->rdma) { 507 + struct smbdirect_socket_parameters *sp = 508 + &server->smbd_conn->socket.parameters; 509 + 510 if (server->sign) 511 /* 512 * Account for SMB2 data transfer packet header and ··· 511 */ 512 wsize = min_t(unsigned int, 513 wsize, 514 + sp->max_fragmented_send_size - 515 SMB2_READWRITE_PDU_HEADER_SIZE - 516 sizeof(struct smb2_transform_hdr)); 517 else 518 wsize = min_t(unsigned int, 519 + wsize, sp->max_read_write_size); 520 } 521 #endif 522 if (!(server->capabilities & SMB2_GLOBAL_CAP_LARGE_MTU)) ··· 552 rsize = min_t(unsigned int, rsize, server->max_read); 553 #ifdef CONFIG_CIFS_SMB_DIRECT 554 if (server->rdma) { 555 + struct smbdirect_socket_parameters *sp = 556 + &server->smbd_conn->socket.parameters; 557 + 558 if (server->sign) 559 /* 560 * Account for SMB2 data transfer packet header and ··· 559 */ 560 rsize = min_t(unsigned int, 561 rsize, 562 + sp->max_fragmented_recv_size - 563 SMB2_READWRITE_PDU_HEADER_SIZE - 564 sizeof(struct smb2_transform_hdr)); 565 else 566 rsize = min_t(unsigned int, 567 + rsize, sp->max_read_write_size); 568 } 569 #endif 570
+32 -18
fs/smb/client/smb2pdu.c
··· 36 #include "smb2glob.h" 37 #include "cifspdu.h" 38 #include "cifs_spnego.h" 39 #include "smbdirect.h" 40 #include "trace.h" 41 #ifdef CONFIG_CIFS_DFS_UPCALL ··· 412 if (!rc && 413 (server->capabilities & SMB2_GLOBAL_CAP_MULTI_CHANNEL) && 414 server->ops->query_server_interfaces) { 415 - mutex_unlock(&ses->session_mutex); 416 - 417 /* 418 - * query server network interfaces, in case they change 419 */ 420 xid = get_xid(); 421 rc = server->ops->query_server_interfaces(xid, tcon, false); 422 free_xid(xid); 423 424 if (rc == -EOPNOTSUPP && ses->chan_count > 1) { 425 /* ··· 448 if (ses->chan_max > ses->chan_count && 449 ses->iface_count && 450 !SERVER_IS_CHAN(server)) { 451 - if (ses->chan_count == 1) { 452 cifs_server_dbg(VFS, "supports multichannel now\n"); 453 - queue_delayed_work(cifsiod_wq, &tcon->query_interfaces, 454 - (SMB_INTERFACE_POLL_INTERVAL * HZ)); 455 - } 456 457 cifs_try_adding_channels(ses); 458 } ··· 567 struct TCP_Server_Info *server, 568 void **request_buf, unsigned int *total_len) 569 { 570 - /* Skip reconnect only for FSCTL_VALIDATE_NEGOTIATE_INFO IOCTLs */ 571 - if (opcode == FSCTL_VALIDATE_NEGOTIATE_INFO) { 572 return __smb2_plain_req_init(SMB2_IOCTL, tcon, server, 573 request_buf, total_len); 574 - } 575 return smb2_plain_req_init(SMB2_IOCTL, tcon, server, 576 request_buf, total_len); 577 } ··· 4463 #ifdef CONFIG_CIFS_SMB_DIRECT 4464 /* 4465 * If we want to do a RDMA write, fill in and append 4466 - * smbd_buffer_descriptor_v1 to the end of read request 4467 */ 4468 if (rdata && smb3_use_rdma_offload(io_parms)) { 4469 - struct smbd_buffer_descriptor_v1 *v1; 4470 bool need_invalidate = server->dialect == SMB30_PROT_ID; 4471 4472 rdata->mr = smbd_register_mr(server->smbd_conn, &rdata->subreq.io_iter, ··· 4480 req->ReadChannelInfoOffset = 4481 cpu_to_le16(offsetof(struct smb2_read_req, Buffer)); 4482 req->ReadChannelInfoLength = 4483 - cpu_to_le16(sizeof(struct smbd_buffer_descriptor_v1)); 4484 - v1 = (struct smbd_buffer_descriptor_v1 *) &req->Buffer[0]; 4485 v1->offset = cpu_to_le64(rdata->mr->mr->iova); 4486 v1->token = cpu_to_le32(rdata->mr->mr->rkey); 4487 v1->length = cpu_to_le32(rdata->mr->mr->length); ··· 4989 #ifdef CONFIG_CIFS_SMB_DIRECT 4990 /* 4991 * If we want to do a server RDMA read, fill in and append 4992 - * smbd_buffer_descriptor_v1 to the end of write request 4993 */ 4994 if (smb3_use_rdma_offload(io_parms)) { 4995 - struct smbd_buffer_descriptor_v1 *v1; 4996 bool need_invalidate = server->dialect == SMB30_PROT_ID; 4997 4998 wdata->mr = smbd_register_mr(server->smbd_conn, &wdata->subreq.io_iter, ··· 5011 req->WriteChannelInfoOffset = 5012 cpu_to_le16(offsetof(struct smb2_write_req, Buffer)); 5013 req->WriteChannelInfoLength = 5014 - cpu_to_le16(sizeof(struct smbd_buffer_descriptor_v1)); 5015 - v1 = (struct smbd_buffer_descriptor_v1 *) &req->Buffer[0]; 5016 v1->offset = cpu_to_le64(wdata->mr->mr->iova); 5017 v1->token = cpu_to_le32(wdata->mr->mr->rkey); 5018 v1->length = cpu_to_le32(wdata->mr->mr->length);
··· 36 #include "smb2glob.h" 37 #include "cifspdu.h" 38 #include "cifs_spnego.h" 39 + #include "../common/smbdirect/smbdirect.h" 40 #include "smbdirect.h" 41 #include "trace.h" 42 #ifdef CONFIG_CIFS_DFS_UPCALL ··· 411 if (!rc && 412 (server->capabilities & SMB2_GLOBAL_CAP_MULTI_CHANNEL) && 413 server->ops->query_server_interfaces) { 414 /* 415 + * query server network interfaces, in case they change. 416 + * Also mark the session as pending this update while the query 417 + * is in progress. This will be used to avoid calling 418 + * smb2_reconnect recursively. 419 */ 420 + ses->flags |= CIFS_SES_FLAGS_PENDING_QUERY_INTERFACES; 421 xid = get_xid(); 422 rc = server->ops->query_server_interfaces(xid, tcon, false); 423 free_xid(xid); 424 + ses->flags &= ~CIFS_SES_FLAGS_PENDING_QUERY_INTERFACES; 425 + 426 + /* regardless of rc value, setup polling */ 427 + queue_delayed_work(cifsiod_wq, &tcon->query_interfaces, 428 + (SMB_INTERFACE_POLL_INTERVAL * HZ)); 429 + 430 + mutex_unlock(&ses->session_mutex); 431 432 if (rc == -EOPNOTSUPP && ses->chan_count > 1) { 433 /* ··· 438 if (ses->chan_max > ses->chan_count && 439 ses->iface_count && 440 !SERVER_IS_CHAN(server)) { 441 + if (ses->chan_count == 1) 442 cifs_server_dbg(VFS, "supports multichannel now\n"); 443 444 cifs_try_adding_channels(ses); 445 } ··· 560 struct TCP_Server_Info *server, 561 void **request_buf, unsigned int *total_len) 562 { 563 + /* 564 + * Skip reconnect in one of the following cases: 565 + * 1. For FSCTL_VALIDATE_NEGOTIATE_INFO IOCTLs 566 + * 2. For FSCTL_QUERY_NETWORK_INTERFACE_INFO IOCTL when called from 567 + * smb2_reconnect (indicated by CIFS_SES_FLAG_SCALE_CHANNELS ses flag) 568 + */ 569 + if (opcode == FSCTL_VALIDATE_NEGOTIATE_INFO || 570 + (opcode == FSCTL_QUERY_NETWORK_INTERFACE_INFO && 571 + (tcon->ses->flags & CIFS_SES_FLAGS_PENDING_QUERY_INTERFACES))) 572 return __smb2_plain_req_init(SMB2_IOCTL, tcon, server, 573 request_buf, total_len); 574 + 575 return smb2_plain_req_init(SMB2_IOCTL, tcon, server, 576 request_buf, total_len); 577 } ··· 4449 #ifdef CONFIG_CIFS_SMB_DIRECT 4450 /* 4451 * If we want to do a RDMA write, fill in and append 4452 + * smbdirect_buffer_descriptor_v1 to the end of read request 4453 */ 4454 if (rdata && smb3_use_rdma_offload(io_parms)) { 4455 + struct smbdirect_buffer_descriptor_v1 *v1; 4456 bool need_invalidate = server->dialect == SMB30_PROT_ID; 4457 4458 rdata->mr = smbd_register_mr(server->smbd_conn, &rdata->subreq.io_iter, ··· 4466 req->ReadChannelInfoOffset = 4467 cpu_to_le16(offsetof(struct smb2_read_req, Buffer)); 4468 req->ReadChannelInfoLength = 4469 + cpu_to_le16(sizeof(struct smbdirect_buffer_descriptor_v1)); 4470 + v1 = (struct smbdirect_buffer_descriptor_v1 *) &req->Buffer[0]; 4471 v1->offset = cpu_to_le64(rdata->mr->mr->iova); 4472 v1->token = cpu_to_le32(rdata->mr->mr->rkey); 4473 v1->length = cpu_to_le32(rdata->mr->mr->length); ··· 4975 #ifdef CONFIG_CIFS_SMB_DIRECT 4976 /* 4977 * If we want to do a server RDMA read, fill in and append 4978 + * smbdirect_buffer_descriptor_v1 to the end of write request 4979 */ 4980 if (smb3_use_rdma_offload(io_parms)) { 4981 + struct smbdirect_buffer_descriptor_v1 *v1; 4982 bool need_invalidate = server->dialect == SMB30_PROT_ID; 4983 4984 wdata->mr = smbd_register_mr(server->smbd_conn, &wdata->subreq.io_iter, ··· 4997 req->WriteChannelInfoOffset = 4998 cpu_to_le16(offsetof(struct smb2_write_req, Buffer)); 4999 req->WriteChannelInfoLength = 5000 + cpu_to_le16(sizeof(struct smbdirect_buffer_descriptor_v1)); 5001 + v1 = (struct smbdirect_buffer_descriptor_v1 *) &req->Buffer[0]; 5002 v1->offset = cpu_to_le64(wdata->mr->mr->iova); 5003 v1->token = cpu_to_le32(wdata->mr->mr->rkey); 5004 v1->length = cpu_to_le32(wdata->mr->mr->length);
+213 -176
fs/smb/client/smbdirect.c
··· 7 #include <linux/module.h> 8 #include <linux/highmem.h> 9 #include <linux/folio_queue.h> 10 #include "smbdirect.h" 11 #include "cifs_debug.h" 12 #include "cifsproto.h" ··· 50 }; 51 static ssize_t smb_extract_iter_to_rdma(struct iov_iter *iter, size_t len, 52 struct smb_extract_to_rdma *rdma); 53 - 54 - /* SMBD version number */ 55 - #define SMBD_V1 0x0100 56 57 /* Port numbers for SMBD transport */ 58 #define SMB_PORT 445 ··· 163 { 164 struct smbd_connection *info = 165 container_of(work, struct smbd_connection, disconnect_work); 166 167 - if (info->transport_status == SMBD_CONNECTED) { 168 - info->transport_status = SMBD_DISCONNECTING; 169 - rdma_disconnect(info->id); 170 } 171 } 172 ··· 181 struct rdma_cm_id *id, struct rdma_cm_event *event) 182 { 183 struct smbd_connection *info = id->context; 184 185 log_rdma_event(INFO, "event=%d status=%d\n", 186 event->event, event->status); ··· 205 206 case RDMA_CM_EVENT_ESTABLISHED: 207 log_rdma_event(INFO, "connected event=%d\n", event->event); 208 - info->transport_status = SMBD_CONNECTED; 209 wake_up_interruptible(&info->conn_wait); 210 break; 211 ··· 213 case RDMA_CM_EVENT_UNREACHABLE: 214 case RDMA_CM_EVENT_REJECTED: 215 log_rdma_event(INFO, "connecting failed event=%d\n", event->event); 216 - info->transport_status = SMBD_DISCONNECTED; 217 wake_up_interruptible(&info->conn_wait); 218 break; 219 220 case RDMA_CM_EVENT_DEVICE_REMOVAL: 221 case RDMA_CM_EVENT_DISCONNECTED: 222 /* This happens when we fail the negotiation */ 223 - if (info->transport_status == SMBD_NEGOTIATE_FAILED) { 224 - info->transport_status = SMBD_DISCONNECTED; 225 wake_up(&info->conn_wait); 226 break; 227 } 228 229 - info->transport_status = SMBD_DISCONNECTED; 230 wake_up_interruptible(&info->disconn_wait); 231 wake_up_interruptible(&info->wait_reassembly_queue); 232 wake_up_interruptible_all(&info->wait_send_queue); ··· 275 int i; 276 struct smbd_request *request = 277 container_of(wc->wr_cqe, struct smbd_request, cqe); 278 279 log_rdma_send(INFO, "smbd_request 0x%p completed wc->status=%d\n", 280 request, wc->status); ··· 288 } 289 290 for (i = 0; i < request->num_sge; i++) 291 - ib_dma_unmap_single(request->info->id->device, 292 request->sge[i].addr, 293 request->sge[i].length, 294 DMA_TO_DEVICE); ··· 301 mempool_free(request, request->info->request_mempool); 302 } 303 304 - static void dump_smbd_negotiate_resp(struct smbd_negotiate_resp *resp) 305 { 306 log_rdma_event(INFO, "resp message min_version %u max_version %u negotiated_version %u credits_requested %u credits_granted %u status %u max_readwrite_size %u preferred_send_size %u max_receive_size %u max_fragmented_size %u\n", 307 resp->min_version, resp->max_version, ··· 320 struct smbd_response *response, int packet_length) 321 { 322 struct smbd_connection *info = response->info; 323 - struct smbd_negotiate_resp *packet = smbd_response_payload(response); 324 325 - if (packet_length < sizeof(struct smbd_negotiate_resp)) { 326 log_rdma_event(ERR, 327 "error: packet_length=%d\n", packet_length); 328 return false; 329 } 330 331 - if (le16_to_cpu(packet->negotiated_version) != SMBD_V1) { 332 log_rdma_event(ERR, "error: negotiated_version=%x\n", 333 le16_to_cpu(packet->negotiated_version)); 334 return false; ··· 351 352 atomic_set(&info->receive_credits, 0); 353 354 - if (le32_to_cpu(packet->preferred_send_size) > info->max_receive_size) { 355 log_rdma_event(ERR, "error: preferred_send_size=%d\n", 356 le32_to_cpu(packet->preferred_send_size)); 357 return false; 358 } 359 - info->max_receive_size = le32_to_cpu(packet->preferred_send_size); 360 361 if (le32_to_cpu(packet->max_receive_size) < SMBD_MIN_RECEIVE_SIZE) { 362 log_rdma_event(ERR, "error: max_receive_size=%d\n", 363 le32_to_cpu(packet->max_receive_size)); 364 return false; 365 } 366 - info->max_send_size = min_t(int, info->max_send_size, 367 - le32_to_cpu(packet->max_receive_size)); 368 369 if (le32_to_cpu(packet->max_fragmented_size) < 370 SMBD_MIN_FRAGMENTED_SIZE) { ··· 372 le32_to_cpu(packet->max_fragmented_size)); 373 return false; 374 } 375 - info->max_fragmented_send_size = 376 le32_to_cpu(packet->max_fragmented_size); 377 info->rdma_readwrite_threshold = 378 - rdma_readwrite_threshold > info->max_fragmented_send_size ? 379 - info->max_fragmented_send_size : 380 rdma_readwrite_threshold; 381 382 383 - info->max_readwrite_size = min_t(u32, 384 le32_to_cpu(packet->max_readwrite_size), 385 info->max_frmr_depth * PAGE_SIZE); 386 - info->max_frmr_depth = info->max_readwrite_size / PAGE_SIZE; 387 388 return true; 389 } ··· 397 struct smbd_connection *info = 398 container_of(work, struct smbd_connection, 399 post_send_credits_work); 400 401 - if (info->transport_status != SMBD_CONNECTED) { 402 wake_up(&info->wait_receive_queues); 403 return; 404 } ··· 453 /* Called from softirq, when recv is done */ 454 static void recv_done(struct ib_cq *cq, struct ib_wc *wc) 455 { 456 - struct smbd_data_transfer *data_transfer; 457 struct smbd_response *response = 458 container_of(wc->wr_cqe, struct smbd_response, cqe); 459 struct smbd_connection *info = response->info; ··· 479 switch (response->type) { 480 /* SMBD negotiation response */ 481 case SMBD_NEGOTIATE_RESP: 482 - dump_smbd_negotiate_resp(smbd_response_payload(response)); 483 info->full_packet_received = true; 484 info->negotiate_done = 485 process_negotiation_response(response, wc->byte_len); ··· 536 /* Send a KEEP_ALIVE response right away if requested */ 537 info->keep_alive_requested = KEEP_ALIVE_NONE; 538 if (le16_to_cpu(data_transfer->flags) & 539 - SMB_DIRECT_RESPONSE_REQUESTED) { 540 info->keep_alive_requested = KEEP_ALIVE_PENDING; 541 } 542 ··· 640 struct smbd_connection *info, 641 struct sockaddr *dstaddr, int port) 642 { 643 int rc; 644 645 - info->id = smbd_create_id(info, dstaddr, port); 646 - if (IS_ERR(info->id)) { 647 - rc = PTR_ERR(info->id); 648 goto out1; 649 } 650 651 - if (!frwr_is_supported(&info->id->device->attrs)) { 652 log_rdma_event(ERR, "Fast Registration Work Requests (FRWR) is not supported\n"); 653 log_rdma_event(ERR, "Device capability flags = %llx max_fast_reg_page_list_len = %u\n", 654 - info->id->device->attrs.device_cap_flags, 655 - info->id->device->attrs.max_fast_reg_page_list_len); 656 rc = -EPROTONOSUPPORT; 657 goto out2; 658 } 659 info->max_frmr_depth = min_t(int, 660 smbd_max_frmr_depth, 661 - info->id->device->attrs.max_fast_reg_page_list_len); 662 info->mr_type = IB_MR_TYPE_MEM_REG; 663 - if (info->id->device->attrs.kernel_cap_flags & IBK_SG_GAPS_REG) 664 info->mr_type = IB_MR_TYPE_SG_GAPS; 665 666 - info->pd = ib_alloc_pd(info->id->device, 0); 667 - if (IS_ERR(info->pd)) { 668 - rc = PTR_ERR(info->pd); 669 log_rdma_event(ERR, "ib_alloc_pd() returned %d\n", rc); 670 goto out2; 671 } ··· 675 return 0; 676 677 out2: 678 - rdma_destroy_id(info->id); 679 - info->id = NULL; 680 681 out1: 682 return rc; ··· 690 */ 691 static int smbd_post_send_negotiate_req(struct smbd_connection *info) 692 { 693 struct ib_send_wr send_wr; 694 int rc = -ENOMEM; 695 struct smbd_request *request; 696 - struct smbd_negotiate_req *packet; 697 698 request = mempool_alloc(info->request_mempool, GFP_KERNEL); 699 if (!request) ··· 704 request->info = info; 705 706 packet = smbd_request_payload(request); 707 - packet->min_version = cpu_to_le16(SMBD_V1); 708 - packet->max_version = cpu_to_le16(SMBD_V1); 709 packet->reserved = 0; 710 - packet->credits_requested = cpu_to_le16(info->send_credit_target); 711 - packet->preferred_send_size = cpu_to_le32(info->max_send_size); 712 - packet->max_receive_size = cpu_to_le32(info->max_receive_size); 713 packet->max_fragmented_size = 714 - cpu_to_le32(info->max_fragmented_recv_size); 715 716 request->num_sge = 1; 717 request->sge[0].addr = ib_dma_map_single( 718 - info->id->device, (void *)packet, 719 sizeof(*packet), DMA_TO_DEVICE); 720 - if (ib_dma_mapping_error(info->id->device, request->sge[0].addr)) { 721 rc = -EIO; 722 goto dma_mapping_failed; 723 } 724 725 request->sge[0].length = sizeof(*packet); 726 - request->sge[0].lkey = info->pd->local_dma_lkey; 727 728 ib_dma_sync_single_for_device( 729 - info->id->device, request->sge[0].addr, 730 request->sge[0].length, DMA_TO_DEVICE); 731 732 request->cqe.done = send_done; ··· 743 request->sge[0].length, request->sge[0].lkey); 744 745 atomic_inc(&info->send_pending); 746 - rc = ib_post_send(info->id->qp, &send_wr, NULL); 747 if (!rc) 748 return 0; 749 750 /* if we reach here, post send failed */ 751 log_rdma_send(ERR, "ib_post_send failed rc=%d\n", rc); 752 atomic_dec(&info->send_pending); 753 - ib_dma_unmap_single(info->id->device, request->sge[0].addr, 754 request->sge[0].length, DMA_TO_DEVICE); 755 756 smbd_disconnect_rdma_connection(info); ··· 783 /* 784 * Check if we need to send a KEEP_ALIVE message 785 * The idle connection timer triggers a KEEP_ALIVE message when expires 786 - * SMB_DIRECT_RESPONSE_REQUESTED is set in the message flag to have peer send 787 * back a response. 788 * return value: 789 - * 1 if SMB_DIRECT_RESPONSE_REQUESTED needs to be set 790 * 0: otherwise 791 */ 792 static int manage_keep_alive_before_sending(struct smbd_connection *info) ··· 802 static int smbd_post_send(struct smbd_connection *info, 803 struct smbd_request *request) 804 { 805 struct ib_send_wr send_wr; 806 int rc, i; 807 ··· 812 "rdma_request sge[%d] addr=0x%llx length=%u\n", 813 i, request->sge[i].addr, request->sge[i].length); 814 ib_dma_sync_single_for_device( 815 - info->id->device, 816 request->sge[i].addr, 817 request->sge[i].length, 818 DMA_TO_DEVICE); ··· 827 send_wr.opcode = IB_WR_SEND; 828 send_wr.send_flags = IB_SEND_SIGNALED; 829 830 - rc = ib_post_send(info->id->qp, &send_wr, NULL); 831 if (rc) { 832 log_rdma_send(ERR, "ib_post_send failed rc=%d\n", rc); 833 smbd_disconnect_rdma_connection(info); ··· 835 } else 836 /* Reset timer for idle connection after packet is sent */ 837 mod_delayed_work(info->workqueue, &info->idle_timer_work, 838 - info->keep_alive_interval*HZ); 839 840 return rc; 841 } ··· 844 struct iov_iter *iter, 845 int *_remaining_data_length) 846 { 847 int i, rc; 848 int header_length; 849 int data_length; 850 struct smbd_request *request; 851 - struct smbd_data_transfer *packet; 852 int new_credits = 0; 853 854 wait_credit: 855 /* Wait for send credits. A SMBD packet needs one credit */ 856 rc = wait_event_interruptible(info->wait_send_queue, 857 atomic_read(&info->send_credits) > 0 || 858 - info->transport_status != SMBD_CONNECTED); 859 if (rc) 860 goto err_wait_credit; 861 862 - if (info->transport_status != SMBD_CONNECTED) { 863 log_outgoing(ERR, "disconnected not sending on wait_credit\n"); 864 rc = -EAGAIN; 865 goto err_wait_credit; ··· 873 874 wait_send_queue: 875 wait_event(info->wait_post_send, 876 - atomic_read(&info->send_pending) < info->send_credit_target || 877 - info->transport_status != SMBD_CONNECTED); 878 879 - if (info->transport_status != SMBD_CONNECTED) { 880 log_outgoing(ERR, "disconnected not sending on wait_send_queue\n"); 881 rc = -EAGAIN; 882 goto err_wait_send_queue; 883 } 884 885 if (unlikely(atomic_inc_return(&info->send_pending) > 886 - info->send_credit_target)) { 887 atomic_dec(&info->send_pending); 888 goto wait_send_queue; 889 } ··· 903 .nr_sge = 1, 904 .max_sge = SMBDIRECT_MAX_SEND_SGE, 905 .sge = request->sge, 906 - .device = info->id->device, 907 - .local_dma_lkey = info->pd->local_dma_lkey, 908 .direction = DMA_TO_DEVICE, 909 }; 910 ··· 922 923 /* Fill in the packet header */ 924 packet = smbd_request_payload(request); 925 - packet->credits_requested = cpu_to_le16(info->send_credit_target); 926 927 new_credits = manage_credits_prior_sending(info); 928 atomic_add(new_credits, &info->receive_credits); ··· 932 933 packet->flags = 0; 934 if (manage_keep_alive_before_sending(info)) 935 - packet->flags |= cpu_to_le16(SMB_DIRECT_RESPONSE_REQUESTED); 936 937 packet->reserved = 0; 938 if (!data_length) ··· 951 le32_to_cpu(packet->remaining_data_length)); 952 953 /* Map the packet to DMA */ 954 - header_length = sizeof(struct smbd_data_transfer); 955 /* If this is a packet without payload, don't send padding */ 956 if (!data_length) 957 - header_length = offsetof(struct smbd_data_transfer, padding); 958 959 - request->sge[0].addr = ib_dma_map_single(info->id->device, 960 (void *)packet, 961 header_length, 962 DMA_TO_DEVICE); 963 - if (ib_dma_mapping_error(info->id->device, request->sge[0].addr)) { 964 rc = -EIO; 965 request->sge[0].addr = 0; 966 goto err_dma; 967 } 968 969 request->sge[0].length = header_length; 970 - request->sge[0].lkey = info->pd->local_dma_lkey; 971 972 rc = smbd_post_send(info, request); 973 if (!rc) ··· 976 err_dma: 977 for (i = 0; i < request->num_sge; i++) 978 if (request->sge[i].addr) 979 - ib_dma_unmap_single(info->id->device, 980 request->sge[i].addr, 981 request->sge[i].length, 982 DMA_TO_DEVICE); ··· 1021 static int smbd_post_recv( 1022 struct smbd_connection *info, struct smbd_response *response) 1023 { 1024 struct ib_recv_wr recv_wr; 1025 int rc = -EIO; 1026 1027 response->sge.addr = ib_dma_map_single( 1028 - info->id->device, response->packet, 1029 - info->max_receive_size, DMA_FROM_DEVICE); 1030 - if (ib_dma_mapping_error(info->id->device, response->sge.addr)) 1031 return rc; 1032 1033 - response->sge.length = info->max_receive_size; 1034 - response->sge.lkey = info->pd->local_dma_lkey; 1035 1036 response->cqe.done = recv_done; 1037 ··· 1042 recv_wr.sg_list = &response->sge; 1043 recv_wr.num_sge = 1; 1044 1045 - rc = ib_post_recv(info->id->qp, &recv_wr, NULL); 1046 if (rc) { 1047 - ib_dma_unmap_single(info->id->device, response->sge.addr, 1048 response->sge.length, DMA_FROM_DEVICE); 1049 smbd_disconnect_rdma_connection(info); 1050 log_rdma_recv(ERR, "ib_post_recv failed rc=%d\n", rc); ··· 1202 static void put_receive_buffer( 1203 struct smbd_connection *info, struct smbd_response *response) 1204 { 1205 unsigned long flags; 1206 1207 - ib_dma_unmap_single(info->id->device, response->sge.addr, 1208 response->sge.length, DMA_FROM_DEVICE); 1209 1210 spin_lock_irqsave(&info->receive_queue_lock, flags); ··· 1280 struct smbd_connection *info = container_of( 1281 work, struct smbd_connection, 1282 idle_timer_work.work); 1283 1284 if (info->keep_alive_requested != KEEP_ALIVE_NONE) { 1285 log_keep_alive(ERR, ··· 1296 1297 /* Setup the next idle timeout work */ 1298 queue_delayed_work(info->workqueue, &info->idle_timer_work, 1299 - info->keep_alive_interval*HZ); 1300 } 1301 1302 /* ··· 1307 void smbd_destroy(struct TCP_Server_Info *server) 1308 { 1309 struct smbd_connection *info = server->smbd_conn; 1310 struct smbd_response *response; 1311 unsigned long flags; 1312 ··· 1316 log_rdma_event(INFO, "rdma session already destroyed\n"); 1317 return; 1318 } 1319 1320 log_rdma_event(INFO, "destroying rdma session\n"); 1321 - if (info->transport_status != SMBD_DISCONNECTED) { 1322 - rdma_disconnect(server->smbd_conn->id); 1323 log_rdma_event(INFO, "wait for transport being disconnected\n"); 1324 wait_event_interruptible( 1325 info->disconn_wait, 1326 - info->transport_status == SMBD_DISCONNECTED); 1327 } 1328 1329 log_rdma_event(INFO, "destroying qp\n"); 1330 - ib_drain_qp(info->id->qp); 1331 - rdma_destroy_qp(info->id); 1332 1333 log_rdma_event(INFO, "cancelling idle timer\n"); 1334 cancel_delayed_work_sync(&info->idle_timer_work); ··· 1359 log_rdma_event(INFO, "free receive buffers\n"); 1360 wait_event(info->wait_receive_queues, 1361 info->count_receive_queue + info->count_empty_packet_queue 1362 - == info->receive_credit_max); 1363 destroy_receive_buffers(info); 1364 1365 /* ··· 1378 } 1379 destroy_mr_list(info); 1380 1381 - ib_free_cq(info->send_cq); 1382 - ib_free_cq(info->recv_cq); 1383 - ib_dealloc_pd(info->pd); 1384 - rdma_destroy_id(info->id); 1385 1386 /* free mempools */ 1387 mempool_destroy(info->request_mempool); ··· 1390 mempool_destroy(info->response_mempool); 1391 kmem_cache_destroy(info->response_cache); 1392 1393 - info->transport_status = SMBD_DESTROYED; 1394 1395 destroy_workqueue(info->workqueue); 1396 log_rdma_event(INFO, "rdma session destroyed\n"); ··· 1415 * This is possible if transport is disconnected and we haven't received 1416 * notification from RDMA, but upper layer has detected timeout 1417 */ 1418 - if (server->smbd_conn->transport_status == SMBD_CONNECTED) { 1419 log_rdma_event(INFO, "disconnecting transport\n"); 1420 smbd_destroy(server); 1421 } ··· 1447 #define MAX_NAME_LEN 80 1448 static int allocate_caches_and_workqueue(struct smbd_connection *info) 1449 { 1450 char name[MAX_NAME_LEN]; 1451 int rc; 1452 ··· 1457 kmem_cache_create( 1458 name, 1459 sizeof(struct smbd_request) + 1460 - sizeof(struct smbd_data_transfer), 1461 0, SLAB_HWCACHE_ALIGN, NULL); 1462 if (!info->request_cache) 1463 return -ENOMEM; 1464 1465 info->request_mempool = 1466 - mempool_create(info->send_credit_target, mempool_alloc_slab, 1467 mempool_free_slab, info->request_cache); 1468 if (!info->request_mempool) 1469 goto out1; ··· 1473 kmem_cache_create( 1474 name, 1475 sizeof(struct smbd_response) + 1476 - info->max_receive_size, 1477 0, SLAB_HWCACHE_ALIGN, NULL); 1478 if (!info->response_cache) 1479 goto out2; 1480 1481 info->response_mempool = 1482 - mempool_create(info->receive_credit_max, mempool_alloc_slab, 1483 mempool_free_slab, info->response_cache); 1484 if (!info->response_mempool) 1485 goto out3; ··· 1489 if (!info->workqueue) 1490 goto out4; 1491 1492 - rc = allocate_receive_buffers(info, info->receive_credit_max); 1493 if (rc) { 1494 log_rdma_event(ERR, "failed to allocate receive buffers\n"); 1495 goto out5; ··· 1516 { 1517 int rc; 1518 struct smbd_connection *info; 1519 struct rdma_conn_param conn_param; 1520 struct ib_qp_init_attr qp_attr; 1521 struct sockaddr_in *addr_in = (struct sockaddr_in *) dstaddr; ··· 1527 info = kzalloc(sizeof(struct smbd_connection), GFP_KERNEL); 1528 if (!info) 1529 return NULL; 1530 1531 - info->transport_status = SMBD_CONNECTING; 1532 rc = smbd_ia_open(info, dstaddr, port); 1533 if (rc) { 1534 log_rdma_event(INFO, "smbd_ia_open rc=%d\n", rc); 1535 goto create_id_failed; 1536 } 1537 1538 - if (smbd_send_credit_target > info->id->device->attrs.max_cqe || 1539 - smbd_send_credit_target > info->id->device->attrs.max_qp_wr) { 1540 log_rdma_event(ERR, "consider lowering send_credit_target = %d. Possible CQE overrun, device reporting max_cqe %d max_qp_wr %d\n", 1541 smbd_send_credit_target, 1542 - info->id->device->attrs.max_cqe, 1543 - info->id->device->attrs.max_qp_wr); 1544 goto config_failed; 1545 } 1546 1547 - if (smbd_receive_credit_max > info->id->device->attrs.max_cqe || 1548 - smbd_receive_credit_max > info->id->device->attrs.max_qp_wr) { 1549 log_rdma_event(ERR, "consider lowering receive_credit_max = %d. Possible CQE overrun, device reporting max_cqe %d max_qp_wr %d\n", 1550 smbd_receive_credit_max, 1551 - info->id->device->attrs.max_cqe, 1552 - info->id->device->attrs.max_qp_wr); 1553 goto config_failed; 1554 } 1555 1556 - info->receive_credit_max = smbd_receive_credit_max; 1557 - info->send_credit_target = smbd_send_credit_target; 1558 - info->max_send_size = smbd_max_send_size; 1559 - info->max_fragmented_recv_size = smbd_max_fragmented_recv_size; 1560 - info->max_receive_size = smbd_max_receive_size; 1561 - info->keep_alive_interval = smbd_keep_alive_interval; 1562 1563 - if (info->id->device->attrs.max_send_sge < SMBDIRECT_MAX_SEND_SGE || 1564 - info->id->device->attrs.max_recv_sge < SMBDIRECT_MAX_RECV_SGE) { 1565 log_rdma_event(ERR, 1566 "device %.*s max_send_sge/max_recv_sge = %d/%d too small\n", 1567 IB_DEVICE_NAME_MAX, 1568 - info->id->device->name, 1569 - info->id->device->attrs.max_send_sge, 1570 - info->id->device->attrs.max_recv_sge); 1571 goto config_failed; 1572 } 1573 1574 - info->send_cq = NULL; 1575 - info->recv_cq = NULL; 1576 - info->send_cq = 1577 - ib_alloc_cq_any(info->id->device, info, 1578 - info->send_credit_target, IB_POLL_SOFTIRQ); 1579 - if (IS_ERR(info->send_cq)) { 1580 - info->send_cq = NULL; 1581 goto alloc_cq_failed; 1582 } 1583 1584 - info->recv_cq = 1585 - ib_alloc_cq_any(info->id->device, info, 1586 - info->receive_credit_max, IB_POLL_SOFTIRQ); 1587 - if (IS_ERR(info->recv_cq)) { 1588 - info->recv_cq = NULL; 1589 goto alloc_cq_failed; 1590 } 1591 1592 memset(&qp_attr, 0, sizeof(qp_attr)); 1593 qp_attr.event_handler = smbd_qp_async_error_upcall; 1594 qp_attr.qp_context = info; 1595 - qp_attr.cap.max_send_wr = info->send_credit_target; 1596 - qp_attr.cap.max_recv_wr = info->receive_credit_max; 1597 qp_attr.cap.max_send_sge = SMBDIRECT_MAX_SEND_SGE; 1598 qp_attr.cap.max_recv_sge = SMBDIRECT_MAX_RECV_SGE; 1599 qp_attr.cap.max_inline_data = 0; 1600 qp_attr.sq_sig_type = IB_SIGNAL_REQ_WR; 1601 qp_attr.qp_type = IB_QPT_RC; 1602 - qp_attr.send_cq = info->send_cq; 1603 - qp_attr.recv_cq = info->recv_cq; 1604 qp_attr.port_num = ~0; 1605 1606 - rc = rdma_create_qp(info->id, info->pd, &qp_attr); 1607 if (rc) { 1608 log_rdma_event(ERR, "rdma_create_qp failed %i\n", rc); 1609 goto create_qp_failed; 1610 } 1611 1612 memset(&conn_param, 0, sizeof(conn_param)); 1613 conn_param.initiator_depth = 0; 1614 1615 conn_param.responder_resources = 1616 - min(info->id->device->attrs.max_qp_rd_atom, 1617 SMBD_CM_RESPONDER_RESOURCES); 1618 info->responder_resources = conn_param.responder_resources; 1619 log_rdma_mr(INFO, "responder_resources=%d\n", 1620 info->responder_resources); 1621 1622 /* Need to send IRD/ORD in private data for iWARP */ 1623 - info->id->device->ops.get_port_immutable( 1624 - info->id->device, info->id->port_num, &port_immutable); 1625 if (port_immutable.core_cap_flags & RDMA_CORE_PORT_IWARP) { 1626 ird_ord_hdr[0] = info->responder_resources; 1627 ird_ord_hdr[1] = 1; ··· 1643 init_waitqueue_head(&info->conn_wait); 1644 init_waitqueue_head(&info->disconn_wait); 1645 init_waitqueue_head(&info->wait_reassembly_queue); 1646 - rc = rdma_connect(info->id, &conn_param); 1647 if (rc) { 1648 log_rdma_event(ERR, "rdma_connect() failed with %i\n", rc); 1649 goto rdma_connect_failed; 1650 } 1651 1652 wait_event_interruptible( 1653 - info->conn_wait, info->transport_status != SMBD_CONNECTING); 1654 1655 - if (info->transport_status != SMBD_CONNECTED) { 1656 log_rdma_event(ERR, "rdma_connect failed port=%d\n", port); 1657 goto rdma_connect_failed; 1658 } ··· 1668 init_waitqueue_head(&info->wait_send_queue); 1669 INIT_DELAYED_WORK(&info->idle_timer_work, idle_connection_timer); 1670 queue_delayed_work(info->workqueue, &info->idle_timer_work, 1671 - info->keep_alive_interval*HZ); 1672 1673 init_waitqueue_head(&info->wait_send_pending); 1674 atomic_set(&info->send_pending, 0); ··· 1703 negotiation_failed: 1704 cancel_delayed_work_sync(&info->idle_timer_work); 1705 destroy_caches_and_workqueue(info); 1706 - info->transport_status = SMBD_NEGOTIATE_FAILED; 1707 init_waitqueue_head(&info->conn_wait); 1708 - rdma_disconnect(info->id); 1709 wait_event(info->conn_wait, 1710 - info->transport_status == SMBD_DISCONNECTED); 1711 1712 allocate_cache_failed: 1713 rdma_connect_failed: 1714 - rdma_destroy_qp(info->id); 1715 1716 create_qp_failed: 1717 alloc_cq_failed: 1718 - if (info->send_cq) 1719 - ib_free_cq(info->send_cq); 1720 - if (info->recv_cq) 1721 - ib_free_cq(info->recv_cq); 1722 1723 config_failed: 1724 - ib_dealloc_pd(info->pd); 1725 - rdma_destroy_id(info->id); 1726 1727 create_id_failed: 1728 kfree(info); ··· 1762 static int smbd_recv_buf(struct smbd_connection *info, char *buf, 1763 unsigned int size) 1764 { 1765 struct smbd_response *response; 1766 - struct smbd_data_transfer *data_transfer; 1767 int to_copy, to_read, data_read, offset; 1768 u32 data_length, remaining_data_length, data_offset; 1769 int rc; ··· 1877 rc = wait_event_interruptible( 1878 info->wait_reassembly_queue, 1879 info->reassembly_data_length >= size || 1880 - info->transport_status != SMBD_CONNECTED); 1881 /* Don't return any data if interrupted */ 1882 if (rc) 1883 return rc; 1884 1885 - if (info->transport_status != SMBD_CONNECTED) { 1886 log_read(ERR, "disconnected\n"); 1887 return -ECONNABORTED; 1888 } ··· 1900 struct page *page, unsigned int page_offset, 1901 unsigned int to_read) 1902 { 1903 int ret; 1904 char *to_address; 1905 void *page_address; ··· 1909 ret = wait_event_interruptible( 1910 info->wait_reassembly_queue, 1911 info->reassembly_data_length >= to_read || 1912 - info->transport_status != SMBD_CONNECTED); 1913 if (ret) 1914 return ret; 1915 ··· 1984 int num_rqst, struct smb_rqst *rqst_array) 1985 { 1986 struct smbd_connection *info = server->smbd_conn; 1987 struct smb_rqst *rqst; 1988 struct iov_iter iter; 1989 unsigned int remaining_data_length, klen; 1990 int rc, i, rqst_idx; 1991 1992 - if (info->transport_status != SMBD_CONNECTED) 1993 return -EAGAIN; 1994 1995 /* ··· 2003 for (i = 0; i < num_rqst; i++) 2004 remaining_data_length += smb_rqst_len(server, &rqst_array[i]); 2005 2006 - if (unlikely(remaining_data_length > info->max_fragmented_send_size)) { 2007 /* assertion: payload never exceeds negotiated maximum */ 2008 log_write(ERR, "payload size %d > max size %d\n", 2009 - remaining_data_length, info->max_fragmented_send_size); 2010 return -EINVAL; 2011 } 2012 ··· 2085 { 2086 struct smbd_connection *info = 2087 container_of(work, struct smbd_connection, mr_recovery_work); 2088 struct smbd_mr *smbdirect_mr; 2089 int rc; 2090 ··· 2103 } 2104 2105 smbdirect_mr->mr = ib_alloc_mr( 2106 - info->pd, info->mr_type, 2107 info->max_frmr_depth); 2108 if (IS_ERR(smbdirect_mr->mr)) { 2109 log_rdma_mr(ERR, "ib_alloc_mr failed mr_type=%x max_frmr_depth=%x\n", ··· 2132 2133 static void destroy_mr_list(struct smbd_connection *info) 2134 { 2135 struct smbd_mr *mr, *tmp; 2136 2137 cancel_work_sync(&info->mr_recovery_work); 2138 list_for_each_entry_safe(mr, tmp, &info->mr_list, list) { 2139 if (mr->state == MR_INVALIDATED) 2140 - ib_dma_unmap_sg(info->id->device, mr->sgt.sgl, 2141 mr->sgt.nents, mr->dir); 2142 ib_dereg_mr(mr->mr); 2143 kfree(mr->sgt.sgl); ··· 2155 */ 2156 static int allocate_mr_list(struct smbd_connection *info) 2157 { 2158 int i; 2159 struct smbd_mr *smbdirect_mr, *tmp; 2160 ··· 2171 smbdirect_mr = kzalloc(sizeof(*smbdirect_mr), GFP_KERNEL); 2172 if (!smbdirect_mr) 2173 goto cleanup_entries; 2174 - smbdirect_mr->mr = ib_alloc_mr(info->pd, info->mr_type, 2175 info->max_frmr_depth); 2176 if (IS_ERR(smbdirect_mr->mr)) { 2177 log_rdma_mr(ERR, "ib_alloc_mr failed mr_type=%x max_frmr_depth=%x\n", ··· 2216 */ 2217 static struct smbd_mr *get_mr(struct smbd_connection *info) 2218 { 2219 struct smbd_mr *ret; 2220 int rc; 2221 again: 2222 rc = wait_event_interruptible(info->wait_mr, 2223 atomic_read(&info->mr_ready_count) || 2224 - info->transport_status != SMBD_CONNECTED); 2225 if (rc) { 2226 log_rdma_mr(ERR, "wait_event_interruptible rc=%x\n", rc); 2227 return NULL; 2228 } 2229 2230 - if (info->transport_status != SMBD_CONNECTED) { 2231 - log_rdma_mr(ERR, "info->transport_status=%x\n", 2232 - info->transport_status); 2233 return NULL; 2234 } 2235 ··· 2282 struct iov_iter *iter, 2283 bool writing, bool need_invalidate) 2284 { 2285 struct smbd_mr *smbdirect_mr; 2286 int rc, num_pages; 2287 enum dma_data_direction dir; ··· 2312 num_pages, iov_iter_count(iter), info->max_frmr_depth); 2313 smbd_iter_to_mr(info, iter, &smbdirect_mr->sgt, info->max_frmr_depth); 2314 2315 - rc = ib_dma_map_sg(info->id->device, smbdirect_mr->sgt.sgl, 2316 smbdirect_mr->sgt.nents, dir); 2317 if (!rc) { 2318 log_rdma_mr(ERR, "ib_dma_map_sg num_pages=%x dir=%x rc=%x\n", ··· 2348 * on IB_WR_REG_MR. Hardware enforces a barrier and order of execution 2349 * on the next ib_post_send when we actually send I/O to remote peer 2350 */ 2351 - rc = ib_post_send(info->id->qp, &reg_wr->wr, NULL); 2352 if (!rc) 2353 return smbdirect_mr; 2354 ··· 2357 2358 /* If all failed, attempt to recover this MR by setting it MR_ERROR*/ 2359 map_mr_error: 2360 - ib_dma_unmap_sg(info->id->device, smbdirect_mr->sgt.sgl, 2361 smbdirect_mr->sgt.nents, smbdirect_mr->dir); 2362 2363 dma_map_error: ··· 2395 { 2396 struct ib_send_wr *wr; 2397 struct smbd_connection *info = smbdirect_mr->conn; 2398 int rc = 0; 2399 2400 if (smbdirect_mr->need_invalidate) { ··· 2409 wr->send_flags = IB_SEND_SIGNALED; 2410 2411 init_completion(&smbdirect_mr->invalidate_done); 2412 - rc = ib_post_send(info->id->qp, wr, NULL); 2413 if (rc) { 2414 log_rdma_mr(ERR, "ib_post_send failed rc=%x\n", rc); 2415 smbd_disconnect_rdma_connection(info); ··· 2426 2427 if (smbdirect_mr->state == MR_INVALIDATED) { 2428 ib_dma_unmap_sg( 2429 - info->id->device, smbdirect_mr->sgt.sgl, 2430 smbdirect_mr->sgt.nents, 2431 smbdirect_mr->dir); 2432 smbdirect_mr->state = MR_READY;
··· 7 #include <linux/module.h> 8 #include <linux/highmem.h> 9 #include <linux/folio_queue.h> 10 + #include "../common/smbdirect/smbdirect_pdu.h" 11 #include "smbdirect.h" 12 #include "cifs_debug.h" 13 #include "cifsproto.h" ··· 49 }; 50 static ssize_t smb_extract_iter_to_rdma(struct iov_iter *iter, size_t len, 51 struct smb_extract_to_rdma *rdma); 52 53 /* Port numbers for SMBD transport */ 54 #define SMB_PORT 445 ··· 165 { 166 struct smbd_connection *info = 167 container_of(work, struct smbd_connection, disconnect_work); 168 + struct smbdirect_socket *sc = &info->socket; 169 170 + if (sc->status == SMBDIRECT_SOCKET_CONNECTED) { 171 + sc->status = SMBDIRECT_SOCKET_DISCONNECTING; 172 + rdma_disconnect(sc->rdma.cm_id); 173 } 174 } 175 ··· 182 struct rdma_cm_id *id, struct rdma_cm_event *event) 183 { 184 struct smbd_connection *info = id->context; 185 + struct smbdirect_socket *sc = &info->socket; 186 187 log_rdma_event(INFO, "event=%d status=%d\n", 188 event->event, event->status); ··· 205 206 case RDMA_CM_EVENT_ESTABLISHED: 207 log_rdma_event(INFO, "connected event=%d\n", event->event); 208 + sc->status = SMBDIRECT_SOCKET_CONNECTED; 209 wake_up_interruptible(&info->conn_wait); 210 break; 211 ··· 213 case RDMA_CM_EVENT_UNREACHABLE: 214 case RDMA_CM_EVENT_REJECTED: 215 log_rdma_event(INFO, "connecting failed event=%d\n", event->event); 216 + sc->status = SMBDIRECT_SOCKET_DISCONNECTED; 217 wake_up_interruptible(&info->conn_wait); 218 break; 219 220 case RDMA_CM_EVENT_DEVICE_REMOVAL: 221 case RDMA_CM_EVENT_DISCONNECTED: 222 /* This happens when we fail the negotiation */ 223 + if (sc->status == SMBDIRECT_SOCKET_NEGOTIATE_FAILED) { 224 + sc->status = SMBDIRECT_SOCKET_DISCONNECTED; 225 wake_up(&info->conn_wait); 226 break; 227 } 228 229 + sc->status = SMBDIRECT_SOCKET_DISCONNECTED; 230 wake_up_interruptible(&info->disconn_wait); 231 wake_up_interruptible(&info->wait_reassembly_queue); 232 wake_up_interruptible_all(&info->wait_send_queue); ··· 275 int i; 276 struct smbd_request *request = 277 container_of(wc->wr_cqe, struct smbd_request, cqe); 278 + struct smbd_connection *info = request->info; 279 + struct smbdirect_socket *sc = &info->socket; 280 281 log_rdma_send(INFO, "smbd_request 0x%p completed wc->status=%d\n", 282 request, wc->status); ··· 286 } 287 288 for (i = 0; i < request->num_sge; i++) 289 + ib_dma_unmap_single(sc->ib.dev, 290 request->sge[i].addr, 291 request->sge[i].length, 292 DMA_TO_DEVICE); ··· 299 mempool_free(request, request->info->request_mempool); 300 } 301 302 + static void dump_smbdirect_negotiate_resp(struct smbdirect_negotiate_resp *resp) 303 { 304 log_rdma_event(INFO, "resp message min_version %u max_version %u negotiated_version %u credits_requested %u credits_granted %u status %u max_readwrite_size %u preferred_send_size %u max_receive_size %u max_fragmented_size %u\n", 305 resp->min_version, resp->max_version, ··· 318 struct smbd_response *response, int packet_length) 319 { 320 struct smbd_connection *info = response->info; 321 + struct smbdirect_socket *sc = &info->socket; 322 + struct smbdirect_socket_parameters *sp = &sc->parameters; 323 + struct smbdirect_negotiate_resp *packet = smbd_response_payload(response); 324 325 + if (packet_length < sizeof(struct smbdirect_negotiate_resp)) { 326 log_rdma_event(ERR, 327 "error: packet_length=%d\n", packet_length); 328 return false; 329 } 330 331 + if (le16_to_cpu(packet->negotiated_version) != SMBDIRECT_V1) { 332 log_rdma_event(ERR, "error: negotiated_version=%x\n", 333 le16_to_cpu(packet->negotiated_version)); 334 return false; ··· 347 348 atomic_set(&info->receive_credits, 0); 349 350 + if (le32_to_cpu(packet->preferred_send_size) > sp->max_recv_size) { 351 log_rdma_event(ERR, "error: preferred_send_size=%d\n", 352 le32_to_cpu(packet->preferred_send_size)); 353 return false; 354 } 355 + sp->max_recv_size = le32_to_cpu(packet->preferred_send_size); 356 357 if (le32_to_cpu(packet->max_receive_size) < SMBD_MIN_RECEIVE_SIZE) { 358 log_rdma_event(ERR, "error: max_receive_size=%d\n", 359 le32_to_cpu(packet->max_receive_size)); 360 return false; 361 } 362 + sp->max_send_size = min_t(u32, sp->max_send_size, 363 + le32_to_cpu(packet->max_receive_size)); 364 365 if (le32_to_cpu(packet->max_fragmented_size) < 366 SMBD_MIN_FRAGMENTED_SIZE) { ··· 368 le32_to_cpu(packet->max_fragmented_size)); 369 return false; 370 } 371 + sp->max_fragmented_send_size = 372 le32_to_cpu(packet->max_fragmented_size); 373 info->rdma_readwrite_threshold = 374 + rdma_readwrite_threshold > sp->max_fragmented_send_size ? 375 + sp->max_fragmented_send_size : 376 rdma_readwrite_threshold; 377 378 379 + sp->max_read_write_size = min_t(u32, 380 le32_to_cpu(packet->max_readwrite_size), 381 info->max_frmr_depth * PAGE_SIZE); 382 + info->max_frmr_depth = sp->max_read_write_size / PAGE_SIZE; 383 384 return true; 385 } ··· 393 struct smbd_connection *info = 394 container_of(work, struct smbd_connection, 395 post_send_credits_work); 396 + struct smbdirect_socket *sc = &info->socket; 397 398 + if (sc->status != SMBDIRECT_SOCKET_CONNECTED) { 399 wake_up(&info->wait_receive_queues); 400 return; 401 } ··· 448 /* Called from softirq, when recv is done */ 449 static void recv_done(struct ib_cq *cq, struct ib_wc *wc) 450 { 451 + struct smbdirect_data_transfer *data_transfer; 452 struct smbd_response *response = 453 container_of(wc->wr_cqe, struct smbd_response, cqe); 454 struct smbd_connection *info = response->info; ··· 474 switch (response->type) { 475 /* SMBD negotiation response */ 476 case SMBD_NEGOTIATE_RESP: 477 + dump_smbdirect_negotiate_resp(smbd_response_payload(response)); 478 info->full_packet_received = true; 479 info->negotiate_done = 480 process_negotiation_response(response, wc->byte_len); ··· 531 /* Send a KEEP_ALIVE response right away if requested */ 532 info->keep_alive_requested = KEEP_ALIVE_NONE; 533 if (le16_to_cpu(data_transfer->flags) & 534 + SMBDIRECT_FLAG_RESPONSE_REQUESTED) { 535 info->keep_alive_requested = KEEP_ALIVE_PENDING; 536 } 537 ··· 635 struct smbd_connection *info, 636 struct sockaddr *dstaddr, int port) 637 { 638 + struct smbdirect_socket *sc = &info->socket; 639 int rc; 640 641 + sc->rdma.cm_id = smbd_create_id(info, dstaddr, port); 642 + if (IS_ERR(sc->rdma.cm_id)) { 643 + rc = PTR_ERR(sc->rdma.cm_id); 644 goto out1; 645 } 646 + sc->ib.dev = sc->rdma.cm_id->device; 647 648 + if (!frwr_is_supported(&sc->ib.dev->attrs)) { 649 log_rdma_event(ERR, "Fast Registration Work Requests (FRWR) is not supported\n"); 650 log_rdma_event(ERR, "Device capability flags = %llx max_fast_reg_page_list_len = %u\n", 651 + sc->ib.dev->attrs.device_cap_flags, 652 + sc->ib.dev->attrs.max_fast_reg_page_list_len); 653 rc = -EPROTONOSUPPORT; 654 goto out2; 655 } 656 info->max_frmr_depth = min_t(int, 657 smbd_max_frmr_depth, 658 + sc->ib.dev->attrs.max_fast_reg_page_list_len); 659 info->mr_type = IB_MR_TYPE_MEM_REG; 660 + if (sc->ib.dev->attrs.kernel_cap_flags & IBK_SG_GAPS_REG) 661 info->mr_type = IB_MR_TYPE_SG_GAPS; 662 663 + sc->ib.pd = ib_alloc_pd(sc->ib.dev, 0); 664 + if (IS_ERR(sc->ib.pd)) { 665 + rc = PTR_ERR(sc->ib.pd); 666 log_rdma_event(ERR, "ib_alloc_pd() returned %d\n", rc); 667 goto out2; 668 } ··· 668 return 0; 669 670 out2: 671 + rdma_destroy_id(sc->rdma.cm_id); 672 + sc->rdma.cm_id = NULL; 673 674 out1: 675 return rc; ··· 683 */ 684 static int smbd_post_send_negotiate_req(struct smbd_connection *info) 685 { 686 + struct smbdirect_socket *sc = &info->socket; 687 + struct smbdirect_socket_parameters *sp = &sc->parameters; 688 struct ib_send_wr send_wr; 689 int rc = -ENOMEM; 690 struct smbd_request *request; 691 + struct smbdirect_negotiate_req *packet; 692 693 request = mempool_alloc(info->request_mempool, GFP_KERNEL); 694 if (!request) ··· 695 request->info = info; 696 697 packet = smbd_request_payload(request); 698 + packet->min_version = cpu_to_le16(SMBDIRECT_V1); 699 + packet->max_version = cpu_to_le16(SMBDIRECT_V1); 700 packet->reserved = 0; 701 + packet->credits_requested = cpu_to_le16(sp->send_credit_target); 702 + packet->preferred_send_size = cpu_to_le32(sp->max_send_size); 703 + packet->max_receive_size = cpu_to_le32(sp->max_recv_size); 704 packet->max_fragmented_size = 705 + cpu_to_le32(sp->max_fragmented_recv_size); 706 707 request->num_sge = 1; 708 request->sge[0].addr = ib_dma_map_single( 709 + sc->ib.dev, (void *)packet, 710 sizeof(*packet), DMA_TO_DEVICE); 711 + if (ib_dma_mapping_error(sc->ib.dev, request->sge[0].addr)) { 712 rc = -EIO; 713 goto dma_mapping_failed; 714 } 715 716 request->sge[0].length = sizeof(*packet); 717 + request->sge[0].lkey = sc->ib.pd->local_dma_lkey; 718 719 ib_dma_sync_single_for_device( 720 + sc->ib.dev, request->sge[0].addr, 721 request->sge[0].length, DMA_TO_DEVICE); 722 723 request->cqe.done = send_done; ··· 734 request->sge[0].length, request->sge[0].lkey); 735 736 atomic_inc(&info->send_pending); 737 + rc = ib_post_send(sc->ib.qp, &send_wr, NULL); 738 if (!rc) 739 return 0; 740 741 /* if we reach here, post send failed */ 742 log_rdma_send(ERR, "ib_post_send failed rc=%d\n", rc); 743 atomic_dec(&info->send_pending); 744 + ib_dma_unmap_single(sc->ib.dev, request->sge[0].addr, 745 request->sge[0].length, DMA_TO_DEVICE); 746 747 smbd_disconnect_rdma_connection(info); ··· 774 /* 775 * Check if we need to send a KEEP_ALIVE message 776 * The idle connection timer triggers a KEEP_ALIVE message when expires 777 + * SMBDIRECT_FLAG_RESPONSE_REQUESTED is set in the message flag to have peer send 778 * back a response. 779 * return value: 780 + * 1 if SMBDIRECT_FLAG_RESPONSE_REQUESTED needs to be set 781 * 0: otherwise 782 */ 783 static int manage_keep_alive_before_sending(struct smbd_connection *info) ··· 793 static int smbd_post_send(struct smbd_connection *info, 794 struct smbd_request *request) 795 { 796 + struct smbdirect_socket *sc = &info->socket; 797 + struct smbdirect_socket_parameters *sp = &sc->parameters; 798 struct ib_send_wr send_wr; 799 int rc, i; 800 ··· 801 "rdma_request sge[%d] addr=0x%llx length=%u\n", 802 i, request->sge[i].addr, request->sge[i].length); 803 ib_dma_sync_single_for_device( 804 + sc->ib.dev, 805 request->sge[i].addr, 806 request->sge[i].length, 807 DMA_TO_DEVICE); ··· 816 send_wr.opcode = IB_WR_SEND; 817 send_wr.send_flags = IB_SEND_SIGNALED; 818 819 + rc = ib_post_send(sc->ib.qp, &send_wr, NULL); 820 if (rc) { 821 log_rdma_send(ERR, "ib_post_send failed rc=%d\n", rc); 822 smbd_disconnect_rdma_connection(info); ··· 824 } else 825 /* Reset timer for idle connection after packet is sent */ 826 mod_delayed_work(info->workqueue, &info->idle_timer_work, 827 + msecs_to_jiffies(sp->keepalive_interval_msec)); 828 829 return rc; 830 } ··· 833 struct iov_iter *iter, 834 int *_remaining_data_length) 835 { 836 + struct smbdirect_socket *sc = &info->socket; 837 + struct smbdirect_socket_parameters *sp = &sc->parameters; 838 int i, rc; 839 int header_length; 840 int data_length; 841 struct smbd_request *request; 842 + struct smbdirect_data_transfer *packet; 843 int new_credits = 0; 844 845 wait_credit: 846 /* Wait for send credits. A SMBD packet needs one credit */ 847 rc = wait_event_interruptible(info->wait_send_queue, 848 atomic_read(&info->send_credits) > 0 || 849 + sc->status != SMBDIRECT_SOCKET_CONNECTED); 850 if (rc) 851 goto err_wait_credit; 852 853 + if (sc->status != SMBDIRECT_SOCKET_CONNECTED) { 854 log_outgoing(ERR, "disconnected not sending on wait_credit\n"); 855 rc = -EAGAIN; 856 goto err_wait_credit; ··· 860 861 wait_send_queue: 862 wait_event(info->wait_post_send, 863 + atomic_read(&info->send_pending) < sp->send_credit_target || 864 + sc->status != SMBDIRECT_SOCKET_CONNECTED); 865 866 + if (sc->status != SMBDIRECT_SOCKET_CONNECTED) { 867 log_outgoing(ERR, "disconnected not sending on wait_send_queue\n"); 868 rc = -EAGAIN; 869 goto err_wait_send_queue; 870 } 871 872 if (unlikely(atomic_inc_return(&info->send_pending) > 873 + sp->send_credit_target)) { 874 atomic_dec(&info->send_pending); 875 goto wait_send_queue; 876 } ··· 890 .nr_sge = 1, 891 .max_sge = SMBDIRECT_MAX_SEND_SGE, 892 .sge = request->sge, 893 + .device = sc->ib.dev, 894 + .local_dma_lkey = sc->ib.pd->local_dma_lkey, 895 .direction = DMA_TO_DEVICE, 896 }; 897 ··· 909 910 /* Fill in the packet header */ 911 packet = smbd_request_payload(request); 912 + packet->credits_requested = cpu_to_le16(sp->send_credit_target); 913 914 new_credits = manage_credits_prior_sending(info); 915 atomic_add(new_credits, &info->receive_credits); ··· 919 920 packet->flags = 0; 921 if (manage_keep_alive_before_sending(info)) 922 + packet->flags |= cpu_to_le16(SMBDIRECT_FLAG_RESPONSE_REQUESTED); 923 924 packet->reserved = 0; 925 if (!data_length) ··· 938 le32_to_cpu(packet->remaining_data_length)); 939 940 /* Map the packet to DMA */ 941 + header_length = sizeof(struct smbdirect_data_transfer); 942 /* If this is a packet without payload, don't send padding */ 943 if (!data_length) 944 + header_length = offsetof(struct smbdirect_data_transfer, padding); 945 946 + request->sge[0].addr = ib_dma_map_single(sc->ib.dev, 947 (void *)packet, 948 header_length, 949 DMA_TO_DEVICE); 950 + if (ib_dma_mapping_error(sc->ib.dev, request->sge[0].addr)) { 951 rc = -EIO; 952 request->sge[0].addr = 0; 953 goto err_dma; 954 } 955 956 request->sge[0].length = header_length; 957 + request->sge[0].lkey = sc->ib.pd->local_dma_lkey; 958 959 rc = smbd_post_send(info, request); 960 if (!rc) ··· 963 err_dma: 964 for (i = 0; i < request->num_sge; i++) 965 if (request->sge[i].addr) 966 + ib_dma_unmap_single(sc->ib.dev, 967 request->sge[i].addr, 968 request->sge[i].length, 969 DMA_TO_DEVICE); ··· 1008 static int smbd_post_recv( 1009 struct smbd_connection *info, struct smbd_response *response) 1010 { 1011 + struct smbdirect_socket *sc = &info->socket; 1012 + struct smbdirect_socket_parameters *sp = &sc->parameters; 1013 struct ib_recv_wr recv_wr; 1014 int rc = -EIO; 1015 1016 response->sge.addr = ib_dma_map_single( 1017 + sc->ib.dev, response->packet, 1018 + sp->max_recv_size, DMA_FROM_DEVICE); 1019 + if (ib_dma_mapping_error(sc->ib.dev, response->sge.addr)) 1020 return rc; 1021 1022 + response->sge.length = sp->max_recv_size; 1023 + response->sge.lkey = sc->ib.pd->local_dma_lkey; 1024 1025 response->cqe.done = recv_done; 1026 ··· 1027 recv_wr.sg_list = &response->sge; 1028 recv_wr.num_sge = 1; 1029 1030 + rc = ib_post_recv(sc->ib.qp, &recv_wr, NULL); 1031 if (rc) { 1032 + ib_dma_unmap_single(sc->ib.dev, response->sge.addr, 1033 response->sge.length, DMA_FROM_DEVICE); 1034 smbd_disconnect_rdma_connection(info); 1035 log_rdma_recv(ERR, "ib_post_recv failed rc=%d\n", rc); ··· 1187 static void put_receive_buffer( 1188 struct smbd_connection *info, struct smbd_response *response) 1189 { 1190 + struct smbdirect_socket *sc = &info->socket; 1191 unsigned long flags; 1192 1193 + ib_dma_unmap_single(sc->ib.dev, response->sge.addr, 1194 response->sge.length, DMA_FROM_DEVICE); 1195 1196 spin_lock_irqsave(&info->receive_queue_lock, flags); ··· 1264 struct smbd_connection *info = container_of( 1265 work, struct smbd_connection, 1266 idle_timer_work.work); 1267 + struct smbdirect_socket *sc = &info->socket; 1268 + struct smbdirect_socket_parameters *sp = &sc->parameters; 1269 1270 if (info->keep_alive_requested != KEEP_ALIVE_NONE) { 1271 log_keep_alive(ERR, ··· 1278 1279 /* Setup the next idle timeout work */ 1280 queue_delayed_work(info->workqueue, &info->idle_timer_work, 1281 + msecs_to_jiffies(sp->keepalive_interval_msec)); 1282 } 1283 1284 /* ··· 1289 void smbd_destroy(struct TCP_Server_Info *server) 1290 { 1291 struct smbd_connection *info = server->smbd_conn; 1292 + struct smbdirect_socket *sc; 1293 + struct smbdirect_socket_parameters *sp; 1294 struct smbd_response *response; 1295 unsigned long flags; 1296 ··· 1296 log_rdma_event(INFO, "rdma session already destroyed\n"); 1297 return; 1298 } 1299 + sc = &info->socket; 1300 + sp = &sc->parameters; 1301 1302 log_rdma_event(INFO, "destroying rdma session\n"); 1303 + if (sc->status != SMBDIRECT_SOCKET_DISCONNECTED) { 1304 + rdma_disconnect(sc->rdma.cm_id); 1305 log_rdma_event(INFO, "wait for transport being disconnected\n"); 1306 wait_event_interruptible( 1307 info->disconn_wait, 1308 + sc->status == SMBDIRECT_SOCKET_DISCONNECTED); 1309 } 1310 1311 log_rdma_event(INFO, "destroying qp\n"); 1312 + ib_drain_qp(sc->ib.qp); 1313 + rdma_destroy_qp(sc->rdma.cm_id); 1314 + sc->ib.qp = NULL; 1315 1316 log_rdma_event(INFO, "cancelling idle timer\n"); 1317 cancel_delayed_work_sync(&info->idle_timer_work); ··· 1336 log_rdma_event(INFO, "free receive buffers\n"); 1337 wait_event(info->wait_receive_queues, 1338 info->count_receive_queue + info->count_empty_packet_queue 1339 + == sp->recv_credit_max); 1340 destroy_receive_buffers(info); 1341 1342 /* ··· 1355 } 1356 destroy_mr_list(info); 1357 1358 + ib_free_cq(sc->ib.send_cq); 1359 + ib_free_cq(sc->ib.recv_cq); 1360 + ib_dealloc_pd(sc->ib.pd); 1361 + rdma_destroy_id(sc->rdma.cm_id); 1362 1363 /* free mempools */ 1364 mempool_destroy(info->request_mempool); ··· 1367 mempool_destroy(info->response_mempool); 1368 kmem_cache_destroy(info->response_cache); 1369 1370 + sc->status = SMBDIRECT_SOCKET_DESTROYED; 1371 1372 destroy_workqueue(info->workqueue); 1373 log_rdma_event(INFO, "rdma session destroyed\n"); ··· 1392 * This is possible if transport is disconnected and we haven't received 1393 * notification from RDMA, but upper layer has detected timeout 1394 */ 1395 + if (server->smbd_conn->socket.status == SMBDIRECT_SOCKET_CONNECTED) { 1396 log_rdma_event(INFO, "disconnecting transport\n"); 1397 smbd_destroy(server); 1398 } ··· 1424 #define MAX_NAME_LEN 80 1425 static int allocate_caches_and_workqueue(struct smbd_connection *info) 1426 { 1427 + struct smbdirect_socket *sc = &info->socket; 1428 + struct smbdirect_socket_parameters *sp = &sc->parameters; 1429 char name[MAX_NAME_LEN]; 1430 int rc; 1431 ··· 1432 kmem_cache_create( 1433 name, 1434 sizeof(struct smbd_request) + 1435 + sizeof(struct smbdirect_data_transfer), 1436 0, SLAB_HWCACHE_ALIGN, NULL); 1437 if (!info->request_cache) 1438 return -ENOMEM; 1439 1440 info->request_mempool = 1441 + mempool_create(sp->send_credit_target, mempool_alloc_slab, 1442 mempool_free_slab, info->request_cache); 1443 if (!info->request_mempool) 1444 goto out1; ··· 1448 kmem_cache_create( 1449 name, 1450 sizeof(struct smbd_response) + 1451 + sp->max_recv_size, 1452 0, SLAB_HWCACHE_ALIGN, NULL); 1453 if (!info->response_cache) 1454 goto out2; 1455 1456 info->response_mempool = 1457 + mempool_create(sp->recv_credit_max, mempool_alloc_slab, 1458 mempool_free_slab, info->response_cache); 1459 if (!info->response_mempool) 1460 goto out3; ··· 1464 if (!info->workqueue) 1465 goto out4; 1466 1467 + rc = allocate_receive_buffers(info, sp->recv_credit_max); 1468 if (rc) { 1469 log_rdma_event(ERR, "failed to allocate receive buffers\n"); 1470 goto out5; ··· 1491 { 1492 int rc; 1493 struct smbd_connection *info; 1494 + struct smbdirect_socket *sc; 1495 + struct smbdirect_socket_parameters *sp; 1496 struct rdma_conn_param conn_param; 1497 struct ib_qp_init_attr qp_attr; 1498 struct sockaddr_in *addr_in = (struct sockaddr_in *) dstaddr; ··· 1500 info = kzalloc(sizeof(struct smbd_connection), GFP_KERNEL); 1501 if (!info) 1502 return NULL; 1503 + sc = &info->socket; 1504 + sp = &sc->parameters; 1505 1506 + sc->status = SMBDIRECT_SOCKET_CONNECTING; 1507 rc = smbd_ia_open(info, dstaddr, port); 1508 if (rc) { 1509 log_rdma_event(INFO, "smbd_ia_open rc=%d\n", rc); 1510 goto create_id_failed; 1511 } 1512 1513 + if (smbd_send_credit_target > sc->ib.dev->attrs.max_cqe || 1514 + smbd_send_credit_target > sc->ib.dev->attrs.max_qp_wr) { 1515 log_rdma_event(ERR, "consider lowering send_credit_target = %d. Possible CQE overrun, device reporting max_cqe %d max_qp_wr %d\n", 1516 smbd_send_credit_target, 1517 + sc->ib.dev->attrs.max_cqe, 1518 + sc->ib.dev->attrs.max_qp_wr); 1519 goto config_failed; 1520 } 1521 1522 + if (smbd_receive_credit_max > sc->ib.dev->attrs.max_cqe || 1523 + smbd_receive_credit_max > sc->ib.dev->attrs.max_qp_wr) { 1524 log_rdma_event(ERR, "consider lowering receive_credit_max = %d. Possible CQE overrun, device reporting max_cqe %d max_qp_wr %d\n", 1525 smbd_receive_credit_max, 1526 + sc->ib.dev->attrs.max_cqe, 1527 + sc->ib.dev->attrs.max_qp_wr); 1528 goto config_failed; 1529 } 1530 1531 + sp->recv_credit_max = smbd_receive_credit_max; 1532 + sp->send_credit_target = smbd_send_credit_target; 1533 + sp->max_send_size = smbd_max_send_size; 1534 + sp->max_fragmented_recv_size = smbd_max_fragmented_recv_size; 1535 + sp->max_recv_size = smbd_max_receive_size; 1536 + sp->keepalive_interval_msec = smbd_keep_alive_interval * 1000; 1537 1538 + if (sc->ib.dev->attrs.max_send_sge < SMBDIRECT_MAX_SEND_SGE || 1539 + sc->ib.dev->attrs.max_recv_sge < SMBDIRECT_MAX_RECV_SGE) { 1540 log_rdma_event(ERR, 1541 "device %.*s max_send_sge/max_recv_sge = %d/%d too small\n", 1542 IB_DEVICE_NAME_MAX, 1543 + sc->ib.dev->name, 1544 + sc->ib.dev->attrs.max_send_sge, 1545 + sc->ib.dev->attrs.max_recv_sge); 1546 goto config_failed; 1547 } 1548 1549 + sc->ib.send_cq = 1550 + ib_alloc_cq_any(sc->ib.dev, info, 1551 + sp->send_credit_target, IB_POLL_SOFTIRQ); 1552 + if (IS_ERR(sc->ib.send_cq)) { 1553 + sc->ib.send_cq = NULL; 1554 goto alloc_cq_failed; 1555 } 1556 1557 + sc->ib.recv_cq = 1558 + ib_alloc_cq_any(sc->ib.dev, info, 1559 + sp->recv_credit_max, IB_POLL_SOFTIRQ); 1560 + if (IS_ERR(sc->ib.recv_cq)) { 1561 + sc->ib.recv_cq = NULL; 1562 goto alloc_cq_failed; 1563 } 1564 1565 memset(&qp_attr, 0, sizeof(qp_attr)); 1566 qp_attr.event_handler = smbd_qp_async_error_upcall; 1567 qp_attr.qp_context = info; 1568 + qp_attr.cap.max_send_wr = sp->send_credit_target; 1569 + qp_attr.cap.max_recv_wr = sp->recv_credit_max; 1570 qp_attr.cap.max_send_sge = SMBDIRECT_MAX_SEND_SGE; 1571 qp_attr.cap.max_recv_sge = SMBDIRECT_MAX_RECV_SGE; 1572 qp_attr.cap.max_inline_data = 0; 1573 qp_attr.sq_sig_type = IB_SIGNAL_REQ_WR; 1574 qp_attr.qp_type = IB_QPT_RC; 1575 + qp_attr.send_cq = sc->ib.send_cq; 1576 + qp_attr.recv_cq = sc->ib.recv_cq; 1577 qp_attr.port_num = ~0; 1578 1579 + rc = rdma_create_qp(sc->rdma.cm_id, sc->ib.pd, &qp_attr); 1580 if (rc) { 1581 log_rdma_event(ERR, "rdma_create_qp failed %i\n", rc); 1582 goto create_qp_failed; 1583 } 1584 + sc->ib.qp = sc->rdma.cm_id->qp; 1585 1586 memset(&conn_param, 0, sizeof(conn_param)); 1587 conn_param.initiator_depth = 0; 1588 1589 conn_param.responder_resources = 1590 + min(sc->ib.dev->attrs.max_qp_rd_atom, 1591 SMBD_CM_RESPONDER_RESOURCES); 1592 info->responder_resources = conn_param.responder_resources; 1593 log_rdma_mr(INFO, "responder_resources=%d\n", 1594 info->responder_resources); 1595 1596 /* Need to send IRD/ORD in private data for iWARP */ 1597 + sc->ib.dev->ops.get_port_immutable( 1598 + sc->ib.dev, sc->rdma.cm_id->port_num, &port_immutable); 1599 if (port_immutable.core_cap_flags & RDMA_CORE_PORT_IWARP) { 1600 ird_ord_hdr[0] = info->responder_resources; 1601 ird_ord_hdr[1] = 1; ··· 1615 init_waitqueue_head(&info->conn_wait); 1616 init_waitqueue_head(&info->disconn_wait); 1617 init_waitqueue_head(&info->wait_reassembly_queue); 1618 + rc = rdma_connect(sc->rdma.cm_id, &conn_param); 1619 if (rc) { 1620 log_rdma_event(ERR, "rdma_connect() failed with %i\n", rc); 1621 goto rdma_connect_failed; 1622 } 1623 1624 wait_event_interruptible( 1625 + info->conn_wait, sc->status != SMBDIRECT_SOCKET_CONNECTING); 1626 1627 + if (sc->status != SMBDIRECT_SOCKET_CONNECTED) { 1628 log_rdma_event(ERR, "rdma_connect failed port=%d\n", port); 1629 goto rdma_connect_failed; 1630 } ··· 1640 init_waitqueue_head(&info->wait_send_queue); 1641 INIT_DELAYED_WORK(&info->idle_timer_work, idle_connection_timer); 1642 queue_delayed_work(info->workqueue, &info->idle_timer_work, 1643 + msecs_to_jiffies(sp->keepalive_interval_msec)); 1644 1645 init_waitqueue_head(&info->wait_send_pending); 1646 atomic_set(&info->send_pending, 0); ··· 1675 negotiation_failed: 1676 cancel_delayed_work_sync(&info->idle_timer_work); 1677 destroy_caches_and_workqueue(info); 1678 + sc->status = SMBDIRECT_SOCKET_NEGOTIATE_FAILED; 1679 init_waitqueue_head(&info->conn_wait); 1680 + rdma_disconnect(sc->rdma.cm_id); 1681 wait_event(info->conn_wait, 1682 + sc->status == SMBDIRECT_SOCKET_DISCONNECTED); 1683 1684 allocate_cache_failed: 1685 rdma_connect_failed: 1686 + rdma_destroy_qp(sc->rdma.cm_id); 1687 1688 create_qp_failed: 1689 alloc_cq_failed: 1690 + if (sc->ib.send_cq) 1691 + ib_free_cq(sc->ib.send_cq); 1692 + if (sc->ib.recv_cq) 1693 + ib_free_cq(sc->ib.recv_cq); 1694 1695 config_failed: 1696 + ib_dealloc_pd(sc->ib.pd); 1697 + rdma_destroy_id(sc->rdma.cm_id); 1698 1699 create_id_failed: 1700 kfree(info); ··· 1734 static int smbd_recv_buf(struct smbd_connection *info, char *buf, 1735 unsigned int size) 1736 { 1737 + struct smbdirect_socket *sc = &info->socket; 1738 struct smbd_response *response; 1739 + struct smbdirect_data_transfer *data_transfer; 1740 int to_copy, to_read, data_read, offset; 1741 u32 data_length, remaining_data_length, data_offset; 1742 int rc; ··· 1848 rc = wait_event_interruptible( 1849 info->wait_reassembly_queue, 1850 info->reassembly_data_length >= size || 1851 + sc->status != SMBDIRECT_SOCKET_CONNECTED); 1852 /* Don't return any data if interrupted */ 1853 if (rc) 1854 return rc; 1855 1856 + if (sc->status != SMBDIRECT_SOCKET_CONNECTED) { 1857 log_read(ERR, "disconnected\n"); 1858 return -ECONNABORTED; 1859 } ··· 1871 struct page *page, unsigned int page_offset, 1872 unsigned int to_read) 1873 { 1874 + struct smbdirect_socket *sc = &info->socket; 1875 int ret; 1876 char *to_address; 1877 void *page_address; ··· 1879 ret = wait_event_interruptible( 1880 info->wait_reassembly_queue, 1881 info->reassembly_data_length >= to_read || 1882 + sc->status != SMBDIRECT_SOCKET_CONNECTED); 1883 if (ret) 1884 return ret; 1885 ··· 1954 int num_rqst, struct smb_rqst *rqst_array) 1955 { 1956 struct smbd_connection *info = server->smbd_conn; 1957 + struct smbdirect_socket *sc = &info->socket; 1958 + struct smbdirect_socket_parameters *sp = &sc->parameters; 1959 struct smb_rqst *rqst; 1960 struct iov_iter iter; 1961 unsigned int remaining_data_length, klen; 1962 int rc, i, rqst_idx; 1963 1964 + if (sc->status != SMBDIRECT_SOCKET_CONNECTED) 1965 return -EAGAIN; 1966 1967 /* ··· 1971 for (i = 0; i < num_rqst; i++) 1972 remaining_data_length += smb_rqst_len(server, &rqst_array[i]); 1973 1974 + if (unlikely(remaining_data_length > sp->max_fragmented_send_size)) { 1975 /* assertion: payload never exceeds negotiated maximum */ 1976 log_write(ERR, "payload size %d > max size %d\n", 1977 + remaining_data_length, sp->max_fragmented_send_size); 1978 return -EINVAL; 1979 } 1980 ··· 2053 { 2054 struct smbd_connection *info = 2055 container_of(work, struct smbd_connection, mr_recovery_work); 2056 + struct smbdirect_socket *sc = &info->socket; 2057 struct smbd_mr *smbdirect_mr; 2058 int rc; 2059 ··· 2070 } 2071 2072 smbdirect_mr->mr = ib_alloc_mr( 2073 + sc->ib.pd, info->mr_type, 2074 info->max_frmr_depth); 2075 if (IS_ERR(smbdirect_mr->mr)) { 2076 log_rdma_mr(ERR, "ib_alloc_mr failed mr_type=%x max_frmr_depth=%x\n", ··· 2099 2100 static void destroy_mr_list(struct smbd_connection *info) 2101 { 2102 + struct smbdirect_socket *sc = &info->socket; 2103 struct smbd_mr *mr, *tmp; 2104 2105 cancel_work_sync(&info->mr_recovery_work); 2106 list_for_each_entry_safe(mr, tmp, &info->mr_list, list) { 2107 if (mr->state == MR_INVALIDATED) 2108 + ib_dma_unmap_sg(sc->ib.dev, mr->sgt.sgl, 2109 mr->sgt.nents, mr->dir); 2110 ib_dereg_mr(mr->mr); 2111 kfree(mr->sgt.sgl); ··· 2121 */ 2122 static int allocate_mr_list(struct smbd_connection *info) 2123 { 2124 + struct smbdirect_socket *sc = &info->socket; 2125 int i; 2126 struct smbd_mr *smbdirect_mr, *tmp; 2127 ··· 2136 smbdirect_mr = kzalloc(sizeof(*smbdirect_mr), GFP_KERNEL); 2137 if (!smbdirect_mr) 2138 goto cleanup_entries; 2139 + smbdirect_mr->mr = ib_alloc_mr(sc->ib.pd, info->mr_type, 2140 info->max_frmr_depth); 2141 if (IS_ERR(smbdirect_mr->mr)) { 2142 log_rdma_mr(ERR, "ib_alloc_mr failed mr_type=%x max_frmr_depth=%x\n", ··· 2181 */ 2182 static struct smbd_mr *get_mr(struct smbd_connection *info) 2183 { 2184 + struct smbdirect_socket *sc = &info->socket; 2185 struct smbd_mr *ret; 2186 int rc; 2187 again: 2188 rc = wait_event_interruptible(info->wait_mr, 2189 atomic_read(&info->mr_ready_count) || 2190 + sc->status != SMBDIRECT_SOCKET_CONNECTED); 2191 if (rc) { 2192 log_rdma_mr(ERR, "wait_event_interruptible rc=%x\n", rc); 2193 return NULL; 2194 } 2195 2196 + if (sc->status != SMBDIRECT_SOCKET_CONNECTED) { 2197 + log_rdma_mr(ERR, "sc->status=%x\n", sc->status); 2198 return NULL; 2199 } 2200 ··· 2247 struct iov_iter *iter, 2248 bool writing, bool need_invalidate) 2249 { 2250 + struct smbdirect_socket *sc = &info->socket; 2251 struct smbd_mr *smbdirect_mr; 2252 int rc, num_pages; 2253 enum dma_data_direction dir; ··· 2276 num_pages, iov_iter_count(iter), info->max_frmr_depth); 2277 smbd_iter_to_mr(info, iter, &smbdirect_mr->sgt, info->max_frmr_depth); 2278 2279 + rc = ib_dma_map_sg(sc->ib.dev, smbdirect_mr->sgt.sgl, 2280 smbdirect_mr->sgt.nents, dir); 2281 if (!rc) { 2282 log_rdma_mr(ERR, "ib_dma_map_sg num_pages=%x dir=%x rc=%x\n", ··· 2312 * on IB_WR_REG_MR. Hardware enforces a barrier and order of execution 2313 * on the next ib_post_send when we actually send I/O to remote peer 2314 */ 2315 + rc = ib_post_send(sc->ib.qp, &reg_wr->wr, NULL); 2316 if (!rc) 2317 return smbdirect_mr; 2318 ··· 2321 2322 /* If all failed, attempt to recover this MR by setting it MR_ERROR*/ 2323 map_mr_error: 2324 + ib_dma_unmap_sg(sc->ib.dev, smbdirect_mr->sgt.sgl, 2325 smbdirect_mr->sgt.nents, smbdirect_mr->dir); 2326 2327 dma_map_error: ··· 2359 { 2360 struct ib_send_wr *wr; 2361 struct smbd_connection *info = smbdirect_mr->conn; 2362 + struct smbdirect_socket *sc = &info->socket; 2363 int rc = 0; 2364 2365 if (smbdirect_mr->need_invalidate) { ··· 2372 wr->send_flags = IB_SEND_SIGNALED; 2373 2374 init_completion(&smbdirect_mr->invalidate_done); 2375 + rc = ib_post_send(sc->ib.qp, wr, NULL); 2376 if (rc) { 2377 log_rdma_mr(ERR, "ib_post_send failed rc=%x\n", rc); 2378 smbd_disconnect_rdma_connection(info); ··· 2389 2390 if (smbdirect_mr->state == MR_INVALIDATED) { 2391 ib_dma_unmap_sg( 2392 + sc->ib.dev, smbdirect_mr->sgt.sgl, 2393 smbdirect_mr->sgt.nents, 2394 smbdirect_mr->dir); 2395 smbdirect_mr->state = MR_READY;
+5 -64
fs/smb/client/smbdirect.h
··· 15 #include <rdma/rdma_cm.h> 16 #include <linux/mempool.h> 17 18 extern int rdma_readwrite_threshold; 19 extern int smbd_max_frmr_depth; 20 extern int smbd_keep_alive_interval; ··· 53 * 5. mempools for allocating packets 54 */ 55 struct smbd_connection { 56 - enum smbd_connection_status transport_status; 57 58 - /* RDMA related */ 59 - struct rdma_cm_id *id; 60 - struct ib_qp_init_attr qp_attr; 61 - struct ib_pd *pd; 62 - struct ib_cq *send_cq, *recv_cq; 63 - struct ib_device_attr dev_attr; 64 int ri_rc; 65 struct completion ri_done; 66 wait_queue_head_t conn_wait; ··· 69 spinlock_t lock_new_credits_offered; 70 int new_credits_offered; 71 72 - /* Connection parameters defined in [MS-SMBD] 3.1.1.1 */ 73 - int receive_credit_max; 74 - int send_credit_target; 75 - int max_send_size; 76 - int max_fragmented_recv_size; 77 - int max_fragmented_send_size; 78 - int max_receive_size; 79 - int keep_alive_interval; 80 - int max_readwrite_size; 81 enum keep_alive_status keep_alive_requested; 82 int protocol; 83 atomic_t send_credits; ··· 165 SMBD_NEGOTIATE_RESP, 166 SMBD_TRANSFER_DATA, 167 }; 168 - 169 - #define SMB_DIRECT_RESPONSE_REQUESTED 0x0001 170 - 171 - /* SMBD negotiation request packet [MS-SMBD] 2.2.1 */ 172 - struct smbd_negotiate_req { 173 - __le16 min_version; 174 - __le16 max_version; 175 - __le16 reserved; 176 - __le16 credits_requested; 177 - __le32 preferred_send_size; 178 - __le32 max_receive_size; 179 - __le32 max_fragmented_size; 180 - } __packed; 181 - 182 - /* SMBD negotiation response packet [MS-SMBD] 2.2.2 */ 183 - struct smbd_negotiate_resp { 184 - __le16 min_version; 185 - __le16 max_version; 186 - __le16 negotiated_version; 187 - __le16 reserved; 188 - __le16 credits_requested; 189 - __le16 credits_granted; 190 - __le32 status; 191 - __le32 max_readwrite_size; 192 - __le32 preferred_send_size; 193 - __le32 max_receive_size; 194 - __le32 max_fragmented_size; 195 - } __packed; 196 - 197 - /* SMBD data transfer packet with payload [MS-SMBD] 2.2.3 */ 198 - struct smbd_data_transfer { 199 - __le16 credits_requested; 200 - __le16 credits_granted; 201 - __le16 flags; 202 - __le16 reserved; 203 - __le32 remaining_data_length; 204 - __le32 data_offset; 205 - __le32 data_length; 206 - __le32 padding; 207 - __u8 buffer[]; 208 - } __packed; 209 - 210 - /* The packet fields for a registered RDMA buffer */ 211 - struct smbd_buffer_descriptor_v1 { 212 - __le64 offset; 213 - __le32 token; 214 - __le32 length; 215 - } __packed; 216 217 /* Maximum number of SGEs used by smbdirect.c in any send work request */ 218 #define SMBDIRECT_MAX_SEND_SGE 6
··· 15 #include <rdma/rdma_cm.h> 16 #include <linux/mempool.h> 17 18 + #include "../common/smbdirect/smbdirect.h" 19 + #include "../common/smbdirect/smbdirect_socket.h" 20 + 21 extern int rdma_readwrite_threshold; 22 extern int smbd_max_frmr_depth; 23 extern int smbd_keep_alive_interval; ··· 50 * 5. mempools for allocating packets 51 */ 52 struct smbd_connection { 53 + struct smbdirect_socket socket; 54 55 int ri_rc; 56 struct completion ri_done; 57 wait_queue_head_t conn_wait; ··· 72 spinlock_t lock_new_credits_offered; 73 int new_credits_offered; 74 75 + /* dynamic connection parameters defined in [MS-SMBD] 3.1.1.1 */ 76 enum keep_alive_status keep_alive_requested; 77 int protocol; 78 atomic_t send_credits; ··· 176 SMBD_NEGOTIATE_RESP, 177 SMBD_TRANSFER_DATA, 178 }; 179 180 /* Maximum number of SGEs used by smbdirect.c in any send work request */ 181 #define SMBDIRECT_MAX_SEND_SGE 6
+7 -7
fs/smb/client/transport.c
··· 1018 uint index = 0; 1019 unsigned int min_in_flight = UINT_MAX, max_in_flight = 0; 1020 struct TCP_Server_Info *server = NULL; 1021 - int i; 1022 1023 if (!ses) 1024 return NULL; 1025 1026 spin_lock(&ses->chan_lock); 1027 for (i = 0; i < ses->chan_count; i++) { 1028 - server = ses->chans[i].server; 1029 if (!server || server->terminate) 1030 continue; 1031 ··· 1044 */ 1045 if (server->in_flight < min_in_flight) { 1046 min_in_flight = server->in_flight; 1047 - index = i; 1048 } 1049 if (server->in_flight > max_in_flight) 1050 max_in_flight = server->in_flight; 1051 } 1052 1053 /* if all channels are equally loaded, fall back to round-robin */ 1054 - if (min_in_flight == max_in_flight) { 1055 - index = (uint)atomic_inc_return(&ses->chan_seq); 1056 - index %= ses->chan_count; 1057 - } 1058 1059 server = ses->chans[index].server; 1060 spin_unlock(&ses->chan_lock);
··· 1018 uint index = 0; 1019 unsigned int min_in_flight = UINT_MAX, max_in_flight = 0; 1020 struct TCP_Server_Info *server = NULL; 1021 + int i, start, cur; 1022 1023 if (!ses) 1024 return NULL; 1025 1026 spin_lock(&ses->chan_lock); 1027 + start = atomic_inc_return(&ses->chan_seq); 1028 for (i = 0; i < ses->chan_count; i++) { 1029 + cur = (start + i) % ses->chan_count; 1030 + server = ses->chans[cur].server; 1031 if (!server || server->terminate) 1032 continue; 1033 ··· 1042 */ 1043 if (server->in_flight < min_in_flight) { 1044 min_in_flight = server->in_flight; 1045 + index = cur; 1046 } 1047 if (server->in_flight > max_in_flight) 1048 max_in_flight = server->in_flight; 1049 } 1050 1051 /* if all channels are equally loaded, fall back to round-robin */ 1052 + if (min_in_flight == max_in_flight) 1053 + index = (uint)start % ses->chan_count; 1054 1055 server = ses->chans[index].server; 1056 spin_unlock(&ses->chan_lock);
+37
fs/smb/common/smbdirect/smbdirect.h
···
··· 1 + /* SPDX-License-Identifier: GPL-2.0-or-later */ 2 + /* 3 + * Copyright (C) 2017, Microsoft Corporation. 4 + * Copyright (C) 2018, LG Electronics. 5 + */ 6 + 7 + #ifndef __FS_SMB_COMMON_SMBDIRECT_SMBDIRECT_H__ 8 + #define __FS_SMB_COMMON_SMBDIRECT_SMBDIRECT_H__ 9 + 10 + /* SMB-DIRECT buffer descriptor V1 structure [MS-SMBD] 2.2.3.1 */ 11 + struct smbdirect_buffer_descriptor_v1 { 12 + __le64 offset; 13 + __le32 token; 14 + __le32 length; 15 + } __packed; 16 + 17 + /* 18 + * Connection parameters mostly from [MS-SMBD] 3.1.1.1 19 + * 20 + * These are setup and negotiated at the beginning of a 21 + * connection and remain constant unless explicitly changed. 22 + * 23 + * Some values are important for the upper layer. 24 + */ 25 + struct smbdirect_socket_parameters { 26 + __u16 recv_credit_max; 27 + __u16 send_credit_target; 28 + __u32 max_send_size; 29 + __u32 max_fragmented_send_size; 30 + __u32 max_recv_size; 31 + __u32 max_fragmented_recv_size; 32 + __u32 max_read_write_size; 33 + __u32 keepalive_interval_msec; 34 + __u32 keepalive_timeout_msec; 35 + } __packed; 36 + 37 + #endif /* __FS_SMB_COMMON_SMBDIRECT_SMBDIRECT_H__ */
+55
fs/smb/common/smbdirect/smbdirect_pdu.h
···
··· 1 + /* SPDX-License-Identifier: GPL-2.0-or-later */ 2 + /* 3 + * Copyright (c) 2017 Stefan Metzmacher 4 + */ 5 + 6 + #ifndef __FS_SMB_COMMON_SMBDIRECT_SMBDIRECT_PDU_H__ 7 + #define __FS_SMB_COMMON_SMBDIRECT_SMBDIRECT_PDU_H__ 8 + 9 + #define SMBDIRECT_V1 0x0100 10 + 11 + /* SMBD negotiation request packet [MS-SMBD] 2.2.1 */ 12 + struct smbdirect_negotiate_req { 13 + __le16 min_version; 14 + __le16 max_version; 15 + __le16 reserved; 16 + __le16 credits_requested; 17 + __le32 preferred_send_size; 18 + __le32 max_receive_size; 19 + __le32 max_fragmented_size; 20 + } __packed; 21 + 22 + /* SMBD negotiation response packet [MS-SMBD] 2.2.2 */ 23 + struct smbdirect_negotiate_resp { 24 + __le16 min_version; 25 + __le16 max_version; 26 + __le16 negotiated_version; 27 + __le16 reserved; 28 + __le16 credits_requested; 29 + __le16 credits_granted; 30 + __le32 status; 31 + __le32 max_readwrite_size; 32 + __le32 preferred_send_size; 33 + __le32 max_receive_size; 34 + __le32 max_fragmented_size; 35 + } __packed; 36 + 37 + #define SMBDIRECT_DATA_MIN_HDR_SIZE 0x14 38 + #define SMBDIRECT_DATA_OFFSET 0x18 39 + 40 + #define SMBDIRECT_FLAG_RESPONSE_REQUESTED 0x0001 41 + 42 + /* SMBD data transfer packet with payload [MS-SMBD] 2.2.3 */ 43 + struct smbdirect_data_transfer { 44 + __le16 credits_requested; 45 + __le16 credits_granted; 46 + __le16 flags; 47 + __le16 reserved; 48 + __le32 remaining_data_length; 49 + __le32 data_offset; 50 + __le32 data_length; 51 + __le32 padding; 52 + __u8 buffer[]; 53 + } __packed; 54 + 55 + #endif /* __FS_SMB_COMMON_SMBDIRECT_SMBDIRECT_PDU_H__ */
+43
fs/smb/common/smbdirect/smbdirect_socket.h
···
··· 1 + /* SPDX-License-Identifier: GPL-2.0-or-later */ 2 + /* 3 + * Copyright (c) 2025 Stefan Metzmacher 4 + */ 5 + 6 + #ifndef __FS_SMB_COMMON_SMBDIRECT_SMBDIRECT_SOCKET_H__ 7 + #define __FS_SMB_COMMON_SMBDIRECT_SMBDIRECT_SOCKET_H__ 8 + 9 + enum smbdirect_socket_status { 10 + SMBDIRECT_SOCKET_CREATED, 11 + SMBDIRECT_SOCKET_CONNECTING, 12 + SMBDIRECT_SOCKET_CONNECTED, 13 + SMBDIRECT_SOCKET_NEGOTIATE_FAILED, 14 + SMBDIRECT_SOCKET_DISCONNECTING, 15 + SMBDIRECT_SOCKET_DISCONNECTED, 16 + SMBDIRECT_SOCKET_DESTROYED 17 + }; 18 + 19 + struct smbdirect_socket { 20 + enum smbdirect_socket_status status; 21 + 22 + /* RDMA related */ 23 + struct { 24 + struct rdma_cm_id *cm_id; 25 + } rdma; 26 + 27 + /* IB verbs related */ 28 + struct { 29 + struct ib_pd *pd; 30 + struct ib_cq *send_cq; 31 + struct ib_cq *recv_cq; 32 + 33 + /* 34 + * shortcuts for rdma.cm_id->{qp,device}; 35 + */ 36 + struct ib_qp *qp; 37 + struct ib_device *dev; 38 + } ib; 39 + 40 + struct smbdirect_socket_parameters parameters; 41 + }; 42 + 43 + #endif /* __FS_SMB_COMMON_SMBDIRECT_SMBDIRECT_SOCKET_H__ */