tjh.dev
Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
1.. SPDX-License-Identifier: GPL-2.0
2
3===========================
4SMB Direct - SMB3 over RDMA
5===========================
6
7This document describes how to set up the Linux SMB client and server to
8use RDMA.
9
10Overview
11========
12The Linux SMB kernel client supports SMB Direct, which is a transport
13scheme for SMB3 that uses RDMA (Remote Direct Memory Access) to provide
14high throughput and low latencies by bypassing the traditional TCP/IP
15stack.
16SMB Direct on the Linux SMB client can be tested against KSMBD - a
17kernel-space SMB server.
18
19Installation
20=============
21- Install an RDMA device. As long as the RDMA device driver is supported
22 by the kernel, it should work. This includes both software emulators (soft
23 RoCE, soft iWARP) and hardware devices (InfiniBand, RoCE, iWARP).
24
25- Install a kernel with SMB Direct support. The first kernel release to
26 support SMB Direct on both the client and server side is 5.15. Therefore,
27 a distribution compatible with kernel 5.15 or later is required.
28
29- Install cifs-utils, which provides the `mount.cifs` command to mount SMB
30 shares.
31
32- Configure the RDMA stack
33
34 Make sure that your kernel configuration has RDMA support enabled. Under
35 Device Drivers -> Infiniband support, update the kernel configuration to
36 enable Infiniband support.
37
38 Enable the appropriate IB HCA support or iWARP adapter support,
39 depending on your hardware.
40
41 If you are using InfiniBand, enable IP-over-InfiniBand support.
42
43 For soft RDMA, enable either the soft iWARP (`RDMA _SIW`) or soft RoCE
44 (`RDMA_RXE`) module. Install the `iproute2` package and use the
45 `rdma link add` command to load the module and create an
46 RDMA interface.
47
48 e.g. if your local ethernet interface is `eth0`, you can use:
49
50 .. code-block:: bash
51
52 sudo rdma link add siw0 type siw netdev eth0
53
54- Enable SMB Direct support for both the server and the client in the kernel
55 configuration.
56
57 Server Setup
58
59 .. code-block:: text
60
61 Network File Systems --->
62 <M> SMB3 server support
63 [*] Support for SMB Direct protocol
64
65 Client Setup
66
67 .. code-block:: text
68
69 Network File Systems --->
70 <M> SMB3 and CIFS support (advanced network filesystem)
71 [*] SMB Direct support
72
73- Build and install the kernel. SMB Direct support will be enabled in the
74 cifs.ko and ksmbd.ko modules.
75
76Setup and Usage
77================
78
79- Set up and start a KSMBD server as described in the `KSMBD documentation
80 <https://www.kernel.org/doc/Documentation/filesystems/smb/ksmbd.rst>`_.
81 Also add the "server multi channel support = yes" parameter to ksmbd.conf.
82
83- On the client, mount the share with `rdma` mount option to use SMB Direct
84 (specify a SMB version 3.0 or higher using `vers`).
85
86 For example:
87
88 .. code-block:: bash
89
90 mount -t cifs //server/share /mnt/point -o vers=3.1.1,rdma
91
92- To verify that the mount is using SMB Direct, you can check dmesg for the
93 following log line after mounting:
94
95 .. code-block:: text
96
97 CIFS: VFS: RDMA transport established
98
99 Or, verify `rdma` mount option for the share in `/proc/mounts`:
100
101 .. code-block:: bash
102
103 cat /proc/mounts | grep cifs