Documentation: nfs-rdma: convert to ReST

+1

Documentation/admin-guide/nfs/index.rst

··· 7 7 8 8 nfs-client 9 9 nfsroot 10 + nfs-rdma

+292

Documentation/admin-guide/nfs/nfs-rdma.rst

··· 1 + =================== 2 + Setting up NFS/RDMA 3 + =================== 4 + 5 + :Author: 6 + NetApp and Open Grid Computing (May 29, 2008) 7 + 8 + .. warning:: 9 + This document is probably obsolete. 10 + 11 + Overview 12 + ======== 13 + 14 + This document describes how to install and setup the Linux NFS/RDMA client 15 + and server software. 16 + 17 + The NFS/RDMA client was first included in Linux 2.6.24. The NFS/RDMA server 18 + was first included in the following release, Linux 2.6.25. 19 + 20 + In our testing, we have obtained excellent performance results (full 10Gbit 21 + wire bandwidth at minimal client CPU) under many workloads. The code passes 22 + the full Connectathon test suite and operates over both Infiniband and iWARP 23 + RDMA adapters. 24 + 25 + Getting Help 26 + ============ 27 + 28 + If you get stuck, you can ask questions on the 29 + nfs-rdma-devel@lists.sourceforge.net mailing list. 30 + 31 + Installation 32 + ============ 33 + 34 + These instructions are a step by step guide to building a machine for 35 + use with NFS/RDMA. 36 + 37 + - Install an RDMA device 38 + 39 + Any device supported by the drivers in drivers/infiniband/hw is acceptable. 40 + 41 + Testing has been performed using several Mellanox-based IB cards, the 42 + Ammasso AMS1100 iWARP adapter, and the Chelsio cxgb3 iWARP adapter. 43 + 44 + - Install a Linux distribution and tools 45 + 46 + The first kernel release to contain both the NFS/RDMA client and server was 47 + Linux 2.6.25 Therefore, a distribution compatible with this and subsequent 48 + Linux kernel release should be installed. 49 + 50 + The procedures described in this document have been tested with 51 + distributions from Red Hat's Fedora Project (http://fedora.redhat.com/). 52 + 53 + - Install nfs-utils-1.1.2 or greater on the client 54 + 55 + An NFS/RDMA mount point can be obtained by using the mount.nfs command in 56 + nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils 57 + version with support for NFS/RDMA mounts, but for various reasons we 58 + recommend using nfs-utils-1.1.2 or greater). To see which version of 59 + mount.nfs you are using, type: 60 + 61 + .. code-block:: sh 62 + 63 + $ /sbin/mount.nfs -V 64 + 65 + If the version is less than 1.1.2 or the command does not exist, 66 + you should install the latest version of nfs-utils. 67 + 68 + Download the latest package from: http://www.kernel.org/pub/linux/utils/nfs 69 + 70 + Uncompress the package and follow the installation instructions. 71 + 72 + If you will not need the idmapper and gssd executables (you do not need 73 + these to create an NFS/RDMA enabled mount command), the installation 74 + process can be simplified by disabling these features when running 75 + configure: 76 + 77 + .. code-block:: sh 78 + 79 + $ ./configure --disable-gss --disable-nfsv4 80 + 81 + To build nfs-utils you will need the tcp_wrappers package installed. For 82 + more information on this see the package's README and INSTALL files. 83 + 84 + After building the nfs-utils package, there will be a mount.nfs binary in 85 + the utils/mount directory. This binary can be used to initiate NFS v2, v3, 86 + or v4 mounts. To initiate a v4 mount, the binary must be called 87 + mount.nfs4. The standard technique is to create a symlink called 88 + mount.nfs4 to mount.nfs. 89 + 90 + This mount.nfs binary should be installed at /sbin/mount.nfs as follows: 91 + 92 + .. code-block:: sh 93 + 94 + $ sudo cp utils/mount/mount.nfs /sbin/mount.nfs 95 + 96 + In this location, mount.nfs will be invoked automatically for NFS mounts 97 + by the system mount command. 98 + 99 + .. note:: 100 + mount.nfs and therefore nfs-utils-1.1.2 or greater is only needed 101 + on the NFS client machine. You do not need this specific version of 102 + nfs-utils on the server. Furthermore, only the mount.nfs command from 103 + nfs-utils-1.1.2 is needed on the client. 104 + 105 + - Install a Linux kernel with NFS/RDMA 106 + 107 + The NFS/RDMA client and server are both included in the mainline Linux 108 + kernel version 2.6.25 and later. This and other versions of the Linux 109 + kernel can be found at: https://www.kernel.org/pub/linux/kernel/ 110 + 111 + Download the sources and place them in an appropriate location. 112 + 113 + - Configure the RDMA stack 114 + 115 + Make sure your kernel configuration has RDMA support enabled. Under 116 + Device Drivers -> InfiniBand support, update the kernel configuration 117 + to enable InfiniBand support [NOTE: the option name is misleading. Enabling 118 + InfiniBand support is required for all RDMA devices (IB, iWARP, etc.)]. 119 + 120 + Enable the appropriate IB HCA support (mlx4, mthca, ehca, ipath, etc.) or 121 + iWARP adapter support (amso, cxgb3, etc.). 122 + 123 + If you are using InfiniBand, be sure to enable IP-over-InfiniBand support. 124 + 125 + - Configure the NFS client and server 126 + 127 + Your kernel configuration must also have NFS file system support and/or 128 + NFS server support enabled. These and other NFS related configuration 129 + options can be found under File Systems -> Network File Systems. 130 + 131 + - Build, install, reboot 132 + 133 + The NFS/RDMA code will be enabled automatically if NFS and RDMA 134 + are turned on. The NFS/RDMA client and server are configured via the hidden 135 + SUNRPC_XPRT_RDMA config option that depends on SUNRPC and INFINIBAND. The 136 + value of SUNRPC_XPRT_RDMA will be: 137 + 138 + #. N if either SUNRPC or INFINIBAND are N, in this case the NFS/RDMA client 139 + and server will not be built 140 + 141 + #. M if both SUNRPC and INFINIBAND are on (M or Y) and at least one is M, 142 + in this case the NFS/RDMA client and server will be built as modules 143 + 144 + #. Y if both SUNRPC and INFINIBAND are Y, in this case the NFS/RDMA client 145 + and server will be built into the kernel 146 + 147 + Therefore, if you have followed the steps above and turned no NFS and RDMA, 148 + the NFS/RDMA client and server will be built. 149 + 150 + Build a new kernel, install it, boot it. 151 + 152 + Check RDMA and NFS Setup 153 + ======================== 154 + 155 + Before configuring the NFS/RDMA software, it is a good idea to test 156 + your new kernel to ensure that the kernel is working correctly. 157 + In particular, it is a good idea to verify that the RDMA stack 158 + is functioning as expected and standard NFS over TCP/IP and/or UDP/IP 159 + is working properly. 160 + 161 + - Check RDMA Setup 162 + 163 + If you built the RDMA components as modules, load them at 164 + this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel 165 + card: 166 + 167 + .. code-block:: sh 168 + 169 + $ modprobe ib_mthca 170 + $ modprobe ib_ipoib 171 + 172 + If you are using InfiniBand, make sure there is a Subnet Manager (SM) 173 + running on the network. If your IB switch has an embedded SM, you can 174 + use it. Otherwise, you will need to run an SM, such as OpenSM, on one 175 + of your end nodes. 176 + 177 + If an SM is running on your network, you should see the following: 178 + 179 + .. code-block:: sh 180 + 181 + $ cat /sys/class/infiniband/driverX/ports/1/state 182 + 4: ACTIVE 183 + 184 + where driverX is mthca0, ipath5, ehca3, etc. 185 + 186 + To further test the InfiniBand software stack, use IPoIB (this 187 + assumes you have two IB hosts named host1 and host2): 188 + 189 + .. code-block:: sh 190 + 191 + host1$ ip link set dev ib0 up 192 + host1$ ip address add dev ib0 a.b.c.x 193 + host2$ ip link set dev ib0 up 194 + host2$ ip address add dev ib0 a.b.c.y 195 + host1$ ping a.b.c.y 196 + host2$ ping a.b.c.x 197 + 198 + For other device types, follow the appropriate procedures. 199 + 200 + - Check NFS Setup 201 + 202 + For the NFS components enabled above (client and/or server), 203 + test their functionality over standard Ethernet using TCP/IP or UDP/IP. 204 + 205 + NFS/RDMA Setup 206 + ============== 207 + 208 + We recommend that you use two machines, one to act as the client and 209 + one to act as the server. 210 + 211 + One time configuration: 212 + ----------------------- 213 + 214 + - On the server system, configure the /etc/exports file and start the NFS/RDMA server. 215 + 216 + Exports entries with the following formats have been tested:: 217 + 218 + /vol0 192.168.0.47(fsid=0,rw,async,insecure,no_root_squash) 219 + /vol0 192.168.0.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash) 220 + 221 + The IP address(es) is(are) the client's IPoIB address for an InfiniBand 222 + HCA or the client's iWARP address(es) for an RNIC. 223 + 224 + .. note:: 225 + The "insecure" option must be used because the NFS/RDMA client does 226 + not use a reserved port. 227 + 228 + Each time a machine boots: 229 + -------------------------- 230 + 231 + - Load and configure the RDMA drivers 232 + 233 + For InfiniBand using a Mellanox adapter: 234 + 235 + .. code-block:: sh 236 + 237 + $ modprobe ib_mthca 238 + $ modprobe ib_ipoib 239 + $ ip li set dev ib0 up 240 + $ ip addr add dev ib0 a.b.c.d 241 + 242 + .. note:: 243 + Please use unique addresses for the client and server! 244 + 245 + - Start the NFS server 246 + 247 + If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in 248 + kernel config), load the RDMA transport module: 249 + 250 + .. code-block:: sh 251 + 252 + $ modprobe svcrdma 253 + 254 + Regardless of how the server was built (module or built-in), start the 255 + server: 256 + 257 + .. code-block:: sh 258 + 259 + $ /etc/init.d/nfs start 260 + 261 + or 262 + 263 + .. code-block:: sh 264 + 265 + $ service nfs start 266 + 267 + Instruct the server to listen on the RDMA transport: 268 + 269 + .. code-block:: sh 270 + 271 + $ echo rdma 20049 > /proc/fs/nfsd/portlist 272 + 273 + - On the client system 274 + 275 + If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in 276 + kernel config), load the RDMA client module: 277 + 278 + .. code-block:: sh 279 + 280 + $ modprobe xprtrdma.ko 281 + 282 + Regardless of how the client was built (module or built-in), use this 283 + command to mount the NFS/RDMA server: 284 + 285 + .. code-block:: sh 286 + 287 + $ mount -o rdma,port=20049 <IPoIB-server-name-or-address>:/<export> /mnt 288 + 289 + To verify that the mount is using RDMA, run "cat /proc/mounts" and check 290 + the "proto" field for the given mount. 291 + 292 + Congratulations! You're using NFS/RDMA!

-274

Documentation/filesystems/nfs/nfs-rdma.txt

··· 1 - ################################################################################ 2 - # # 3 - # NFS/RDMA README # 4 - # # 5 - ################################################################################ 6 - 7 - Author: NetApp and Open Grid Computing 8 - Date: May 29, 2008 9 - 10 - Table of Contents 11 - ~~~~~~~~~~~~~~~~~ 12 - - Overview 13 - - Getting Help 14 - - Installation 15 - - Check RDMA and NFS Setup 16 - - NFS/RDMA Setup 17 - 18 - Overview 19 - ~~~~~~~~ 20 - 21 - This document describes how to install and setup the Linux NFS/RDMA client 22 - and server software. 23 - 24 - The NFS/RDMA client was first included in Linux 2.6.24. The NFS/RDMA server 25 - was first included in the following release, Linux 2.6.25. 26 - 27 - In our testing, we have obtained excellent performance results (full 10Gbit 28 - wire bandwidth at minimal client CPU) under many workloads. The code passes 29 - the full Connectathon test suite and operates over both Infiniband and iWARP 30 - RDMA adapters. 31 - 32 - Getting Help 33 - ~~~~~~~~~~~~ 34 - 35 - If you get stuck, you can ask questions on the 36 - 37 - nfs-rdma-devel@lists.sourceforge.net 38 - 39 - mailing list. 40 - 41 - Installation 42 - ~~~~~~~~~~~~ 43 - 44 - These instructions are a step by step guide to building a machine for 45 - use with NFS/RDMA. 46 - 47 - - Install an RDMA device 48 - 49 - Any device supported by the drivers in drivers/infiniband/hw is acceptable. 50 - 51 - Testing has been performed using several Mellanox-based IB cards, the 52 - Ammasso AMS1100 iWARP adapter, and the Chelsio cxgb3 iWARP adapter. 53 - 54 - - Install a Linux distribution and tools 55 - 56 - The first kernel release to contain both the NFS/RDMA client and server was 57 - Linux 2.6.25 Therefore, a distribution compatible with this and subsequent 58 - Linux kernel release should be installed. 59 - 60 - The procedures described in this document have been tested with 61 - distributions from Red Hat's Fedora Project (http://fedora.redhat.com/). 62 - 63 - - Install nfs-utils-1.1.2 or greater on the client 64 - 65 - An NFS/RDMA mount point can be obtained by using the mount.nfs command in 66 - nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils 67 - version with support for NFS/RDMA mounts, but for various reasons we 68 - recommend using nfs-utils-1.1.2 or greater). To see which version of 69 - mount.nfs you are using, type: 70 - 71 - $ /sbin/mount.nfs -V 72 - 73 - If the version is less than 1.1.2 or the command does not exist, 74 - you should install the latest version of nfs-utils. 75 - 76 - Download the latest package from: 77 - 78 - http://www.kernel.org/pub/linux/utils/nfs 79 - 80 - Uncompress the package and follow the installation instructions. 81 - 82 - If you will not need the idmapper and gssd executables (you do not need 83 - these to create an NFS/RDMA enabled mount command), the installation 84 - process can be simplified by disabling these features when running 85 - configure: 86 - 87 - $ ./configure --disable-gss --disable-nfsv4 88 - 89 - To build nfs-utils you will need the tcp_wrappers package installed. For 90 - more information on this see the package's README and INSTALL files. 91 - 92 - After building the nfs-utils package, there will be a mount.nfs binary in 93 - the utils/mount directory. This binary can be used to initiate NFS v2, v3, 94 - or v4 mounts. To initiate a v4 mount, the binary must be called 95 - mount.nfs4. The standard technique is to create a symlink called 96 - mount.nfs4 to mount.nfs. 97 - 98 - This mount.nfs binary should be installed at /sbin/mount.nfs as follows: 99 - 100 - $ sudo cp utils/mount/mount.nfs /sbin/mount.nfs 101 - 102 - In this location, mount.nfs will be invoked automatically for NFS mounts 103 - by the system mount command. 104 - 105 - NOTE: mount.nfs and therefore nfs-utils-1.1.2 or greater is only needed 106 - on the NFS client machine. You do not need this specific version of 107 - nfs-utils on the server. Furthermore, only the mount.nfs command from 108 - nfs-utils-1.1.2 is needed on the client. 109 - 110 - - Install a Linux kernel with NFS/RDMA 111 - 112 - The NFS/RDMA client and server are both included in the mainline Linux 113 - kernel version 2.6.25 and later. This and other versions of the Linux 114 - kernel can be found at: 115 - 116 - https://www.kernel.org/pub/linux/kernel/ 117 - 118 - Download the sources and place them in an appropriate location. 119 - 120 - - Configure the RDMA stack 121 - 122 - Make sure your kernel configuration has RDMA support enabled. Under 123 - Device Drivers -> InfiniBand support, update the kernel configuration 124 - to enable InfiniBand support [NOTE: the option name is misleading. Enabling 125 - InfiniBand support is required for all RDMA devices (IB, iWARP, etc.)]. 126 - 127 - Enable the appropriate IB HCA support (mlx4, mthca, ehca, ipath, etc.) or 128 - iWARP adapter support (amso, cxgb3, etc.). 129 - 130 - If you are using InfiniBand, be sure to enable IP-over-InfiniBand support. 131 - 132 - - Configure the NFS client and server 133 - 134 - Your kernel configuration must also have NFS file system support and/or 135 - NFS server support enabled. These and other NFS related configuration 136 - options can be found under File Systems -> Network File Systems. 137 - 138 - - Build, install, reboot 139 - 140 - The NFS/RDMA code will be enabled automatically if NFS and RDMA 141 - are turned on. The NFS/RDMA client and server are configured via the hidden 142 - SUNRPC_XPRT_RDMA config option that depends on SUNRPC and INFINIBAND. The 143 - value of SUNRPC_XPRT_RDMA will be: 144 - 145 - - N if either SUNRPC or INFINIBAND are N, in this case the NFS/RDMA client 146 - and server will not be built 147 - - M if both SUNRPC and INFINIBAND are on (M or Y) and at least one is M, 148 - in this case the NFS/RDMA client and server will be built as modules 149 - - Y if both SUNRPC and INFINIBAND are Y, in this case the NFS/RDMA client 150 - and server will be built into the kernel 151 - 152 - Therefore, if you have followed the steps above and turned no NFS and RDMA, 153 - the NFS/RDMA client and server will be built. 154 - 155 - Build a new kernel, install it, boot it. 156 - 157 - Check RDMA and NFS Setup 158 - ~~~~~~~~~~~~~~~~~~~~~~~~ 159 - 160 - Before configuring the NFS/RDMA software, it is a good idea to test 161 - your new kernel to ensure that the kernel is working correctly. 162 - In particular, it is a good idea to verify that the RDMA stack 163 - is functioning as expected and standard NFS over TCP/IP and/or UDP/IP 164 - is working properly. 165 - 166 - - Check RDMA Setup 167 - 168 - If you built the RDMA components as modules, load them at 169 - this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel 170 - card: 171 - 172 - $ modprobe ib_mthca 173 - $ modprobe ib_ipoib 174 - 175 - If you are using InfiniBand, make sure there is a Subnet Manager (SM) 176 - running on the network. If your IB switch has an embedded SM, you can 177 - use it. Otherwise, you will need to run an SM, such as OpenSM, on one 178 - of your end nodes. 179 - 180 - If an SM is running on your network, you should see the following: 181 - 182 - $ cat /sys/class/infiniband/driverX/ports/1/state 183 - 4: ACTIVE 184 - 185 - where driverX is mthca0, ipath5, ehca3, etc. 186 - 187 - To further test the InfiniBand software stack, use IPoIB (this 188 - assumes you have two IB hosts named host1 and host2): 189 - 190 - host1$ ip link set dev ib0 up 191 - host1$ ip address add dev ib0 a.b.c.x 192 - host2$ ip link set dev ib0 up 193 - host2$ ip address add dev ib0 a.b.c.y 194 - host1$ ping a.b.c.y 195 - host2$ ping a.b.c.x 196 - 197 - For other device types, follow the appropriate procedures. 198 - 199 - - Check NFS Setup 200 - 201 - For the NFS components enabled above (client and/or server), 202 - test their functionality over standard Ethernet using TCP/IP or UDP/IP. 203 - 204 - NFS/RDMA Setup 205 - ~~~~~~~~~~~~~~ 206 - 207 - We recommend that you use two machines, one to act as the client and 208 - one to act as the server. 209 - 210 - One time configuration: 211 - 212 - - On the server system, configure the /etc/exports file and 213 - start the NFS/RDMA server. 214 - 215 - Exports entries with the following formats have been tested: 216 - 217 - /vol0 192.168.0.47(fsid=0,rw,async,insecure,no_root_squash) 218 - /vol0 192.168.0.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash) 219 - 220 - The IP address(es) is(are) the client's IPoIB address for an InfiniBand 221 - HCA or the client's iWARP address(es) for an RNIC. 222 - 223 - NOTE: The "insecure" option must be used because the NFS/RDMA client does 224 - not use a reserved port. 225 - 226 - Each time a machine boots: 227 - 228 - - Load and configure the RDMA drivers 229 - 230 - For InfiniBand using a Mellanox adapter: 231 - 232 - $ modprobe ib_mthca 233 - $ modprobe ib_ipoib 234 - $ ip li set dev ib0 up 235 - $ ip addr add dev ib0 a.b.c.d 236 - 237 - NOTE: use unique addresses for the client and server 238 - 239 - - Start the NFS server 240 - 241 - If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in 242 - kernel config), load the RDMA transport module: 243 - 244 - $ modprobe svcrdma 245 - 246 - Regardless of how the server was built (module or built-in), start the 247 - server: 248 - 249 - $ /etc/init.d/nfs start 250 - 251 - or 252 - 253 - $ service nfs start 254 - 255 - Instruct the server to listen on the RDMA transport: 256 - 257 - $ echo rdma 20049 > /proc/fs/nfsd/portlist 258 - 259 - - On the client system 260 - 261 - If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in 262 - kernel config), load the RDMA client module: 263 - 264 - $ modprobe xprtrdma.ko 265 - 266 - Regardless of how the client was built (module or built-in), use this 267 - command to mount the NFS/RDMA server: 268 - 269 - $ mount -o rdma,port=20049 <IPoIB-server-name-or-address>:/<export> /mnt 270 - 271 - To verify that the mount is using RDMA, run "cat /proc/mounts" and check 272 - the "proto" field for the given mount. 273 - 274 - Congratulations! You're using NFS/RDMA!