tjh.dev
Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
docs: networking: convert vrf.txt to ReST
- add SPDX header;
- adjust title markup;
- Add a subtitle for the first section;
- mark code blocks and literals as such;
- adjust identation, whitespaces and blank lines;
- add to networking/index.rst.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Acked-by: David Ahern <dsahern@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
authored by
Mauro Carvalho Chehab
and committed by
David S. Miller
58ccb2b2
961fb1ff
+453
-419
+1
Documentation/networking/index.rst
+1
Documentation/networking/index.rst
+451
Documentation/networking/vrf.rst
+451
Documentation/networking/vrf.rst
···
1
+
.. SPDX-License-Identifier: GPL-2.0
2
+
3
+
====================================
4
+
Virtual Routing and Forwarding (VRF)
5
+
====================================
6
+
7
+
The VRF Device
8
+
==============
9
+
10
+
The VRF device combined with ip rules provides the ability to create virtual
11
+
routing and forwarding domains (aka VRFs, VRF-lite to be specific) in the
12
+
Linux network stack. One use case is the multi-tenancy problem where each
13
+
tenant has their own unique routing tables and in the very least need
14
+
different default gateways.
15
+
16
+
Processes can be "VRF aware" by binding a socket to the VRF device. Packets
17
+
through the socket then use the routing table associated with the VRF
18
+
device. An important feature of the VRF device implementation is that it
19
+
impacts only Layer 3 and above so L2 tools (e.g., LLDP) are not affected
20
+
(ie., they do not need to be run in each VRF). The design also allows
21
+
the use of higher priority ip rules (Policy Based Routing, PBR) to take
22
+
precedence over the VRF device rules directing specific traffic as desired.
23
+
24
+
In addition, VRF devices allow VRFs to be nested within namespaces. For
25
+
example network namespaces provide separation of network interfaces at the
26
+
device layer, VLANs on the interfaces within a namespace provide L2 separation
27
+
and then VRF devices provide L3 separation.
28
+
29
+
Design
30
+
------
31
+
A VRF device is created with an associated route table. Network interfaces
32
+
are then enslaved to a VRF device::
33
+
34
+
+-----------------------------+
35
+
| vrf-blue | ===> route table 10
36
+
+-----------------------------+
37
+
| | |
38
+
+------+ +------+ +-------------+
39
+
| eth1 | | eth2 | ... | bond1 |
40
+
+------+ +------+ +-------------+
41
+
| |
42
+
+------+ +------+
43
+
| eth8 | | eth9 |
44
+
+------+ +------+
45
+
46
+
Packets received on an enslaved device and are switched to the VRF device
47
+
in the IPv4 and IPv6 processing stacks giving the impression that packets
48
+
flow through the VRF device. Similarly on egress routing rules are used to
49
+
send packets to the VRF device driver before getting sent out the actual
50
+
interface. This allows tcpdump on a VRF device to capture all packets into
51
+
and out of the VRF as a whole\ [1]_. Similarly, netfilter\ [2]_ and tc rules
52
+
can be applied using the VRF device to specify rules that apply to the VRF
53
+
domain as a whole.
54
+
55
+
.. [1] Packets in the forwarded state do not flow through the device, so those
56
+
packets are not seen by tcpdump. Will revisit this limitation in a
57
+
future release.
58
+
59
+
.. [2] Iptables on ingress supports PREROUTING with skb->dev set to the real
60
+
ingress device and both INPUT and PREROUTING rules with skb->dev set to
61
+
the VRF device. For egress POSTROUTING and OUTPUT rules can be written
62
+
using either the VRF device or real egress device.
63
+
64
+
Setup
65
+
-----
66
+
1. VRF device is created with an association to a FIB table.
67
+
e.g,::
68
+
69
+
ip link add vrf-blue type vrf table 10
70
+
ip link set dev vrf-blue up
71
+
72
+
2. An l3mdev FIB rule directs lookups to the table associated with the device.
73
+
A single l3mdev rule is sufficient for all VRFs. The VRF device adds the
74
+
l3mdev rule for IPv4 and IPv6 when the first device is created with a
75
+
default preference of 1000. Users may delete the rule if desired and add
76
+
with a different priority or install per-VRF rules.
77
+
78
+
Prior to the v4.8 kernel iif and oif rules are needed for each VRF device::
79
+
80
+
ip ru add oif vrf-blue table 10
81
+
ip ru add iif vrf-blue table 10
82
+
83
+
3. Set the default route for the table (and hence default route for the VRF)::
84
+
85
+
ip route add table 10 unreachable default metric 4278198272
86
+
87
+
This high metric value ensures that the default unreachable route can
88
+
be overridden by a routing protocol suite. FRRouting interprets
89
+
kernel metrics as a combined admin distance (upper byte) and priority
90
+
(lower 3 bytes). Thus the above metric translates to [255/8192].
91
+
92
+
4. Enslave L3 interfaces to a VRF device::
93
+
94
+
ip link set dev eth1 master vrf-blue
95
+
96
+
Local and connected routes for enslaved devices are automatically moved to
97
+
the table associated with VRF device. Any additional routes depending on
98
+
the enslaved device are dropped and will need to be reinserted to the VRF
99
+
FIB table following the enslavement.
100
+
101
+
The IPv6 sysctl option keep_addr_on_down can be enabled to keep IPv6 global
102
+
addresses as VRF enslavement changes::
103
+
104
+
sysctl -w net.ipv6.conf.all.keep_addr_on_down=1
105
+
106
+
5. Additional VRF routes are added to associated table::
107
+
108
+
ip route add table 10 ...
109
+
110
+
111
+
Applications
112
+
------------
113
+
Applications that are to work within a VRF need to bind their socket to the
114
+
VRF device::
115
+
116
+
setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);
117
+
118
+
or to specify the output device using cmsg and IP_PKTINFO.
119
+
120
+
By default the scope of the port bindings for unbound sockets is
121
+
limited to the default VRF. That is, it will not be matched by packets
122
+
arriving on interfaces enslaved to an l3mdev and processes may bind to
123
+
the same port if they bind to an l3mdev.
124
+
125
+
TCP & UDP services running in the default VRF context (ie., not bound
126
+
to any VRF device) can work across all VRF domains by enabling the
127
+
tcp_l3mdev_accept and udp_l3mdev_accept sysctl options::
128
+
129
+
sysctl -w net.ipv4.tcp_l3mdev_accept=1
130
+
sysctl -w net.ipv4.udp_l3mdev_accept=1
131
+
132
+
These options are disabled by default so that a socket in a VRF is only
133
+
selected for packets in that VRF. There is a similar option for RAW
134
+
sockets, which is enabled by default for reasons of backwards compatibility.
135
+
This is so as to specify the output device with cmsg and IP_PKTINFO, but
136
+
using a socket not bound to the corresponding VRF. This allows e.g. older ping
137
+
implementations to be run with specifying the device but without executing it
138
+
in the VRF. This option can be disabled so that packets received in a VRF
139
+
context are only handled by a raw socket bound to the VRF, and packets in the
140
+
default VRF are only handled by a socket not bound to any VRF::
141
+
142
+
sysctl -w net.ipv4.raw_l3mdev_accept=0
143
+
144
+
netfilter rules on the VRF device can be used to limit access to services
145
+
running in the default VRF context as well.
146
+
147
+
--------------------------------------------------------------------------------
148
+
149
+
Using iproute2 for VRFs
150
+
=======================
151
+
iproute2 supports the vrf keyword as of v4.7. For backwards compatibility this
152
+
section lists both commands where appropriate -- with the vrf keyword and the
153
+
older form without it.
154
+
155
+
1. Create a VRF
156
+
157
+
To instantiate a VRF device and associate it with a table::
158
+
159
+
$ ip link add dev NAME type vrf table ID
160
+
161
+
As of v4.8 the kernel supports the l3mdev FIB rule where a single rule
162
+
covers all VRFs. The l3mdev rule is created for IPv4 and IPv6 on first
163
+
device create.
164
+
165
+
2. List VRFs
166
+
167
+
To list VRFs that have been created::
168
+
169
+
$ ip [-d] link show type vrf
170
+
NOTE: The -d option is needed to show the table id
171
+
172
+
For example::
173
+
174
+
$ ip -d link show type vrf
175
+
11: mgmt: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
176
+
link/ether 72:b3:ba:91:e2:24 brd ff:ff:ff:ff:ff:ff promiscuity 0
177
+
vrf table 1 addrgenmode eui64
178
+
12: red: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
179
+
link/ether b6:6f:6e:f6:da:73 brd ff:ff:ff:ff:ff:ff promiscuity 0
180
+
vrf table 10 addrgenmode eui64
181
+
13: blue: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
182
+
link/ether 36:62:e8:7d:bb:8c brd ff:ff:ff:ff:ff:ff promiscuity 0
183
+
vrf table 66 addrgenmode eui64
184
+
14: green: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
185
+
link/ether e6:28:b8:63:70:bb brd ff:ff:ff:ff:ff:ff promiscuity 0
186
+
vrf table 81 addrgenmode eui64
187
+
188
+
189
+
Or in brief output::
190
+
191
+
$ ip -br link show type vrf
192
+
mgmt UP 72:b3:ba:91:e2:24 <NOARP,MASTER,UP,LOWER_UP>
193
+
red UP b6:6f:6e:f6:da:73 <NOARP,MASTER,UP,LOWER_UP>
194
+
blue UP 36:62:e8:7d:bb:8c <NOARP,MASTER,UP,LOWER_UP>
195
+
green UP e6:28:b8:63:70:bb <NOARP,MASTER,UP,LOWER_UP>
196
+
197
+
198
+
3. Assign a Network Interface to a VRF
199
+
200
+
Network interfaces are assigned to a VRF by enslaving the netdevice to a
201
+
VRF device::
202
+
203
+
$ ip link set dev NAME master NAME
204
+
205
+
On enslavement connected and local routes are automatically moved to the
206
+
table associated with the VRF device.
207
+
208
+
For example::
209
+
210
+
$ ip link set dev eth0 master mgmt
211
+
212
+
213
+
4. Show Devices Assigned to a VRF
214
+
215
+
To show devices that have been assigned to a specific VRF add the master
216
+
option to the ip command::
217
+
218
+
$ ip link show vrf NAME
219
+
$ ip link show master NAME
220
+
221
+
For example::
222
+
223
+
$ ip link show vrf red
224
+
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
225
+
link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
226
+
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
227
+
link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
228
+
7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN mode DEFAULT group default qlen 1000
229
+
link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
230
+
231
+
232
+
Or using the brief output::
233
+
234
+
$ ip -br link show vrf red
235
+
eth1 UP 02:00:00:00:02:02 <BROADCAST,MULTICAST,UP,LOWER_UP>
236
+
eth2 UP 02:00:00:00:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP>
237
+
eth5 DOWN 02:00:00:00:02:06 <BROADCAST,MULTICAST>
238
+
239
+
240
+
5. Show Neighbor Entries for a VRF
241
+
242
+
To list neighbor entries associated with devices enslaved to a VRF device
243
+
add the master option to the ip command::
244
+
245
+
$ ip [-6] neigh show vrf NAME
246
+
$ ip [-6] neigh show master NAME
247
+
248
+
For example::
249
+
250
+
$ ip neigh show vrf red
251
+
10.2.1.254 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
252
+
10.2.2.254 dev eth2 lladdr 5e:54:01:6a:ee:80 REACHABLE
253
+
254
+
$ ip -6 neigh show vrf red
255
+
2002:1::64 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
256
+
257
+
258
+
6. Show Addresses for a VRF
259
+
260
+
To show addresses for interfaces associated with a VRF add the master
261
+
option to the ip command::
262
+
263
+
$ ip addr show vrf NAME
264
+
$ ip addr show master NAME
265
+
266
+
For example::
267
+
268
+
$ ip addr show vrf red
269
+
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
270
+
link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
271
+
inet 10.2.1.2/24 brd 10.2.1.255 scope global eth1
272
+
valid_lft forever preferred_lft forever
273
+
inet6 2002:1::2/120 scope global
274
+
valid_lft forever preferred_lft forever
275
+
inet6 fe80::ff:fe00:202/64 scope link
276
+
valid_lft forever preferred_lft forever
277
+
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
278
+
link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
279
+
inet 10.2.2.2/24 brd 10.2.2.255 scope global eth2
280
+
valid_lft forever preferred_lft forever
281
+
inet6 2002:2::2/120 scope global
282
+
valid_lft forever preferred_lft forever
283
+
inet6 fe80::ff:fe00:203/64 scope link
284
+
valid_lft forever preferred_lft forever
285
+
7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN group default qlen 1000
286
+
link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
287
+
288
+
Or in brief format::
289
+
290
+
$ ip -br addr show vrf red
291
+
eth1 UP 10.2.1.2/24 2002:1::2/120 fe80::ff:fe00:202/64
292
+
eth2 UP 10.2.2.2/24 2002:2::2/120 fe80::ff:fe00:203/64
293
+
eth5 DOWN
294
+
295
+
296
+
7. Show Routes for a VRF
297
+
298
+
To show routes for a VRF use the ip command to display the table associated
299
+
with the VRF device::
300
+
301
+
$ ip [-6] route show vrf NAME
302
+
$ ip [-6] route show table ID
303
+
304
+
For example::
305
+
306
+
$ ip route show vrf red
307
+
unreachable default metric 4278198272
308
+
broadcast 10.2.1.0 dev eth1 proto kernel scope link src 10.2.1.2
309
+
10.2.1.0/24 dev eth1 proto kernel scope link src 10.2.1.2
310
+
local 10.2.1.2 dev eth1 proto kernel scope host src 10.2.1.2
311
+
broadcast 10.2.1.255 dev eth1 proto kernel scope link src 10.2.1.2
312
+
broadcast 10.2.2.0 dev eth2 proto kernel scope link src 10.2.2.2
313
+
10.2.2.0/24 dev eth2 proto kernel scope link src 10.2.2.2
314
+
local 10.2.2.2 dev eth2 proto kernel scope host src 10.2.2.2
315
+
broadcast 10.2.2.255 dev eth2 proto kernel scope link src 10.2.2.2
316
+
317
+
$ ip -6 route show vrf red
318
+
local 2002:1:: dev lo proto none metric 0 pref medium
319
+
local 2002:1::2 dev lo proto none metric 0 pref medium
320
+
2002:1::/120 dev eth1 proto kernel metric 256 pref medium
321
+
local 2002:2:: dev lo proto none metric 0 pref medium
322
+
local 2002:2::2 dev lo proto none metric 0 pref medium
323
+
2002:2::/120 dev eth2 proto kernel metric 256 pref medium
324
+
local fe80:: dev lo proto none metric 0 pref medium
325
+
local fe80:: dev lo proto none metric 0 pref medium
326
+
local fe80::ff:fe00:202 dev lo proto none metric 0 pref medium
327
+
local fe80::ff:fe00:203 dev lo proto none metric 0 pref medium
328
+
fe80::/64 dev eth1 proto kernel metric 256 pref medium
329
+
fe80::/64 dev eth2 proto kernel metric 256 pref medium
330
+
ff00::/8 dev red metric 256 pref medium
331
+
ff00::/8 dev eth1 metric 256 pref medium
332
+
ff00::/8 dev eth2 metric 256 pref medium
333
+
unreachable default dev lo metric 4278198272 error -101 pref medium
334
+
335
+
8. Route Lookup for a VRF
336
+
337
+
A test route lookup can be done for a VRF::
338
+
339
+
$ ip [-6] route get vrf NAME ADDRESS
340
+
$ ip [-6] route get oif NAME ADDRESS
341
+
342
+
For example::
343
+
344
+
$ ip route get 10.2.1.40 vrf red
345
+
10.2.1.40 dev eth1 table red src 10.2.1.2
346
+
cache
347
+
348
+
$ ip -6 route get 2002:1::32 vrf red
349
+
2002:1::32 from :: dev eth1 table red proto kernel src 2002:1::2 metric 256 pref medium
350
+
351
+
352
+
9. Removing Network Interface from a VRF
353
+
354
+
Network interfaces are removed from a VRF by breaking the enslavement to
355
+
the VRF device::
356
+
357
+
$ ip link set dev NAME nomaster
358
+
359
+
Connected routes are moved back to the default table and local entries are
360
+
moved to the local table.
361
+
362
+
For example::
363
+
364
+
$ ip link set dev eth0 nomaster
365
+
366
+
--------------------------------------------------------------------------------
367
+
368
+
Commands used in this example::
369
+
370
+
cat >> /etc/iproute2/rt_tables.d/vrf.conf <<EOF
371
+
1 mgmt
372
+
10 red
373
+
66 blue
374
+
81 green
375
+
EOF
376
+
377
+
function vrf_create
378
+
{
379
+
VRF=$1
380
+
TBID=$2
381
+
382
+
# create VRF device
383
+
ip link add ${VRF} type vrf table ${TBID}
384
+
385
+
if [ "${VRF}" != "mgmt" ]; then
386
+
ip route add table ${TBID} unreachable default metric 4278198272
387
+
fi
388
+
ip link set dev ${VRF} up
389
+
}
390
+
391
+
vrf_create mgmt 1
392
+
ip link set dev eth0 master mgmt
393
+
394
+
vrf_create red 10
395
+
ip link set dev eth1 master red
396
+
ip link set dev eth2 master red
397
+
ip link set dev eth5 master red
398
+
399
+
vrf_create blue 66
400
+
ip link set dev eth3 master blue
401
+
402
+
vrf_create green 81
403
+
ip link set dev eth4 master green
404
+
405
+
406
+
Interface addresses from /etc/network/interfaces:
407
+
auto eth0
408
+
iface eth0 inet static
409
+
address 10.0.0.2
410
+
netmask 255.255.255.0
411
+
gateway 10.0.0.254
412
+
413
+
iface eth0 inet6 static
414
+
address 2000:1::2
415
+
netmask 120
416
+
417
+
auto eth1
418
+
iface eth1 inet static
419
+
address 10.2.1.2
420
+
netmask 255.255.255.0
421
+
422
+
iface eth1 inet6 static
423
+
address 2002:1::2
424
+
netmask 120
425
+
426
+
auto eth2
427
+
iface eth2 inet static
428
+
address 10.2.2.2
429
+
netmask 255.255.255.0
430
+
431
+
iface eth2 inet6 static
432
+
address 2002:2::2
433
+
netmask 120
434
+
435
+
auto eth3
436
+
iface eth3 inet static
437
+
address 10.2.3.2
438
+
netmask 255.255.255.0
439
+
440
+
iface eth3 inet6 static
441
+
address 2002:3::2
442
+
netmask 120
443
+
444
+
auto eth4
445
+
iface eth4 inet static
446
+
address 10.2.4.2
447
+
netmask 255.255.255.0
448
+
449
+
iface eth4 inet6 static
450
+
address 2002:4::2
451
+
netmask 120
-418
Documentation/networking/vrf.txt
-418
Documentation/networking/vrf.txt
···
1
-
Virtual Routing and Forwarding (VRF)
2
-
====================================
3
-
The VRF device combined with ip rules provides the ability to create virtual
4
-
routing and forwarding domains (aka VRFs, VRF-lite to be specific) in the
5
-
Linux network stack. One use case is the multi-tenancy problem where each
6
-
tenant has their own unique routing tables and in the very least need
7
-
different default gateways.
8
-
9
-
Processes can be "VRF aware" by binding a socket to the VRF device. Packets
10
-
through the socket then use the routing table associated with the VRF
11
-
device. An important feature of the VRF device implementation is that it
12
-
impacts only Layer 3 and above so L2 tools (e.g., LLDP) are not affected
13
-
(ie., they do not need to be run in each VRF). The design also allows
14
-
the use of higher priority ip rules (Policy Based Routing, PBR) to take
15
-
precedence over the VRF device rules directing specific traffic as desired.
16
-
17
-
In addition, VRF devices allow VRFs to be nested within namespaces. For
18
-
example network namespaces provide separation of network interfaces at the
19
-
device layer, VLANs on the interfaces within a namespace provide L2 separation
20
-
and then VRF devices provide L3 separation.
21
-
22
-
Design
23
-
------
24
-
A VRF device is created with an associated route table. Network interfaces
25
-
are then enslaved to a VRF device:
26
-
27
-
+-----------------------------+
28
-
| vrf-blue | ===> route table 10
29
-
+-----------------------------+
30
-
| | |
31
-
+------+ +------+ +-------------+
32
-
| eth1 | | eth2 | ... | bond1 |
33
-
+------+ +------+ +-------------+
34
-
| |
35
-
+------+ +------+
36
-
| eth8 | | eth9 |
37
-
+------+ +------+
38
-
39
-
Packets received on an enslaved device and are switched to the VRF device
40
-
in the IPv4 and IPv6 processing stacks giving the impression that packets
41
-
flow through the VRF device. Similarly on egress routing rules are used to
42
-
send packets to the VRF device driver before getting sent out the actual
43
-
interface. This allows tcpdump on a VRF device to capture all packets into
44
-
and out of the VRF as a whole.[1] Similarly, netfilter[2] and tc rules can be
45
-
applied using the VRF device to specify rules that apply to the VRF domain
46
-
as a whole.
47
-
48
-
[1] Packets in the forwarded state do not flow through the device, so those
49
-
packets are not seen by tcpdump. Will revisit this limitation in a
50
-
future release.
51
-
52
-
[2] Iptables on ingress supports PREROUTING with skb->dev set to the real
53
-
ingress device and both INPUT and PREROUTING rules with skb->dev set to
54
-
the VRF device. For egress POSTROUTING and OUTPUT rules can be written
55
-
using either the VRF device or real egress device.
56
-
57
-
Setup
58
-
-----
59
-
1. VRF device is created with an association to a FIB table.
60
-
e.g, ip link add vrf-blue type vrf table 10
61
-
ip link set dev vrf-blue up
62
-
63
-
2. An l3mdev FIB rule directs lookups to the table associated with the device.
64
-
A single l3mdev rule is sufficient for all VRFs. The VRF device adds the
65
-
l3mdev rule for IPv4 and IPv6 when the first device is created with a
66
-
default preference of 1000. Users may delete the rule if desired and add
67
-
with a different priority or install per-VRF rules.
68
-
69
-
Prior to the v4.8 kernel iif and oif rules are needed for each VRF device:
70
-
ip ru add oif vrf-blue table 10
71
-
ip ru add iif vrf-blue table 10
72
-
73
-
3. Set the default route for the table (and hence default route for the VRF).
74
-
ip route add table 10 unreachable default metric 4278198272
75
-
76
-
This high metric value ensures that the default unreachable route can
77
-
be overridden by a routing protocol suite. FRRouting interprets
78
-
kernel metrics as a combined admin distance (upper byte) and priority
79
-
(lower 3 bytes). Thus the above metric translates to [255/8192].
80
-
81
-
4. Enslave L3 interfaces to a VRF device.
82
-
ip link set dev eth1 master vrf-blue
83
-
84
-
Local and connected routes for enslaved devices are automatically moved to
85
-
the table associated with VRF device. Any additional routes depending on
86
-
the enslaved device are dropped and will need to be reinserted to the VRF
87
-
FIB table following the enslavement.
88
-
89
-
The IPv6 sysctl option keep_addr_on_down can be enabled to keep IPv6 global
90
-
addresses as VRF enslavement changes.
91
-
sysctl -w net.ipv6.conf.all.keep_addr_on_down=1
92
-
93
-
5. Additional VRF routes are added to associated table.
94
-
ip route add table 10 ...
95
-
96
-
97
-
Applications
98
-
------------
99
-
Applications that are to work within a VRF need to bind their socket to the
100
-
VRF device:
101
-
102
-
setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);
103
-
104
-
or to specify the output device using cmsg and IP_PKTINFO.
105
-
106
-
By default the scope of the port bindings for unbound sockets is
107
-
limited to the default VRF. That is, it will not be matched by packets
108
-
arriving on interfaces enslaved to an l3mdev and processes may bind to
109
-
the same port if they bind to an l3mdev.
110
-
111
-
TCP & UDP services running in the default VRF context (ie., not bound
112
-
to any VRF device) can work across all VRF domains by enabling the
113
-
tcp_l3mdev_accept and udp_l3mdev_accept sysctl options:
114
-
115
-
sysctl -w net.ipv4.tcp_l3mdev_accept=1
116
-
sysctl -w net.ipv4.udp_l3mdev_accept=1
117
-
118
-
These options are disabled by default so that a socket in a VRF is only
119
-
selected for packets in that VRF. There is a similar option for RAW
120
-
sockets, which is enabled by default for reasons of backwards compatibility.
121
-
This is so as to specify the output device with cmsg and IP_PKTINFO, but
122
-
using a socket not bound to the corresponding VRF. This allows e.g. older ping
123
-
implementations to be run with specifying the device but without executing it
124
-
in the VRF. This option can be disabled so that packets received in a VRF
125
-
context are only handled by a raw socket bound to the VRF, and packets in the
126
-
default VRF are only handled by a socket not bound to any VRF:
127
-
128
-
sysctl -w net.ipv4.raw_l3mdev_accept=0
129
-
130
-
netfilter rules on the VRF device can be used to limit access to services
131
-
running in the default VRF context as well.
132
-
133
-
################################################################################
134
-
135
-
Using iproute2 for VRFs
136
-
=======================
137
-
iproute2 supports the vrf keyword as of v4.7. For backwards compatibility this
138
-
section lists both commands where appropriate -- with the vrf keyword and the
139
-
older form without it.
140
-
141
-
1. Create a VRF
142
-
143
-
To instantiate a VRF device and associate it with a table:
144
-
$ ip link add dev NAME type vrf table ID
145
-
146
-
As of v4.8 the kernel supports the l3mdev FIB rule where a single rule
147
-
covers all VRFs. The l3mdev rule is created for IPv4 and IPv6 on first
148
-
device create.
149
-
150
-
2. List VRFs
151
-
152
-
To list VRFs that have been created:
153
-
$ ip [-d] link show type vrf
154
-
NOTE: The -d option is needed to show the table id
155
-
156
-
For example:
157
-
$ ip -d link show type vrf
158
-
11: mgmt: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
159
-
link/ether 72:b3:ba:91:e2:24 brd ff:ff:ff:ff:ff:ff promiscuity 0
160
-
vrf table 1 addrgenmode eui64
161
-
12: red: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
162
-
link/ether b6:6f:6e:f6:da:73 brd ff:ff:ff:ff:ff:ff promiscuity 0
163
-
vrf table 10 addrgenmode eui64
164
-
13: blue: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
165
-
link/ether 36:62:e8:7d:bb:8c brd ff:ff:ff:ff:ff:ff promiscuity 0
166
-
vrf table 66 addrgenmode eui64
167
-
14: green: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
168
-
link/ether e6:28:b8:63:70:bb brd ff:ff:ff:ff:ff:ff promiscuity 0
169
-
vrf table 81 addrgenmode eui64
170
-
171
-
172
-
Or in brief output:
173
-
174
-
$ ip -br link show type vrf
175
-
mgmt UP 72:b3:ba:91:e2:24 <NOARP,MASTER,UP,LOWER_UP>
176
-
red UP b6:6f:6e:f6:da:73 <NOARP,MASTER,UP,LOWER_UP>
177
-
blue UP 36:62:e8:7d:bb:8c <NOARP,MASTER,UP,LOWER_UP>
178
-
green UP e6:28:b8:63:70:bb <NOARP,MASTER,UP,LOWER_UP>
179
-
180
-
181
-
3. Assign a Network Interface to a VRF
182
-
183
-
Network interfaces are assigned to a VRF by enslaving the netdevice to a
184
-
VRF device:
185
-
$ ip link set dev NAME master NAME
186
-
187
-
On enslavement connected and local routes are automatically moved to the
188
-
table associated with the VRF device.
189
-
190
-
For example:
191
-
$ ip link set dev eth0 master mgmt
192
-
193
-
194
-
4. Show Devices Assigned to a VRF
195
-
196
-
To show devices that have been assigned to a specific VRF add the master
197
-
option to the ip command:
198
-
$ ip link show vrf NAME
199
-
$ ip link show master NAME
200
-
201
-
For example:
202
-
$ ip link show vrf red
203
-
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
204
-
link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
205
-
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
206
-
link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
207
-
7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN mode DEFAULT group default qlen 1000
208
-
link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
209
-
210
-
211
-
Or using the brief output:
212
-
$ ip -br link show vrf red
213
-
eth1 UP 02:00:00:00:02:02 <BROADCAST,MULTICAST,UP,LOWER_UP>
214
-
eth2 UP 02:00:00:00:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP>
215
-
eth5 DOWN 02:00:00:00:02:06 <BROADCAST,MULTICAST>
216
-
217
-
218
-
5. Show Neighbor Entries for a VRF
219
-
220
-
To list neighbor entries associated with devices enslaved to a VRF device
221
-
add the master option to the ip command:
222
-
$ ip [-6] neigh show vrf NAME
223
-
$ ip [-6] neigh show master NAME
224
-
225
-
For example:
226
-
$ ip neigh show vrf red
227
-
10.2.1.254 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
228
-
10.2.2.254 dev eth2 lladdr 5e:54:01:6a:ee:80 REACHABLE
229
-
230
-
$ ip -6 neigh show vrf red
231
-
2002:1::64 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
232
-
233
-
234
-
6. Show Addresses for a VRF
235
-
236
-
To show addresses for interfaces associated with a VRF add the master
237
-
option to the ip command:
238
-
$ ip addr show vrf NAME
239
-
$ ip addr show master NAME
240
-
241
-
For example:
242
-
$ ip addr show vrf red
243
-
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
244
-
link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
245
-
inet 10.2.1.2/24 brd 10.2.1.255 scope global eth1
246
-
valid_lft forever preferred_lft forever
247
-
inet6 2002:1::2/120 scope global
248
-
valid_lft forever preferred_lft forever
249
-
inet6 fe80::ff:fe00:202/64 scope link
250
-
valid_lft forever preferred_lft forever
251
-
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
252
-
link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
253
-
inet 10.2.2.2/24 brd 10.2.2.255 scope global eth2
254
-
valid_lft forever preferred_lft forever
255
-
inet6 2002:2::2/120 scope global
256
-
valid_lft forever preferred_lft forever
257
-
inet6 fe80::ff:fe00:203/64 scope link
258
-
valid_lft forever preferred_lft forever
259
-
7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN group default qlen 1000
260
-
link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
261
-
262
-
Or in brief format:
263
-
$ ip -br addr show vrf red
264
-
eth1 UP 10.2.1.2/24 2002:1::2/120 fe80::ff:fe00:202/64
265
-
eth2 UP 10.2.2.2/24 2002:2::2/120 fe80::ff:fe00:203/64
266
-
eth5 DOWN
267
-
268
-
269
-
7. Show Routes for a VRF
270
-
271
-
To show routes for a VRF use the ip command to display the table associated
272
-
with the VRF device:
273
-
$ ip [-6] route show vrf NAME
274
-
$ ip [-6] route show table ID
275
-
276
-
For example:
277
-
$ ip route show vrf red
278
-
unreachable default metric 4278198272
279
-
broadcast 10.2.1.0 dev eth1 proto kernel scope link src 10.2.1.2
280
-
10.2.1.0/24 dev eth1 proto kernel scope link src 10.2.1.2
281
-
local 10.2.1.2 dev eth1 proto kernel scope host src 10.2.1.2
282
-
broadcast 10.2.1.255 dev eth1 proto kernel scope link src 10.2.1.2
283
-
broadcast 10.2.2.0 dev eth2 proto kernel scope link src 10.2.2.2
284
-
10.2.2.0/24 dev eth2 proto kernel scope link src 10.2.2.2
285
-
local 10.2.2.2 dev eth2 proto kernel scope host src 10.2.2.2
286
-
broadcast 10.2.2.255 dev eth2 proto kernel scope link src 10.2.2.2
287
-
288
-
$ ip -6 route show vrf red
289
-
local 2002:1:: dev lo proto none metric 0 pref medium
290
-
local 2002:1::2 dev lo proto none metric 0 pref medium
291
-
2002:1::/120 dev eth1 proto kernel metric 256 pref medium
292
-
local 2002:2:: dev lo proto none metric 0 pref medium
293
-
local 2002:2::2 dev lo proto none metric 0 pref medium
294
-
2002:2::/120 dev eth2 proto kernel metric 256 pref medium
295
-
local fe80:: dev lo proto none metric 0 pref medium
296
-
local fe80:: dev lo proto none metric 0 pref medium
297
-
local fe80::ff:fe00:202 dev lo proto none metric 0 pref medium
298
-
local fe80::ff:fe00:203 dev lo proto none metric 0 pref medium
299
-
fe80::/64 dev eth1 proto kernel metric 256 pref medium
300
-
fe80::/64 dev eth2 proto kernel metric 256 pref medium
301
-
ff00::/8 dev red metric 256 pref medium
302
-
ff00::/8 dev eth1 metric 256 pref medium
303
-
ff00::/8 dev eth2 metric 256 pref medium
304
-
unreachable default dev lo metric 4278198272 error -101 pref medium
305
-
306
-
8. Route Lookup for a VRF
307
-
308
-
A test route lookup can be done for a VRF:
309
-
$ ip [-6] route get vrf NAME ADDRESS
310
-
$ ip [-6] route get oif NAME ADDRESS
311
-
312
-
For example:
313
-
$ ip route get 10.2.1.40 vrf red
314
-
10.2.1.40 dev eth1 table red src 10.2.1.2
315
-
cache
316
-
317
-
$ ip -6 route get 2002:1::32 vrf red
318
-
2002:1::32 from :: dev eth1 table red proto kernel src 2002:1::2 metric 256 pref medium
319
-
320
-
321
-
9. Removing Network Interface from a VRF
322
-
323
-
Network interfaces are removed from a VRF by breaking the enslavement to
324
-
the VRF device:
325
-
$ ip link set dev NAME nomaster
326
-
327
-
Connected routes are moved back to the default table and local entries are
328
-
moved to the local table.
329
-
330
-
For example:
331
-
$ ip link set dev eth0 nomaster
332
-
333
-
--------------------------------------------------------------------------------
334
-
335
-
Commands used in this example:
336
-
337
-
cat >> /etc/iproute2/rt_tables.d/vrf.conf <<EOF
338
-
1 mgmt
339
-
10 red
340
-
66 blue
341
-
81 green
342
-
EOF
343
-
344
-
function vrf_create
345
-
{
346
-
VRF=$1
347
-
TBID=$2
348
-
349
-
# create VRF device
350
-
ip link add ${VRF} type vrf table ${TBID}
351
-
352
-
if [ "${VRF}" != "mgmt" ]; then
353
-
ip route add table ${TBID} unreachable default metric 4278198272
354
-
fi
355
-
ip link set dev ${VRF} up
356
-
}
357
-
358
-
vrf_create mgmt 1
359
-
ip link set dev eth0 master mgmt
360
-
361
-
vrf_create red 10
362
-
ip link set dev eth1 master red
363
-
ip link set dev eth2 master red
364
-
ip link set dev eth5 master red
365
-
366
-
vrf_create blue 66
367
-
ip link set dev eth3 master blue
368
-
369
-
vrf_create green 81
370
-
ip link set dev eth4 master green
371
-
372
-
373
-
Interface addresses from /etc/network/interfaces:
374
-
auto eth0
375
-
iface eth0 inet static
376
-
address 10.0.0.2
377
-
netmask 255.255.255.0
378
-
gateway 10.0.0.254
379
-
380
-
iface eth0 inet6 static
381
-
address 2000:1::2
382
-
netmask 120
383
-
384
-
auto eth1
385
-
iface eth1 inet static
386
-
address 10.2.1.2
387
-
netmask 255.255.255.0
388
-
389
-
iface eth1 inet6 static
390
-
address 2002:1::2
391
-
netmask 120
392
-
393
-
auto eth2
394
-
iface eth2 inet static
395
-
address 10.2.2.2
396
-
netmask 255.255.255.0
397
-
398
-
iface eth2 inet6 static
399
-
address 2002:2::2
400
-
netmask 120
401
-
402
-
auto eth3
403
-
iface eth3 inet static
404
-
address 10.2.3.2
405
-
netmask 255.255.255.0
406
-
407
-
iface eth3 inet6 static
408
-
address 2002:3::2
409
-
netmask 120
410
-
411
-
auto eth4
412
-
iface eth4 inet static
413
-
address 10.2.4.2
414
-
netmask 255.255.255.0
415
-
416
-
iface eth4 inet6 static
417
-
address 2002:4::2
418
-
netmask 120