tjh.dev
Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
docs: Add Documentation for Mediated devices
Add file Documentation/vfio-mediated-device.txt that include details of
mediated device framework.
Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
Signed-off-by: Neo Jia <cjia@nvidia.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
authored by
Kirti Wankhede
and committed by
Alex Williamson
8e1c5a40
2818c6e9
+298
+297
Documentation/vfio-mediated-device.txt
+297
Documentation/vfio-mediated-device.txt
···
1
+
/*
2
+
* VFIO Mediated devices
3
+
*
4
+
* Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
5
+
* Author: Neo Jia <cjia@nvidia.com>
6
+
* Kirti Wankhede <kwankhede@nvidia.com>
7
+
*
8
+
* This program is free software; you can redistribute it and/or modify
9
+
* it under the terms of the GNU General Public License version 2 as
10
+
* published by the Free Software Foundation.
11
+
*/
12
+
13
+
Virtual Function I/O (VFIO) Mediated devices[1]
14
+
===============================================
15
+
16
+
The number of use cases for virtualizing DMA devices that do not have built-in
17
+
SR_IOV capability is increasing. Previously, to virtualize such devices,
18
+
developers had to create their own management interfaces and APIs, and then
19
+
integrate them with user space software. To simplify integration with user space
20
+
software, we have identified common requirements and a unified management
21
+
interface for such devices.
22
+
23
+
The VFIO driver framework provides unified APIs for direct device access. It is
24
+
an IOMMU/device-agnostic framework for exposing direct device access to user
25
+
space in a secure, IOMMU-protected environment. This framework is used for
26
+
multiple devices, such as GPUs, network adapters, and compute accelerators. With
27
+
direct device access, virtual machines or user space applications have direct
28
+
access to the physical device. This framework is reused for mediated devices.
29
+
30
+
The mediated core driver provides a common interface for mediated device
31
+
management that can be used by drivers of different devices. This module
32
+
provides a generic interface to perform these operations:
33
+
34
+
* Create and destroy a mediated device
35
+
* Add a mediated device to and remove it from a mediated bus driver
36
+
* Add a mediated device to and remove it from an IOMMU group
37
+
38
+
The mediated core driver also provides an interface to register a bus driver.
39
+
For example, the mediated VFIO mdev driver is designed for mediated devices and
40
+
supports VFIO APIs. The mediated bus driver adds a mediated device to and
41
+
removes it from a VFIO group.
42
+
43
+
The following high-level block diagram shows the main components and interfaces
44
+
in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM
45
+
devices as examples, as these devices are the first devices to use this module.
46
+
47
+
+---------------+
48
+
| |
49
+
| +-----------+ | mdev_register_driver() +--------------+
50
+
| | | +<------------------------+ |
51
+
| | mdev | | | |
52
+
| | bus | +------------------------>+ vfio_mdev.ko |<-> VFIO user
53
+
| | driver | | probe()/remove() | | APIs
54
+
| | | | +--------------+
55
+
| +-----------+ |
56
+
| |
57
+
| MDEV CORE |
58
+
| MODULE |
59
+
| mdev.ko |
60
+
| +-----------+ | mdev_register_device() +--------------+
61
+
| | | +<------------------------+ |
62
+
| | | | | nvidia.ko |<-> physical
63
+
| | | +------------------------>+ | device
64
+
| | | | callbacks +--------------+
65
+
| | Physical | |
66
+
| | device | | mdev_register_device() +--------------+
67
+
| | interface | |<------------------------+ |
68
+
| | | | | i915.ko |<-> physical
69
+
| | | +------------------------>+ | device
70
+
| | | | callbacks +--------------+
71
+
| | | |
72
+
| | | | mdev_register_device() +--------------+
73
+
| | | +<------------------------+ |
74
+
| | | | | ccw_device.ko|<-> physical
75
+
| | | +------------------------>+ | device
76
+
| | | | callbacks +--------------+
77
+
| +-----------+ |
78
+
+---------------+
79
+
80
+
81
+
Registration Interfaces
82
+
=======================
83
+
84
+
The mediated core driver provides the following types of registration
85
+
interfaces:
86
+
87
+
* Registration interface for a mediated bus driver
88
+
* Physical device driver interface
89
+
90
+
Registration Interface for a Mediated Bus Driver
91
+
------------------------------------------------
92
+
93
+
The registration interface for a mediated bus driver provides the following
94
+
structure to represent a mediated device's driver:
95
+
96
+
/*
97
+
* struct mdev_driver [2] - Mediated device's driver
98
+
* @name: driver name
99
+
* @probe: called when new device created
100
+
* @remove: called when device removed
101
+
* @driver: device driver structure
102
+
*/
103
+
struct mdev_driver {
104
+
const char *name;
105
+
int (*probe) (struct device *dev);
106
+
void (*remove) (struct device *dev);
107
+
struct device_driver driver;
108
+
};
109
+
110
+
A mediated bus driver for mdev should use this structure in the function calls
111
+
to register and unregister itself with the core driver:
112
+
113
+
* Register:
114
+
115
+
extern int mdev_register_driver(struct mdev_driver *drv,
116
+
struct module *owner);
117
+
118
+
* Unregister:
119
+
120
+
extern void mdev_unregister_driver(struct mdev_driver *drv);
121
+
122
+
The mediated bus driver is responsible for adding mediated devices to the VFIO
123
+
group when devices are bound to the driver and removing mediated devices from
124
+
the VFIO when devices are unbound from the driver.
125
+
126
+
127
+
Physical Device Driver Interface
128
+
--------------------------------
129
+
130
+
The physical device driver interface provides the parent_ops[3] structure to
131
+
define the APIs to manage work in the mediated core driver that is related to
132
+
the physical device.
133
+
134
+
The structures in the parent_ops structure are as follows:
135
+
136
+
* dev_attr_groups: attributes of the parent device
137
+
* mdev_attr_groups: attributes of the mediated device
138
+
* supported_config: attributes to define supported configurations
139
+
140
+
The functions in the parent_ops structure are as follows:
141
+
142
+
* create: allocate basic resources in a driver for a mediated device
143
+
* remove: free resources in a driver when a mediated device is destroyed
144
+
145
+
The callbacks in the parent_ops structure are as follows:
146
+
147
+
* open: open callback of mediated device
148
+
* close: close callback of mediated device
149
+
* ioctl: ioctl callback of mediated device
150
+
* read : read emulation callback
151
+
* write: write emulation callback
152
+
* mmap: mmap emulation callback
153
+
154
+
A driver should use the parent_ops structure in the function call to register
155
+
itself with the mdev core driver:
156
+
157
+
extern int mdev_register_device(struct device *dev,
158
+
const struct parent_ops *ops);
159
+
160
+
However, the parent_ops structure is not required in the function call that a
161
+
driver should use to unregister itself with the mdev core driver:
162
+
163
+
extern void mdev_unregister_device(struct device *dev);
164
+
165
+
166
+
Mediated Device Management Interface Through sysfs
167
+
==================================================
168
+
169
+
The management interface through sysfs enables user space software, such as
170
+
libvirt, to query and configure mediated devices in a hardware-agnostic fashion.
171
+
This management interface provides flexibility to the underlying physical
172
+
device's driver to support features such as:
173
+
174
+
* Mediated device hot plug
175
+
* Multiple mediated devices in a single virtual machine
176
+
* Multiple mediated devices from different physical devices
177
+
178
+
Links in the mdev_bus Class Directory
179
+
-------------------------------------
180
+
The /sys/class/mdev_bus/ directory contains links to devices that are registered
181
+
with the mdev core driver.
182
+
183
+
Directories and files under the sysfs for Each Physical Device
184
+
--------------------------------------------------------------
185
+
186
+
|- [parent physical device]
187
+
|--- Vendor-specific-attributes [optional]
188
+
|--- [mdev_supported_types]
189
+
| |--- [<type-id>]
190
+
| | |--- create
191
+
| | |--- name
192
+
| | |--- available_instances
193
+
| | |--- device_api
194
+
| | |--- description
195
+
| | |--- [devices]
196
+
| |--- [<type-id>]
197
+
| | |--- create
198
+
| | |--- name
199
+
| | |--- available_instances
200
+
| | |--- device_api
201
+
| | |--- description
202
+
| | |--- [devices]
203
+
| |--- [<type-id>]
204
+
| |--- create
205
+
| |--- name
206
+
| |--- available_instances
207
+
| |--- device_api
208
+
| |--- description
209
+
| |--- [devices]
210
+
211
+
* [mdev_supported_types]
212
+
213
+
The list of currently supported mediated device types and their details.
214
+
215
+
[<type-id>], device_api, and available_instances are mandatory attributes
216
+
that should be provided by vendor driver.
217
+
218
+
* [<type-id>]
219
+
220
+
The [<type-id>] name is created by adding the the device driver string as a
221
+
prefix to the string provided by the vendor driver. This format of this name
222
+
is as follows:
223
+
224
+
sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name);
225
+
226
+
* device_api
227
+
228
+
This attribute should show which device API is being created, for example,
229
+
"vfio-pci" for a PCI device.
230
+
231
+
* available_instances
232
+
233
+
This attribute should show the number of devices of type <type-id> that can be
234
+
created.
235
+
236
+
* [device]
237
+
238
+
This directory contains links to the devices of type <type-id> that have been
239
+
created.
240
+
241
+
* name
242
+
243
+
This attribute should show human readable name. This is optional attribute.
244
+
245
+
* description
246
+
247
+
This attribute should show brief features/description of the type. This is
248
+
optional attribute.
249
+
250
+
Directories and Files Under the sysfs for Each mdev Device
251
+
----------------------------------------------------------
252
+
253
+
|- [parent phy device]
254
+
|--- [$MDEV_UUID]
255
+
|--- remove
256
+
|--- mdev_type {link to its type}
257
+
|--- vendor-specific-attributes [optional]
258
+
259
+
* remove (write only)
260
+
Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can
261
+
fail the remove() callback if that device is active and the vendor driver
262
+
doesn't support hot unplug.
263
+
264
+
Example:
265
+
# echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove
266
+
267
+
Mediated device Hot plug:
268
+
------------------------
269
+
270
+
Mediated devices can be created and assigned at runtime. The procedure to hot
271
+
plug a mediated device is the same as the procedure to hot plug a PCI device.
272
+
273
+
Translation APIs for Mediated Devices
274
+
=====================================
275
+
276
+
The following APIs are provided for translating user pfn to host pfn in a VFIO
277
+
driver:
278
+
279
+
extern int vfio_pin_pages(struct device *dev, unsigned long *user_pfn,
280
+
int npage, int prot, unsigned long *phys_pfn);
281
+
282
+
extern int vfio_unpin_pages(struct device *dev, unsigned long *user_pfn,
283
+
int npage);
284
+
285
+
These functions call back into the back-end IOMMU module by using the pin_pages
286
+
and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently
287
+
these callbacks are supported in the TYPE1 IOMMU module. To enable them for
288
+
other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide
289
+
these two callback functions.
290
+
291
+
References
292
+
----------
293
+
294
+
[1] See Documentation/vfio.txt for more information on VFIO.
295
+
[2] struct mdev_driver in include/linux/mdev.h
296
+
[3] struct parent_ops in include/linux/mdev.h
297
+
[4] struct vfio_iommu_driver_ops in include/linux/vfio.h